Maybe it's a different Node version, maybe an environment variable is missing, or maybe a system dependency doesn't match. You spend an hour debugging something that was never actually a code problem.

Docker fixes this at the root. With Docker, you stop shipping just code. The Node version, dependencies, and config all travel inside the container. Your laptop, a CI server, a production VM — it behaves the same on all of them. No more environment surprises.

In this tutorial, we'll go through all this step by step: a multi-stage Dockerfile, using Docker Compose with PostgreSQL for local development, and a GitHub Actions workflow that pushes a fresh image to Docker Hub on every merge to main.

The complete code for this tutorial is available on GitHub.

Table of Contents

1. Prerequisites

2. The Sample Application

3. Writing the Dockerfile

4. The .dockerignore File

5. The .gitignore File

6. Build and Test the Image Locally

7. Docker Compose for Local Development

8. Automate the Build with GitHub Actions

9. Deploying the Image

10. Wrapping Up

Prerequisites

• Node.js 18+

• Docker Desktop, which you can download at docs.docker.com/get-docker. Windows users need WSL 2 before Docker starts. Open PowerShell as Administrator and run wsl --install. After the restart, Docker Desktop will install without issues.

• A GitHub account

• A Docker Hub account (free at hub.docker.com)

• Some Express.js experience helps, but isn't required

The Sample Application

We're building a task management API with Express and PostgreSQL. Keep in mind the app is just a vehicle to teach you how this works. The Dockerfile and pipeline we set up here work the same way for any Node.js project.

Create the project:

mkdir nodejs-docker-cicd && cd nodejs-docker-cicd
npm init -y
npm install express pg dotenv
npm install --save-dev nodemon

Create src/index.js:

const express = require('express');
const { Pool } = require('pg');
require('dotenv').config();

const app = express();
app.use(express.json());

const pool = new Pool({
  host: process.env.DB_HOST,
  port: process.env.DB_PORT,
  database: process.env.DB_NAME,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
});

// Create table on startup
pool.query(`
  CREATE TABLE IF NOT EXISTS tasks (
    id SERIAL PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    completed BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT NOW()
  )
`).catch(console.error);

// Health check — required for Docker HEALTHCHECK and load balancers
app.get('/health', (req, res) => {
  res.json({ status: 'ok', timestamp: new Date().toISOString() });
});

app.get('/tasks', async (req, res) => {
  try {
    const result = await pool.query('SELECT * FROM tasks ORDER BY created_at DESC');
    res.json(result.rows);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.post('/tasks', async (req, res) => {
  const { title } = req.body;
  if (!title) return res.status(400).json({ error: 'Title is required' });
  try {
    const result = await pool.query(
      'INSERT INTO tasks (title) VALUES ($1) RETURNING *',
      [title]
    );
    res.status(201).json(result.rows[0]);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.patch('/tasks/:id', async (req, res) => {
  const { id } = req.params;
  const { completed } = req.body;
  try {
    const result = await pool.query(
      'UPDATE tasks SET completed = $1 WHERE id = $2 RETURNING *',
      [completed, id]
    );
    if (result.rows.length === 0) return res.status(404).json({ error: 'Task not found' });
    res.json(result.rows[0]);
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running on port ${PORT}`));

Open package.json and update the "scripts" section:

"scripts": {
  "start": "node src/index.js",
  "dev": "nodemon src/index.js"
}

npm start runs the app directly with Node. npm run dev uses nodemon so the server restarts automatically when you edit a file.

For running without Docker, create a .env file:

DB_HOST=localhost
DB_PORT=5432
DB_NAME=tasksdb
DB_USER=postgres
DB_PASSWORD=yourpassword
PORT=3000

Notice that all database credentials come from environment variables rather than being hardcoded. Swap the variables, and the same image runs against your local database or a production one — no code changes needed. The /health endpoint is what Docker pings to know the app is actually handling requests.

Writing the Dockerfile

Before touching the Dockerfile, there are two terms you'll keep seeing. An image is a packaged, immutable version of your app — Node runtime, code, dependencies, everything together in one artifact. A container is a running instance of that image. One image, many containers, any machine.

Here's the Dockerfile we'll use:

# ── Stage 1: Install dependencies ──────────────────────────────────────────
FROM node:18-alpine AS builder

WORKDIR /app

# Copy package files first — Docker caches this layer separately.
# If you only change src code (not package.json), Docker skips npm ci on rebuild.
COPY package*.json ./
RUN npm ci

COPY . .


# ── Stage 2: Production image ───────────────────────────────────────────────
FROM node:18-alpine AS production

# Create a non-root user — running as root inside a container is a security risk
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodeuser -u 1001

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

# Copy only the source code from the builder stage (not node_modules or dev files)
COPY --from=builder /app/src ./src

RUN chown -R nodeuser:nodejs /app
USER nodeuser

EXPOSE 3000

# Docker will ping /health every 30s. If it fails 3 times, the container is marked unhealthy.
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

CMD ["node", "src/index.js"]

This is a multi-stage build. The first stage (builder) installs everything, including dev dependencies. The second stage (production) starts fresh and only copies what the app needs to run. Nodemon, test frameworks, and anything else dev-only never make it into the final image.

The size difference is real. A node:18 Debian image is over 950MB. Switch to node:18-alpine and cut out the dev dependencies, and the final image lands around 150–200MB instead. A smaller image means faster pushes and faster deploys.

npm ci instead of npm install is a deliberate choice for CI/CD. It reads exact versions from package-lock.json and fails hard if the lockfile doesn't match package.json. Every build on every machine installs the exact same versions — no surprises from a dependency that quietly updated overnight.

The nodeuser account exists because containers run as root by default. That's fine until something goes wrong. A non-root user means that an attacker who gets inside the container can't just do whatever they want.

The .dockerignore File

Create .dockerignore before building:

node_modules
npm-debug.log
.env
.git
.gitignore
README.md
Dockerfile
.dockerignore

The node_modules exclusion is the critical one. Your local modules were compiled for your operating system — macOS or Windows binaries won't work inside a Linux container. Excluding them means Docker installs fresh modules during the build, compiled for the correct platform. Without this exclusion, you'd either copy broken binaries into the image or waste time uploading hundreds of megabytes to the build context.

Never put .env in an image. Passwords, API keys, anything sensitive — those go in at runtime as environment variables, never inside the image itself.

The .gitignore File

One more thing before the first commit: a .gitignore. You don't want node_modules or .env tracked:

node_modules/
.env
.env.local
npm-debug.log*
logs/
.DS_Store
Thumbs.db
.vscode/
.idea/
dist/
build/

Build and Test the Image Locally

Open Docker Desktop first and give it a moment. On Windows, you'll see a whale icon in the taskbar that animates while the engine is starting up. Once it goes still, you're good to run Docker commands. If you try to run Docker before the engine is up, you'll hit this:

ERROR: Error response from daemon: Docker Desktop is unable to start

If that happens, quit Docker Desktop. Open PowerShell as Administrator, run wsl --update, and restart. Then go to Control Panel → Programs → Turn Windows features on or off. Both Hyper-V and Virtual Machine Platform need to be checked. After the restart, Docker Desktop should come up fine.

It's worth knowing about this error too:

docker : The term 'docker' is not recognized as the name of a cmdlet, function,
script file, or operable program.

This means that Docker Desktop isn't running or isn't installed. Open it from the Start menu and wait.

Run the build:

docker build -t nodejs-docker-cicd:latest .

The first time takes roughly 30 seconds since Docker has to pull node:18-alpine from the internet. Once that's cached, subsequent builds are much quicker. Both stages will scroll by:

[+] Building 33.1s (17/17) FINISHED
 => [builder 1/5] FROM docker.io/library/node:18-alpine       20.9s
 => [builder 4/5] RUN npm ci                                   3.5s
 => [production 5/7] RUN npm ci --only=production              3.2s
 => [production 7/7] RUN chown -R nodeuser:nodejs /app         3.2s
 => exporting to image                                         1.5s
 => => naming to docker.io/library/nodejs-docker-cicd:latest     0.0s

When you see (17/17) FINISHED the image is built. Check the size:

docker images nodejs-docker-cicd

IMAGE                     ID             DISK USAGE   CONTENT SIZE
nodejs-docker-cicd:latest   c9eed311d999        198MB         47.5MB

CONTENT SIZE (47.5MB) is the compressed size that gets pushed to Docker Hub. DISK USAGE (198MB) is what it takes up on disk locally. Compare that to a node:18 Debian image at 950MB+, and you can see why the Alpine base and multi-stage approach matter.

On subsequent builds, Docker reuses cached layers. Edit only your source files without touching package.json and the npm ci step gets skipped completely. That 33-second first build becomes 3 seconds.

Docker Compose for Local Development

The app needs a database. Setting up PostgreSQL locally means every developer who clones the repo has to do it, too. Docker Compose handles this: one file defines both services, and one command starts them.

Create docker-compose.yml:

services:
  app:
    build:
      context: .
      target: production
    ports:
      - '3000:3000'
    environment:
      DB_HOST: postgres
      DB_PORT: 5432
      DB_NAME: tasksdb
      DB_USER: postgres
      DB_PASSWORD: postgres
      PORT: 3000
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: tasksdb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - '5432:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres']
      interval: 5s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:

A few things worth pointing out. DB_HOST is set to postgres. That's the service name, not localhost. Containers on the same Docker network reach each other by service name. Put localhost there and the app tries to connect to itself.

depends_on with condition: service_healthy holds the app back until Postgres actually passes its health check. Skip this and the app starts, tries to connect to a database that isn't ready yet, and crashes. The health check pings pg_isready every 5 seconds. Once it gets a green response, the app container starts.

The named volume postgres_data keeps your data alive between restarts. Run docker compose down and the data is still there next time. Add --volumes to wipe it clean.

Start both services:

docker compose up --build

You'll see PostgreSQL initialize and then the app start. Once you see Server running on port 3000 in the logs, the stack is up.

Open a second terminal to test — leave the compose logs running in the first one.

Linux/macOS:

curl -X POST http://localhost:3000/tasks \
  -H "Content-Type: application/json" \
  -d '{"title": "Learn Docker"}'

curl http://localhost:3000/tasks

curl http://localhost:3000/health

Windows PowerShell: Typing curl in PowerShell runs Invoke-WebRequest, not actual curl. Run curl.exe instead. For JSON bodies, write to a file first:

'{"title": "Learn Docker"}' | Set-Content body.json
curl.exe -X POST http://localhost:3000/tasks -H "Content-Type: application/json" --data `@body.json

curl.exe http://localhost:3000/tasks

curl.exe http://localhost:3000/health

The backtick before @body.json is necessary. PowerShell would otherwise try to interpret @ as a splatting operator rather than passing it to curl as a filename prefix.

You should see responses like these:

# POST /tasks
{"id":1,"title":"Learn Docker","completed":false,"created_at":"2026-07-09T22:21:17.073Z"}

# GET /tasks
[{"id":1,"title":"Learn Docker","completed":false,"created_at":"2026-07-09T22:21:17.073Z"}]

# GET /health
{"status":"ok","timestamp":"2026-07-09T22:11:44.700Z"}

The task hit PostgreSQL in one container and came back through the app. Ctrl+C in the compose terminal stops both.

Automate the Build with GitHub Actions

The image works locally, so it's time to stop doing this by hand.

Step 1: Create a Docker Hub Access Token

Go to hub.docker.com and then Account Settings → Security → New Access Token. Set permission to Read & Write, as read-only breaks the push. The token appears once, so copy it before closing the page.

Security warning: Don't paste this token into a chat, email, or commit. If you expose it by accident, delete it immediately, then make a new one.

Step 2: Add Secrets to Your GitHub Repository

Head to Settings → Secrets and variables → Actions in your repo and add:

• DOCKERHUB_USERNAME — your Docker Hub username

• DOCKERHUB_TOKEN — paste the token here, nowhere else

If you ran into Error: Username and password required, the secrets either aren't saved yet or the names are typed wrong. Both are case-sensitive.

A Node 20 deprecation warning in the logs is normal. It comes from the Docker actions internally, not your code.

Step 3: Create the Workflow File

Create .github/workflows/docker-publish.yml:

name: Build and Push Docker Image

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  IMAGE_NAME: ${{ secrets.DOCKERHUB_USERNAME }}/nodejs-docker-cicd

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to Docker Hub
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-
            type=raw,value=latest,enable={{is_default_branch}}

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          target: production
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

The login step has if: github.event_name != 'pull_request'. This skips authentication on pull requests. PRs from forks don't have access to your secrets, so trying to log in would just fail. The build still runs on PRs to validate your Dockerfile, but the image isn't pushed.

The metadata action generates two tags on every merge to main: latest and a short commit SHA like sha-a1b2c3d. The SHA tag is what makes rollbacks practical. If latest breaks in production, you can pull any previous sha- tag and you're back to a known-good state in seconds.

The cache-from/cache-to: type=gha lines store Docker's layer cache in GitHub Actions' built-in cache. The first run builds everything from scratch. After that, unchanged layers are pulled from cache rather than rebuilt. On a typical Node.js app this brings build time from 2–3 minutes down to under 30 seconds.

Push and Watch it Run

git add .
git commit -m "Add Docker configuration and GitHub Actions workflow"
git push origin main

Go to your repo's Actions tab. You'll see the workflow running in real time. Each step turns green as it completes:

✅ Checkout code
✅ Set up Docker Buildx
✅ Log in to Docker Hub
✅ Extract metadata
✅ Build and push

Green across the board means your image is live on Docker Hub — two tags, latest and a commit SHA like sha-a1b2c3d. Every push to main from here builds and ships automatically.

Deploying the Image

With your image on Docker Hub, you can deploy it to any infrastructure:

Any VPS or server:

docker pull yourusername/nodejs-docker-cicd:latest
docker run -d -p 3000:3000 \
  -e DB_HOST=your-db-host \
  -e DB_NAME=tasksdb \
  -e DB_USER=postgres \
  -e DB_PASSWORD=yourpassword \
  yourusername/nodejs-docker-cicd:latest

Railway — Connect your Docker Hub image in the Railway dashboard and it deploys on the next push.

Fly.io — Run fly launch pointing at your Dockerfile and Fly handles the rest.

Render — Paste your Docker Hub image URL into the Render service settings.

Each push to main runs the workflow. New image goes to Docker Hub, platform picks it up — that's your deployment handled.

Wrapping Up

What started as a local Node.js app now runs in a container. You get the same behavior on any machine, real PostgreSQL in development, and a pipeline that builds and ships to Docker Hub without you doing anything after the push.

The multi-stage build keeps the image lean — dev tools stay out, non-root user, health check baked in. Compose gets the full stack up with one command for anyone who clones the repo. The SHA tag on every GitHub Actions build means rolling back is just a matter of pulling an older tag.

These same patterns (multi-stage builds, Compose for local development, automated image publishing) are used across the industry for production Node.js deployments. Pick up these patterns once and they follow you to every project.

From here, you can extend the pipeline: drop a test step in before the build, or add multi-platform support if you're targeting ARM. Once Docker Compose starts feeling limiting in production, that's usually when Kubernetes enters the picture.

So how do you bundle up your application so that it can run the same way in every local environment? That’s where Docker comes in.

For beginners, Docker isn't always a very easy concept to grasp. But once you get it, I promise that it’s very interesting. So interesting that you’ll want to dockerize every application you lay your hands on.

For this article, a Go application will be our case study. The fundamental concept of containerization as explained here is transferable, so don’t worry too much about how dockerizing applications in another language will look like.

We’ll go through the basics of dockerizing a Go app with just Docker, images and containers, setting up multiple containers in one application with Docker Compose, and the constituent of a Docker Compose file.

By the end of this article, you'll have a basic understanding of what Docker is, what an image or container is, and how to orchestrate multiple, dependent containers with Docker Compose.

What We'll Cover:

1. Prerequisites

2. What is Docker?

3. How to Install Docker

4. What is a Dockerfile?

5. What is Docker Compose?

6. The app Container

7. The database Container

8. The phpMyAdmin Container

9. Running Everything Together

10. Wrapping Up

Prerequisites

You don't need any prior knowledge of Docker to follow this tutorial. This article is written with a beginner POV in mind, so it's okay if the concept is new to you.

In order to be fully engaged and understand the Go coding examples used here, it'll be helpful if you have basic knowledge of Golang. If you already understand how to set up a Go application on your local computer, you're good to go. If not, you can check this article on how to get started coding in Go.

What is Docker?

Imagine that you have a box. In that box, you put your code and everything that it needs to run. That is, the programming language it uses and any other external packages you need to install.

If someone needs your application, you can just hand them the box. You can also hand this box to as many people as you want. They don’t need to install the language or any other thing on their computer because everything they need is already inside the box. So, when they run the application, what they're actually doing is running an instance of that box.

The app is running within the box which is the standard environment. This means for everyone who got the box and “opened it”, the application is going to run the exact same way.

With the help of Docker, apps can run under the same conditions across different systems, and you avoid the problem of “it works on my machine”.

In technical Docker terms, this box is called an image and the running instance is called a container.

An image is a lightweight, standalone, executable package that includes everything needed to run a piece of software. That is, code, runtime, libraries, system tools, and even the operating system.

A container is simply a runnable instance of an image. This represents the execution environment for a specific application.

If all this seems to abstract, don’t worry. We’ll get our hands dirty in a little bit.

How to Install Docker

In order to install Docker, we're going to install Docker Desktop which comes bundled up with the Docker Engine. Docker Destop is a GUI for managing containers, and you'll see how useful it is in subsequent sections.

At the time of writing, I'm using WSL (Windows Sub-system for Linux). If you're doing the same, you'll need to take that into consideration before installing because Docker requires different installation prerequisites and steps for different operating systems.

To install Docker Desktop on WSL,

1. Download and install the windows .exe file

2. Start Docker Desktop from the Start Menu and navigate to settings

3. Select Use WSL 2 based engine from the General tab

4. Click on apply.

That’s it for the WSL installation. If you are running another operating system, the official docs have a list of installation options for you.

What is a Dockerfile?

In order to build your box in the first place, Docker needs to follow a couple of outlined steps. It needs to know the dependencies, the run time, and it also needs to have the source code. All these steps we list in a Dockerfile.

Before we get down to cracking anything, let’s create a working directory and navigate into it.

mkdir go_book_api && cd go_book_api

To intialise the Go module in your application, run the following command:

go mod init go_book_api

This creates a go.mod file to keep track of your project dependencies. In the root of the project, create a cmd directory, and a main.go file in it. This will serve as the entry point of your application. In the main.go file, you can have a simple print statement:

// cmd/main.go
package main

import "fmt"

func main() {
	fmt.Println("Look at me gooo!")
}

Now, go ahead and create a file in the root of your project and call it Dockerfile. This file has no extensions, but your system automatically knows that it's a file for Docker commands.

Go ahead and paste the following in that file, and then we'll go through each of them one by one:

# base image
FROM golang:1.24

# define the working directory
WORKDIR /app

# copy the go.mod and go.sum so that the packages to be installed
# are known in the container. ./ here is the WORKDIR, /app
COPY go.mod ./

# command to install modules
RUN go mod download

# copy source code into working dir
COPY . .

# build
RUN CGO_ENABLED=0 GOOS=linux go build -o /docker-gs-ping ./cmd/main.go

# run the compiled binary when the container starts
CMD ["/docker-gs-ping"]

Most Dockerfiles begin with a base image, which is specified by the FROM keyword. A base image is a foundational template that provides minimal operating system environment, libraries, or dependencies required to build and run an application within a container.

In this case, your base image is golang:1.24 . Your base image could have been an operating system like Linux. In that case. when you ship your code to someone who isn’t running a Linux operating system, they wouldn’t have to worry because they will be running the application in an environment that already has a minimal Linux OS. In the same light, someone who doesn’t have Go installed locally can run your application.

To figure out what base image to use when setting up your Dockerfile, you can always peruse the official Docker Hub repository for published images. For this case, you can check out base images that are officially published by Golang here.

The next step is to define a working directory. Inside your box, you have a filesystem that is almost identical to the ones you’d see on a Linux system. You have folders like /app, /bin , /usr , and /var , and so on. The working directory you've defined in this case is /app, and it's done with the WORKDIR command.

After setting a working directory, you want to copy the go.mod and go.sum file into it, so that Docker knows what dependencies to add into your box.

The COPY command in Docker takes at least two arguments: the source directory(ies), and then the destination directory. In this case, you want to copy go.mod and go.sum into the working directory of your box, /app.

In the box, you'll run a command that downloads and installs all the modules defined in the go.mod file. To run a command in Docker environment, use RUN and then the command, which is go mod download in this case.

The next step is to copy any source code you have into the working directory.

At this point, you have the dependencies and the source code. The last step is to build the Go application into a single executable file which can be run inside your environment (inside the container).

Within the container, you’ll have a compiled binary at /docker-gs-ping, which is as a result of the compilation of the code in your main.go file. The last step is a RUN command that just tells Docker to run the executable binary after building it. It’s a way of saying “once the container starts running, execute this binary file”.

With these steps, Docker will build an image (a box per our analogy) that you can run. To build the image, you can run this command in your terminal:

docker build -t go_book_api .

The docker build command tells Docker to build an image based on the steps in the Dockerfile. -t is the flag for a tag, and this helps you refer to the image later when running the container.

To accompany your tag, you'll provide a name to the image which is go_book_api in this case. The . at the end is important because it tells Docker where the Dockerfile in question is, and the files that you need to copy into your image.

This is what the building looks like in my IDE:

If you check the Images tab on Docker Compose, you'll see that an image is built:

You can host this image on a public image repository platform like Docker Hub, and share it with your friends. They can pull your image, set it up, and run your application even if they don’t have Go installed. All they need to do is get the container running.

If you click on the little play button to the far-right, you can spin up an instance of the image (a container).

You can give a descriptive name to the container (Docker will generate a random one if you don’t), and click on the Run button. Once the container starts running, you're redirected to its log page.

Your container is up and running! You can see that this is a running instance of your application.

What is Docker Compose?

If you were building a simple Go application that needed no external dependencies, the above set-up would be more than sufficient.

In our example here, the application is supposed to be for a book API, so you’d expect that we'd have some service like a database and a database administrator client like phpMyAdmin to visualize or tables.

To set all this up in one file would be a little complicated using just Docker. This is because Docker doesn't allow you to have one base image for Go, another base image for a database, and so on, in one file.

You could use the base image of a small operating system, and then run commands to manually install these other services as dependencies, but this method makes your application hard to maintain and scale. This method isn't advisable because if one dependency crashes, the whole application will collapse instantly.

To remedy this situation, Docker compose allows you to have multiple containers for your application that are connected together. Docker compose handles running the containers in the right order, allows one container to use a folder from another container, or even keep its data in another container – and so on.

Our previous analogy of boxes is the same, except with Docker Compose, we don’t necessarily have only one box anymore:

The point of Docker Compose is to help you orchestrate multiple images needed to run your application. You can think of it as connecting several boxes together.

Following the explanation from before, your application would be running in the Go book api container, the book data we'll create with your application would be stored in the mysql container which is the database, and you can visualize your database with phpMyadmin, which is in the phpMyadmin container.

To see this technically, create a docker-compose.yml file in the root of the project. The name of this file is important, and Docker Compose only accepts filenames such as compose.yml , docker-compose.yml , or docker-compose.yaml. The file extension hints that the commands are written in yaml which is a language mostly used for file configurations.

services:
  app:
    depends_on:
      - database
    build: 
      context: .
    container_name: go_book_api
    hostname: go_book_api
    networks:
      - go_book_api_net
    ports:
      - 8080:8080
    env_file:
      - .env
    
  database:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_PASSWORD: ${DB_PASSWORD}
      MYSQL_USER: ${DB_USER}
    volumes:
      - mysql-go:/var/lib/mysql
    ports:
      - 3356:3306
    networks:
      - go_book_api_net

  phpmyadmin:
    image: phpmyadmin
    restart: always
    ports:
      - 9000:80
    environment:
      PMA_HOST: database
      PMA_ARBITRARY: 1
    depends_on:
      - database
    networks:
      - go_book_api_net

volumes:
  mysql-go:

networks:
  go_book_api_net:
    driver: bridge

At the root level of the docker-compose file, you have services . These are all the containers that are your application needs to run, and in the context of Docker Compose, they're each regarded as a service.

The app Container

 app:
    depends_on:
      - database
    build: 
      context: .
    container_name: go_book_api
    hostname: go_book_api
    networks:
      - go_book_api_net
    ports:
      - 8080:8080
    env_file:
      - .env

The very first container is the app container, which is your Go application. Under the app container, you'll need to define a few parameters that this container also needs to run.

The depends_on attribute controls the start-up and shut-down order of services within a container. This ensures that if container A depends on container B to start, the container B should be started first so that container A can use it. In this case, the database container must be started before the app container. Note that this doesn't mean app will always wait for the database to be ready.

The next attribute which is build tells Docker Compose to build the Docker image from the local project. Since the Dockerfile for your application is in the root of your app, you'll specify the root path with the context attribute as . .

To give a specific name to your container, you'll use container_name. hostname is what other containers will use for communication.

Recall that the point of Docker Compose is to have multiple containers communicating with each other. They do this with the help of networks. So you'll create another attribute, networks, and give it a name, go_book_api_net . To every other container that you want to associate with this app, you're going to specify the same network.

The next attribute is ports . Your application is an API, which means it's running on a backend Go server. To access the API, you'll need to map a local port to a port on the container. You're mapping port 8080 on your computer to port 8080 in the container.

The env_file attribute just tells Docker Compose where to read environment variables from. In this case, you can create a .env file in the root of your project to store important variables that your container will need.

The database Container

  database:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_PASSWORD: ${DB_PASSWORD}
      MYSQL_USER: ${DB_USER}
    volumes:
      - mysql-go:/var/lib/mysql
    ports:
      - 3356:3306
    networks:
      - go_book_api_net

The second container is the database container. Note, that you can give whatever name you choose to your listed services, but giving your containers descriptive names is always a good convention to follow.

For your Go application database, you'll be working with a MySQL database in this case. Your application needs MySQL to run, so you must set it up as one of the services.

Remember that to build a container, you need a base image. Your base image in this case is mysql:8.0 , as you've specified with the image property above. When trying to set up this container, Docker Compose knows to build your database container from this already existing official image.

If you’ve set up a database locally before, you know that configuration is a step you can’t skip. Every database you create needs a user, a password, and the database name. You can set these variables up in the environment property. Instead of hardcoding these values, you can set them up in a .env file, and reference the environmental variables as you've done here.

Database servers usually listen on specific ports for incoming connections, whether the database is running locally or remotely. Just as you specified for your app container, you can set a port for your database and map it to a corresponding port in the container. If you want to access the database locally, you'd do that on port 3356, and all requests are forwarded to port 3306 in the database container.

Once your containers go functional and your application starts running, creating, and storing data in the database, you’ll realise that every time you stop and then restart your containers, you lose the data stored in the database.

To avoid this, you'll need to store your data outside the container. That way, you won't lose the contents of your database every time you stop running your containers.

This is what volumes are for. You can allocate a specific location outside the database container to store all that content. For your volume in this case, the storage location you specified is mysql-go:/var/lib/mysql .

Just as you set the network in your app container above to go_book_api_net, you'll specify the same network for this database container. Since you want the containers to communicate with each other, it makes sense that they're within the same network.

The phpMyAdmin Container

The last container or last service you need (but that is optional) to configure in this case is the phpMyAdmin container. I find it easier having a database client because it lets me easily see the structure and content of my database.

 phpmyadmin:
    image: phpmyadmin
    restart: always
    ports:
      - 9000:80
    environment:
      PMA_HOST: database
      PMA_ARBITRARY: 1
    depends_on:
      - database
    networks:
      - go_book_api_net

The process is almost the same as the previous containers you've configured. You'll start by pulling the official phpmyadmin image from Docker so that your container is built on it.

The restart option here is just so that if you stop and restart the container, phpMyAdmin automatically reloads again.

On the host machine, which is your local environment, you can have access to this service via port 9000 and it maps to port 80 in the container.

As for the environment , PMA_HOST tells phpMyAdmin to connect to a host called database (which is your database container). This works because both containers are on the same network, as you can see in the networks attribute. PMA_ARBITRARY is used so that if you decide to connect to another host (say, you set up a another database in future and still wish to connect via phpMyAdmin), you can do that via the UI.

Your database client depends on the database container, and so you need to specify that in depends_on:

volumes:
  mysql-go:

networks:
  go_book_api_net:
    driver: bridge

The final section of your Docker Compose file is where you declared named values for the volume and network you've used in setting up your containers.

For the volumes, you'll declare a value called mysql-go. To the container where you want to attach this volume, you'll assign a specific storage location. You can see this in use in the database container.

 volumes:
      - mysql-go:/var/lib/mysql

The same concept follows for the network. You have a named network called go_book_api_net that every container within this same network can use. The driver option is used here to specify the network type, and bridge is used for private internal networks.

Running Everything Together

Before Docker Compose, you had one Dockerfile that built a single container for your Go application. With Docker Compose, You’re gonna be building three containers (your application container, the database, and phpMyAdmin), and orchestrating them to work together as one single application.

You can push all this to a platform like GitHub, and someone can clone, start, and run the application without having any of these services (MySQL or PhpMyAdmin) installed locally on their computer. But they do need to have Docker installed.

To build your containers all together, you can use the command docker compose build:

If you check your Docker Compose UI again, we see that a new image has been built, and it corresponds to the app service

To start running the containers, you can use the command docker compose up:

If you navigate to the container tab of Docker Compose, you can see that your containers are up and running:

The main app service, go_book_api, isn’t running because when you run your image, your binary runs and exits almost immediately.

In your main.go, let’s rewrite the code to set up a minimal HTTP handler function that listens on port 8080:

// cmd/main.go
package main

import (
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		_, _ = w.Write([]byte("ok"))
	})

	log.Println("listening on :8080")
	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatal(err)
	}
}

If you’re new to Go, don’t let the code above bother you too much. All it does it set up a health endpoint with an associated handler function that listens on a port (8080 in this case) and prints “ok”.

In your Dockerfile, let’s add a command to execute the created binary when the container starts:

# run the compiled binary when the container starts
CMD ["/docker-gs-ping"]

After adding this, you'll need to rebuild the containers and start them again. You can see that all containers are running now:

If you click on the go_book_api container, you can see that your server is running on port 8080 as configured:

Since your app is running on port 8080 and you have a /health endpoint set up for it, you can actually visit that endpoint in a browser to see the output “ok”.

Also, if you click on the exposed phpmyadmin port, you can access the database client locally on port 9000. Based on the environment variables set up in the .env file, you can log in.

Another interesting thing to look for on Docker desktop is volumes. There is a volumes tab where you can see your configured mysql-go volume.

You can always open these volumes/containers on the docker GUI, go through the files and logs, experiment with putting one container down and seeing how the others respond, and so on.

After this entire setup, what do you notice? You didn’t have to install Go, MySQL, or phpMyAdmin locally. You only used officially published base images to orchestrate a full application. That's the magic of Docker.

Wrapping Up

Docker can be very abstract at the beginning, but understanding the fundamental purpose behind it makes everything much clearer.

In this article, you've learned what Docker is, how to containerize a basic Go application, and how to manage multiple containers with Docker Compose.

If you have trouble wrapping your head around why or how the Dockerfile is set up in the order that it is, my advice is not to get too stuck figuring it out on your own. As a Docker beginner, I realised that it’s easier if you imagine it as creating a recipe. If you try to build an image and it fails, you know there’s a step that you’re skipping.

The official docker documentation has amazing resources if you want to understand Docker further than this tutorial. I encourage you to do so because this article only scratches the surface of the amazing things you can achieve with containerization.

I get it. I held that same opinion for years. Compose was the thing I used to spin up a Postgres database on my laptop, not something I'd trust with a staging environment, let alone a workload that needed GPU access.

Then 2024 and 2025 happened. Docker shipped a set of features that quietly transformed Compose from a developer convenience tool into something that can handle complex deployment scenarios. Profiles let you manage multiple environments from a single file. Watch mode killed the painful rebuild cycle that made container-based development feel sluggish. GPU support opened the door to ML inference workloads. And a bunch of smaller improvements (better health checks, Bake integration, structured logging) filled in the gaps that used to make Compose feel like a toy.

Here's what I'll cover: using Docker Compose profiles to manage multiple environments from one file, setting up watch mode for instant code syncing during development, configuring GPU passthrough for machine learning workloads, implementing proper health checks and startup ordering so your services stop crashing on cold starts, and using Bake to bridge the gap between your local Compose workflow and production image builds. I'll also tell you where Compose still falls short and where you should reach for something else.

Prerequisites

You should be comfortable with Docker basics and have written a compose.yaml file before. You'll need Docker Compose v2 installed. The minimum version depends on which features you want: service_healthy dependency conditions require v2.20.0+, watch mode requires v2.22.0+, and the gpus: shorthand requires v2.30.0+. Run docker compose version to check what you have.

Table of Contents

• Prerequisites

• The Modern Compose File: What's Changed

• How to Use Profiles to Manage Multiple Environments

  • Real-World Profile Patterns I've Used

• How to Use Watch Mode to End the Rebuild Cycle

  • Watch Mode vs. Bind Mounts

• How to Set Up GPU Support for Machine Learning Workloads

  • How to Combine Multi-GPU Workloads with Profiles

• How to Configure Health Checks, Dependencies, and Startup Ordering

• How to Use Bake for Production Image Builds

• What Compose Is Not (An Honest Assessment)

• A Practical Adoption Path

• Wrapping Up

The Modern Compose File: What's Changed

If you haven't looked at a Compose file recently, the first thing you'll notice is that the version field is gone. Docker Compose v2 ignores it entirely, and including it actually triggers a deprecation warning. A modern compose.yaml starts cleanly with your services, no preamble needed.

But the structural changes go deeper than that. Here's what a modern, production-aware Compose file looks like for a typical web application stack:

services:
  api:
    image: ghcr.io/myorg/api:${TAG:-latest}
    env_file: [configs/common.env]
    environment:
      - NODE_ENV=${NODE_ENV:-production}
    ports:
      - "8080:8080"
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: "1.0"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  db:
    image: postgres:16-alpine
    volumes:
      - db-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      retries: 5

volumes:
  db-data:

Look at what's in there: resource limits, health checks with dependency conditions, proper volume management. These aren't nice-to-haves. They're the features that make Compose viable beyond your laptop.

Health checks in particular solve one of Compose's oldest and most annoying pain points: the race condition where your web server starts before the database is actually ready to accept connections. If you've ever added sleep 10 to a startup script and crossed your fingers, you know what I'm talking about.

How to Use Profiles to Manage Multiple Environments

This is the feature that changed my relationship with Compose. Before profiles, managing different environments meant choosing between two painful approaches. Either you maintained multiple Compose files (docker-compose.yml, docker-compose.dev.yml, docker-compose.test.yml, docker-compose.prod.yml) and dealt with the inevitable drift between them. Or you used one big bloated file where you commented out services depending on the context. Both approaches were fragile, and both led to those fun "works on my machine" conversations.

Profiles give you a much cleaner path. You assign services to named groups. Services without a profile always start. Services with a profile only start when you explicitly activate that profile. You can also activate profiles with the COMPOSE_PROFILES environment variable instead of the CLI flag, which is handy for CI (see the official profiles docs for the full syntax).

Here's what that looks like:

services:
  api:
    image: myapp:latest
    # No profiles = always starts

  db:
    image: postgres:16
    # No profiles = always starts

  debug-tools:
    image: busybox
    profiles: [debug]
    # Only starts with --profile debug

  prometheus:
    image: prom/prometheus
    profiles: [monitoring]
    # Only starts with --profile monitoring

  grafana:
    image: grafana/grafana
    profiles: [monitoring]
    depends_on: [prometheus]

Now your team operates with simple, memorable commands:

# Development: just the core stack
docker compose up -d

# Development with observability
docker compose --profile monitoring up -d

# CI: core stack only (no monitoring overhead)
docker compose up -d

# Full stack with debugging
docker compose --profile debug --profile monitoring up

One Compose file. No drift. No guesswork about which override file to pass.

Real-World Profile Patterns I've Used

Four patterns I keep coming back to:

The "infra-only" pattern. This is for developers who run application code natively on their host machine but need infrastructure services like databases, message queues, and caches in containers. You leave infrastructure services without a profile and put application services behind one. Your backend developer runs docker compose up to get Postgres and Redis, then starts the API directly on their host with their favorite debugger attached.

The "mock vs. real" pattern. You put a payments-mock service in the dev profile and a real payments gateway service in the prod profile. Same Compose file, totally different behavior depending on context. This one saved my team from accidentally hitting a live payment API during development more than once.

The "CI optimization" pattern. Heavy services like Selenium browsers and monitoring stacks go behind profiles so your CI pipeline skips them. Your test suite runs faster without that overhead, and you only pull those services in when you actually need end-to-end integration tests.

The "AI/ML workloads" pattern. GPU-dependent services (inference servers, model training containers) go into a gpu profile. Developers without GPUs can still work on the rest of the stack without anything breaking.

One practical tip that's saved me a lot of headaches: document your profiles in the project's README. It sounds obvious, but when a new team member runs docker compose up and wonders why the monitoring dashboard isn't starting, they need a single place to find the answer. A quick table listing each profile and what it includes will save you from answering the same Slack question every onboarding cycle.

How to Use Watch Mode to End the Rebuild Cycle

If profiles solved the environment management problem, watch mode solved the developer experience problem.

You probably know the old workflow for container-based development. It went like this: edit code, run docker compose build, run docker compose up, test your change, find a bug, edit again, rebuild, restart, test. Each iteration costs you thirty seconds to a minute of waiting. Over a full day of active development, you're losing an hour or more just sitting there watching build logs scroll by.

Watch mode (introduced in Compose v2.22.0 and significantly improved in later releases) monitors your local files and automatically takes action when something changes. It supports three synchronization strategies, and picking the right one for each situation is the key to making it work well. The official watch mode docs cover the full spec if you want to dig deeper.

sync copies changed files directly into the running container. This works best for interpreted languages like Python, JavaScript, and Ruby, and for frameworks with hot module reloading like React, Vue, or Next.js. The file lands in the container, the framework picks up the change, and your browser updates. No rebuild, no restart. If you're working with a compiled language like Go, Rust, or Java, sync won't help you since the code needs to be recompiled. Use rebuild for those instead.

rebuild triggers a full image rebuild and container replacement. You want this for dependency changes, like when you update package.json or requirements.txt, or when you modify the Dockerfile itself. In those cases, syncing files isn't enough. You need a fresh image.

sync+restart syncs files into the container, then restarts the main process. This is ideal for configuration file changes like nginx.conf or database configs, where the application needs to reload to pick up the new settings but the image itself is fine.

Here's what a real-world watch configuration looks like for a Node.js application:

services:
  api:
    build: .
    ports: ["3000:3000"]
    command: npx nodemon server.js
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src
          ignore:
            - node_modules/
        - action: rebuild
          path: package.json
        - action: sync+restart
          path: ./config
          target: /app/config

You start it with docker compose up --watch, or you can run docker compose watch as a standalone command if you'd rather keep the file sync events separate from your application logs.

A few things to know before you set this up. Watch mode only works with services that have a local build: context. If you're pulling a prebuilt image from a registry, there's nothing for Compose to sync or rebuild, so watch will ignore that service. Your container also needs basic file utilities (stat, mkdir) installed, and the container USER must have write access to the target path. If you're using a minimal base image like scratch or distroless, the sync action won't work. And if you're on an older Compose version, check which actions are supported: sync+restart and sync+exec were added in later minor releases after the initial v2.22.0 launch.

It's a massive improvement. Edit a source file, save it, and the change is live in under a second for frameworks with hot reload. No context switching to run build commands. No waiting. Just code.

Watch Mode vs. Bind Mounts

A fair question you might be asking: bind mounts have provided a form of live-reload for years. Why does watch mode need to exist?

Bind mounts work, but they come with platform-specific issues that have plagued Docker Desktop for a long time. On macOS and Windows, bind mounts go through a filesystem sharing layer between the host OS and the Linux VM running Docker. This introduces permission quirks, performance problems on large directories (ever watched a node_modules folder choke a bind mount on macOS?), and inconsistent file notification behavior that makes hot reload unreliable.

Watch mode sidesteps these issues by explicitly syncing files at the application level. It's more predictable, works consistently across platforms, and gives you more control over what happens when a file changes.

That said, bind mounts still work well for many use cases, especially if you're on native Linux where the performance overhead doesn't exist. Watch mode is the better choice for teams that have run into cross-platform issues, or for anyone who wants the automatic rebuild and restart triggers that bind mounts can't provide.

How to Set Up GPU Support for Machine Learning Workloads

This is the feature that made me rethink what Compose can do.

Docker has supported GPU passthrough for individual containers for years through the NVIDIA Container Toolkit and the --gpus flag. But configuring GPU access in Compose files used to require clunky runtime declarations that were poorly documented and changed between Compose versions. It was the kind of thing where you'd find a Stack Overflow answer from 2021, try it, and discover it didn't work anymore.

The modern Compose spec handles it cleanly through the deploy.resources.reservations.devices block:

services:
  inference:
    image: myorg/model-server:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

If you're on Compose v2.30.0 or later, there's also a shorter syntax using the gpus: field:

services:
  inference:
    image: myorg/model-server:latest
    gpus:
      - driver: nvidia
        count: 1

Both approaches do the same thing. The deploy.resources syntax works on older Compose versions and gives you more control (like setting device_ids to pin specific GPUs). The gpus: shorthand is cleaner when you just need basic access.

One thing that will trip you up if you skip it: your host machine needs the right GPU drivers and nvidia-container-toolkit installed before any of this works. Run nvidia-smi on the host first. If that command doesn't show your GPUs, Compose won't see them either. For CUDA workloads, use official GPU base images like nvidia/cuda or the PyTorch/TensorFlow GPU images. The Compose GPU access docs walk through the full setup.

That's the whole thing. When you run docker compose up, the inference service gets access to one NVIDIA GPU. You can set count to "all" if you want every available GPU, or use device_ids to assign specific GPUs to specific services.

How to Combine Multi-GPU Workloads with Profiles

Here's where profiles and GPU support work really well together. Consider an ML workload where you need an LLM for text generation, an embedding model for vector search, and a vector database:

services:
  vectordb:
    image: milvus/milvus:latest
    # Runs on CPU, no profile needed

  llm-server:
    image: ollama/ollama:latest
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]
    volumes:
      - model-cache:/root/.ollama

  embedding-server:
    image: myorg/embeddings:latest
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

Developers without GPUs work on the application logic with just docker compose up. The vector database starts, they can write code against its API, and everything runs fine. When it's time to test the full ML pipeline, someone with a multi-GPU workstation runs docker compose --profile gpu up and gets the complete stack with specific GPU assignments.

This pattern has become central to our AIOps platform development. The team building alerting logic doesn't need GPUs. The team training anomaly detection models does. One Compose file serves both teams.

How to Configure Health Checks, Dependencies, and Startup Ordering

One of Compose's most underappreciated improvements is how it handles service dependencies. The depends_on directive now supports conditions that actually mean something (this requires Compose v2.20.0+, see the startup ordering docs for the full picture):

depends_on:
  db:
    condition: service_healthy
  redis:
    condition: service_started

When you combine this with proper health checks, you eliminate the "sleep 10 and hope" pattern that plagues so many Compose setups. Your API service actually waits until PostgreSQL is accepting connections before it tries to start. Not just until the container is running, but until the database process inside it has passed its health check.

One detail that catches people: tune your start_period. Databases like PostgreSQL need time to initialize on first boot, especially if they're running migrations. Without a start_period, the health check starts counting retries immediately and can declare the service unhealthy before it even had a chance to finish starting up. A config like this works well for most database services:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 5s
  timeout: 2s
  retries: 10
  start_period: 30s

The start_period gives the container 30 seconds of grace time where failed health checks don't count against the retry limit.

This might seem like a small detail, but if you've ever worked on a stack with eight or ten interconnected services, you know how much time you can waste debugging cascading failures during cold starts. Proper startup ordering prevents all of that and makes your local environment behave much more like production.

How to Use Bake for Production Image Builds

I mentioned Bake integration earlier, and it's worth its own section because it solves a problem you'll hit as soon as you start using Compose for anything beyond local dev: your development Compose file and your production build process have different needs.

During development, you want fast builds, local caches, and single-platform images. For production, you want tagged images pushed to a registry, multi-platform builds, and build attestations. Trying to cram both into your compose.yaml gets messy fast.

Docker Bake (docker buildx bake) can read your compose.yaml and generate build targets from it, but you can override and extend those targets with a separate docker-bake.hcl file. This keeps your development workflow clean while giving CI the knobs it needs. The Bake documentation covers the full HCL syntax and Compose integration.

Here's a minimal docker-bake.hcl:

group "default" {
  targets = ["api", "worker"]
}

target "api" {
  context    = "api"
  dockerfile = "Dockerfile"
  tags       = ["registry.example.com/team/api:release"]
  platforms  = ["linux/amd64"]
}

target "worker" {
  context    = "worker"
  dockerfile = "Dockerfile"
  tags       = ["registry.example.com/team/worker:release"]
}

Then your CI pipeline runs docker buildx bake to produce release images, while developers keep using docker compose up --build locally. The two workflows share the same Dockerfiles but have separate build configurations where they need them.

The pattern I've landed on: use Compose for local development and CI test environments, use Bake in CI to produce the release images, and push those images into whatever deployment target your team uses (staging server, Kubernetes cluster, edge node). Compose gets you from code to running containers fast. Bake gets you from code to production-ready images with proper tags and attestations.

What Compose Is Not (An Honest Assessment)

I've spent this entire article making the case that Compose has grown up. But I should also tell you where it falls short. I'd rather you hear it from me now than discover it the hard way in production.

Compose is not a container orchestrator. It doesn't schedule work across multiple hosts. It doesn't do automatic failover. It won't give you rolling updates with zero downtime, and it has no concept of service mesh networking. If you need any of those things, you need Kubernetes, Nomad, or Docker Swarm (if you're still using it).

Compose doesn't replace Helm or Kustomize. If you're deploying to Kubernetes, Compose files don't translate directly. Docker offers Compose Bridge to convert Compose files into Kubernetes manifests, but it's still experimental and won't handle complex Kubernetes-specific configurations like custom resource definitions or ingress rules.

Compose doesn't handle secrets well in production. The secrets support exists, but it's limited compared to HashiCorp Vault, AWS Secrets Manager, or Kubernetes secrets. For anything beyond a staging environment, you'll want an external secrets management solution.

The sweet spot for modern Compose is clear: local development, CI/CD testing environments, single-node staging environments, and workloads where a single powerful machine (particularly for GPU work) is the right deployment target. Within that scope, Compose is excellent. Outside of it, you'll hit walls fast.

If you do run Compose in a staging or single-node production setup, a few more things are worth adding that I haven't covered here: restart: unless-stopped on every service so containers come back after a host reboot, a logging driver config so your logs go somewhere searchable instead of disappearing into docker logs, and a backup strategy for your named volumes. These aren't Compose-specific problems, but Compose won't solve them for you either.

A Practical Adoption Path

If you're currently working with a basic Compose setup and want to start using these features, here's the order I'd recommend. Each step is incremental, backward-compatible, and valuable on its own. You don't have to do all of this at once.

Week 1: Add health checks and proper depends_on conditions. This alone will eliminate the most common frustration: services crashing on startup because their dependencies aren't ready yet. Start with your database and your main application service. Once those two are wired up with condition: service_healthy, you'll notice the difference immediately.

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres"]
  interval: 5s
  timeout: 2s
  retries: 10
  start_period: 30s

Week 2: Introduce profiles. Start by putting your monitoring stack behind a monitoring profile and your debug tools behind a debug profile. Then delete whatever extra Compose files you've been maintaining. Having one source of truth instead of four files that are almost-but-not-quite the same makes everything simpler.

Week 3: Set up watch mode for your most-edited service. Pick the service where your developers spend the most time iterating. Get watch mode working there first. Once the team sees the difference (saving a file and seeing the change reflected in under a second) they'll ask for it on everything else.

Week 4: Add resource limits. Define memory and CPU limits for every service. This prevents one runaway container from starving the rest and gives you a realistic preview of how your services behave under production constraints. It's also useful for catching memory leaks early.

deploy:
  resources:
    limits:
      memory: 512M
      cpus: "1.0"

Wrapping Up

Docker Compose in 2026 is not the same tool it was a few years ago. Profiles, watch mode, GPU support, proper dependency management, and Bake integration have turned it into something that can handle real, complex workloads, as long as those workloads fit on a single node.

It's not Kubernetes, and it shouldn't try to be. But for local development, CI pipelines, staging environments, and single-machine GPU workloads, it's become hard to argue against. If you've been dismissing Compose because of what it used to be, the current version deserves a second look.

If you found this useful, you can find me writing about DevOps, containers, and AIOps best practices on my blog.

But let's assume you were assigned on a project which requires at least 10 different services in a running state to run your project. For example, let's say your project requires Java 8, Node 14, MySQL, MongoDB, Ruby on rails, RabbitMQ, and others.

In such case, you have to pull all those images individually from Docker and start all of them in their containers. At some point, one process may depend on another to run. So, you have to order them.

It would be good if it's a one time process. But, not just once – every day, every time you start working on your project – you have to start all these services.

That's a tedious process right?

To overcome this, Docker introduced a concept called Multi Containers (Docker Compose). Before learning about Docker Compose, let's quickly learn how to start a database host in Docker.

In the example part of this tutorial, we'll spin up a NodeJS container and MongoDB container together. Learning about MongoDB at the beginning will give you a good understanding for when we move to Docker Compose.

We'll split this tutorial into 2 sections:

1. How to use docker as a Database host (MongoDB)
2. How Docker Compose works with an example (NodeJS and MongoDB)

How to Use Docker as a Database Host

If you've had experience with backend development, you might have had a chance to handle multiple databases. For example, databases like MySQL/Postgres to handle relational data and Cassandra/MongoDB to handle unstructured data.

Want to know a secret? You can work on back-end development without installing the database on your machine locally. Yes, you can use Docker as a database host. It has all the dependencies by default in the particular image file.

Why Do We Need Docker Database?

Docker helps us maintain consistent versions across platforms and environments. Let's say that there are a group of people working on your team on MongoDB version 5.0. If a new member joins your team, they'll need to set up the same version with the exact configuration manually. What if they install the latest version of MongoDB (6.0)? That will lead to some conflicts. This will be nightmare if it spreads to everyone's else's devices.

To get around this, you can use MongoDB in Docker with a custom configuration and push the MongoDB image to Docker Hub internally. If a new person comes in they can pull the image and start the implementation without any manual configuration.

Let's look at the advantages of using a database in Docker.

1. By using this implementation we can ensure that everyone on a team uses the exact runtimes and configuration without any external resources.
2. It's very easy to set up and we can start/stop the server quickly using Docker desktop

How to Setup MongoDB using Docker

If you're not familiar with Docker Hub, here's a short intro. Docker hub is a platform where you can find and share Docker images that are public or private. It is pretty similar to GitHub / GitLab. In a nutshell, it's a repository for Docker images.

The first step is to pull the official Docker image from Docker Hub.

MongoDB Image in Docker Hub

docker pull mongo:latest

Sample output for pulling Mongo image from Docker Hub

Once you're done pulling the Mongo image, open your Docker Desktop and you'll be able to see it there.

Mongo Image available in Docker desktop

Let's run our MongoDB image using the docker run command.

docker run -d -p 27017:27017 --name mongo-server-local mongo: latest

Sample output to run MongoDB in Docker

We've successfully run the Docker image. Now we can see the container running on Docker Desktop.

Mongo container running in Docker Desktop

So, the MongoDB server is running on your machine. Let's confirm this in the browser. Go to http://localhost:27017 on your browser and you should be able to see the message as shown in the screenshot below:

Sample output "It looks like you are trying to access MongoDB over HTTP on the native driver port" for running MongoDB server using Docker

Interesting right?

We can stop/start the MongoDB server using Docker whenever we need.

Important Note

1. It is not recommended to use Docker as a database for production
2. Do not use Docker database for large scale applications

What is docker-compose?

Let's come back to docker-compose.

Docker Compose is a tool you can use to define and share multi-container applications. This means you can run a project with multiple containers using a single source.

For example, assume you're building a project with NodeJS and MongoDB together. You can create a single image that starts both containers as a service – you don't need to start each separately.

Interesting right? And this solves the problem which I called out at the very beginning of this article.

To achieve this we need to define a docker-compose.yml.

docker-compose.yml file

The compose file is a YML file defining services, networks, and volumes for a Docker container. There are several versions of the compose file format available – 1, 2, 2.x, and 3.x.

Before proceeding further, here's a important note to us from the Docker Compose team.

From the end of June 2023 Compose V1 won’t be supported anymore and will be removed from all Docker Desktop versions.

We are using version 3 in this article.

version: '3'
services:
  app:
    image: node:latest
    container_name: app_main
    restart: always
    command: sh -c "yarn install && yarn start"
    ports:
      - 8000:8000
    working_dir: /app
    volumes:
      - ./:/app
    environment:
      MYSQL_HOST: localhost
      MYSQL_USER: root
      MYSQL_PASSWORD: 
      MYSQL_DB: test
  mongo:
    image: mongo
    container_name: app_mongo
    restart: always
    ports:
      - 27017:27017
    volumes:
      - ~/mongo:/data/db
volumes:
  mongodb:

Let's dismantle the above code and understand it piece by piece:

• version refers to the docker-compose version (Latest 3)
• services defines the services that we need to run
• app is a custom name for one of your containers
• image the image which we have to pull. Here we are using node:latest and mongo.
• container_name is the name for each container
• restart starts/restarts a service container
• port defines the custom port to run the container
• working_dir is the current working directory for the service container
• environment defines the environment variables, such as DB credentials, and so on.
• command is the command to run the service

How to run the multi-container

We need to build our multi-container using docker build.

docker compose build

After successfully building, we can run the containers using the up command.

docker compose up

If you want to run the container in detached mode, just use the -d flag.

docker compose up -d

Sample output to run multiple containers using docker-compose in detached mode

Fine, We are good to go. The containers are up and running. Let's check the container list.

docker compose ps

Sample output to list the running container services

Hurray, we can see that there are two containers running at the same time.

Sample output for running nodejs service using docker

Sample output for running mongodb service using docker

To have a look at the data in your MongoDB, you have to install MongoDB Compass.

Here's the screenshot of it.

MongoDB server view in mongodb compass

Conclusion

In this article, you have learned how Docker Compose works with an example. Using multiple containers you can spin up any type of service such as RabbitMQ or Apache Kafka and run in a single source of service. Hope you enjoyed reading this article.

If you wish to learn more about Docker, subscribe to my article on my site (https://5minslearn.gogosoon.com) which has a consolidated list of all my blogs.

According to the Stack Overflow Developer Survey - 2020, Docker is the #1 most wanted platform, #2 most loved platform, and also the #3 most popular platform.

As in-demand as it may be, getting started can seem a bit intimidating at first. So in this book, we'll be learning everything from the basics to a more intermediate level of containerization. After going through the entire book, you should be able to:

• Containerize (almost) any application
• Upload custom Docker Images to online registries
• Work with multiple containers using Docker Compose

Prerequisites

• Familiarity with the Linux Terminal
• Familiarity with JavaScript (some later projects use JavaScript)

Table of Contents

• Introduction to Containerization and Docker
• How to Install Docker
  • How to Install Docker on macOS
  • How to Install Docker on Windows
  • How to Install Docker on Linux
• Hello World in Docker - Intro to Docker Basics
  • What is a Container?
  • What is a Docker Image?
  • What is a Docker Registry?
  • Docker Architecture Overview
  • The Full Picture
• Docker Container Manipulation Basics
  • How to Run a Container
  • How to Publish a Port
  • How to Use Detached Mode
  • How to List Containers
  • How to Name or Rename a Container
  • How to Stop or Kill a Running Container
  • How to Restart a Container
  • How to Create a Container Without Running
  • How to Remove Dangling Containers
  • How to Run a Container in Interactive Mode
  • How to Execute Commands Inside a Container
  • How to Work With Executable Images
• Docker Image Manipulation Basics
  • How to Create a Docker Image
  • How to Tag Docker Images
  • How to List and Remove Docker Images
  • How to Understand the Many Layers of a Docker Image
  • How to Build NGINX from Source
  • How to Optimize Docker Images
  • Embracing Alpine Linux
  • How to Create Executable Docker Images
  • How to Share Your Docker Images Online
• How to Containerize a JavaScript Application
  • How to Write the Development Dockerfile
  • How to Work With Bind Mounts in Docker
  • How to Work With Anonymous Volumes in Docker
  • How to Perform Multi-Staged Builds in Docker
  • How to Ignore Unnecessary Files
• Network Manipulation Basics in Docker
  • Docker Network Basics
  • How to Create a User-Defined Bridge in Docker
  • How to Attach a Container to a Network in Docker
  • How to Detach Containers from a Network in Docker
  • How to Get Rid of Networks in Docker
• How to Containerize a Multi-Container JavaScript Application
  • How to Run the Database Server
  • How to Work with Named Volumes in Docker
  • How to Access Logs from a Container in Docker
  • How to Create a Network and Attaching the Database Server in Docker
  • How to Write the Dockerfile
  • How to Execute Commands in a Running Container
  • How to Write Management Scripts in Docker
• How to Compose Projects Using Docker-Compose
  • Docker Compose Basics
  • How to Start Services in Docker Compose
  • How to List Services in Docker Compose
  • How to Execute Commands Inside a Running Service in Docker Compose
  • How to Access Logs from a Running Service in Docker Compose
  • How to Stop Services in Docker Compose
  • How to Compose a Full-stack Application in Docker Compose
• Conclusion

Project Code

Code for the example projects can be found in the following repository:

You can find the complete code in the completed branch.

Contributions

This book is completely open-source and quality contributions are more than welcome. You can find the full content in the following repository:

I usually do my changes and updates on the GitBook version of the book first and then publish them on freeCodeCamp. You can find the always updated and often unstable version of the book at the following link:

If you're looking for a frozen but stable version of the book, then freeCodeCamp will be the best place to go:

Whichever version of the book you end up reading though, don't forget to let me know your opinion. Constructive criticism is always welcomed.

Introduction to Containerization and Docker

According to IBM,

Containerization involves encapsulating or packaging up software code and all its dependencies so that it can run uniformly and consistently on any infrastructure.

‌In other words, containerization lets you bundle up your software along with all its dependencies in a self-contained package so that it can be run without going through a troublesome setup process.

‌Let's consider a real life scenario here. Assume you have developed an awesome book management application that can store information regarding all the books you own, and can also serve the purpose of a book lending system for your friends.

‌If you make a list of the dependencies, that list may look as follows:

• Node.js
• Express.js
• SQLite3

Well, theoretically this should be it. But practically there are some other things as well. Turns out Node.js uses a build tool known as node-gyp for building native add-ons. And according to the installation instruction in the official repository, this build tool requires Python 2 or 3 and a proper C/C++ compiler tool-chain.

Taking all these into account, the final list of dependencies is as follows:

• Node.js
• Express.js
• SQLite3
• Python 2 or 3
• C/C++ tool-chain

Installing Python 2 or 3 is pretty straightforward regardless of the platform you're on. Setting up the C/C++ tool-chain is pretty easy on Linux, but on Windows and Mac it's a painful task.

On Windows, the C++ build tools package measures at gigabytes and takes quite some time to install. On a Mac, you can either install the gigantic Xcode application or the much smaller Command Line Tools for Xcode package.

Regardless of the one you install, it still may break on OS updates. In fact, the problem is so prevalent that there are Installation notes for macOS Catalina available on the official repository.

Let's assume that you've gone through all the hassle of setting up the dependencies and have started working on the project. Does that mean you're out of danger now? Of course not.

What if you have a teammate who uses Windows while you're using Linux. Now you have to consider the inconsistencies of how these two different operating systems handle paths. Or the fact that popular technologies like nginx are not well optimized to run on Windows. Some technologies like Redis don't even come pre-built for Windows.

Even if you get through the entire development phase, what if the person responsible for managing the servers follows the wrong deployment procedure?

All these issues can be solved if only you could somehow:

• Develop and run the application inside an isolated environment (known as a container) that matches your final deployment environment.
• Put your application inside a single file (known as an image) along with all its dependencies and necessary deployment configurations.
• And share that image through a central server (known as a registry) that is accessible by anyone with proper authorization.

Your teammates will then be able to download the image from the registry, run the application as it is within an isolated environment free from the platform specific inconsistencies, or even deploy directly on a server, since the image comes with all the proper production configurations.

That is the idea behind containerization: putting your applications inside a self-contained package, making it portable and reproducible across various environments.

Now the question is "What role does Docker play here?"

As I've already explained, containerization is an idea that solves a myriad of problems in software development by putting things into boxes.

This very idea has quite a few implementations. Docker is such an implementation. It's an open-source containerization platform that allows you to containerize your applications, share them using public or private registries, and also to orchestrate them.

Now, Docker is not the only containerization tool on the market, it's just the most popular one. Another containerization engine that I love is called Podman developed by Red Hat. Other tools like Kaniko by Google, rkt by CoreOS are amazing, but they're not ready to be a drop-in replacement for Docker just yet.

Also, if you want a history lesson, you may read the amazing A Brief History of Containers: From the 1970s Till Now which covers most of the major turning points for the technology.

How to Install Docker

Installation of Docker varies greatly depending on the operating system you’re using. But it's universally simple across the board.

Docker runs flawlessly on all three major platforms, Mac, Windows, and Linux. Among the three, the installation process on Mac is the easiest, so we'll start there.

How to Install Docker on macOS

On a mac, all you have to do is navigate to the official download page and click the Download for Mac (stable) button.

You’ll get a regular looking Apple Disk Image file and inside the file, there will be the application. All you have to do is drag the file and drop it in your Applications directory.

You can start Docker by simply double-clicking the application icon. Once the application starts, you'll see the Docker icon appear on your menu-bar.

Now, open up the terminal and execute docker --version and docker-compose --version to ensure the success of the installation.

How to Install Docker on Windows

On Windows, the procedure is almost the same, except there are a few extra steps that you’ll need to go through. The installation steps are as follows:

1. Navigate to this site and follow the instructions for installing WSL2 on Windows 10.
2. Then navigate to the official download page and click the Download for Windows (stable) button.
3. Double-click the downloaded installer and go through the installation with the defaults.

Once the installation is done, start Docker Desktop either from the start menu or your desktop. The docker icon should show up on your taskbar.

Now, open up Ubuntu or whatever distribution you've installed from Microsoft Store. Execute the docker --version and docker-compose --version commands to make sure that the installation was successful.

You can access Docker from your regular Command Prompt or PowerShell as well. It's just that I prefer using WSL2 over any other command line on Windows.

How to Install Docker on Linux

Installing Docker on Linux is a bit of a different process, and depending on the distribution you’re on, it may vary even more. But to be honest, the installation is just as easy (if not easier) as the other two platforms.

The Docker Desktop package on Windows or Mac is a collection of tools like Docker Engine, Docker Compose, Docker Dashboard, Kubernetes and a few other goodies.

On Linux however, you don’t get such a bundle. Instead you install all the necessary tools you need manually. Installation procedures for different distributions are as follows:

• If you’re on Ubuntu, you may follow the Install Docker Engine on Ubuntu section from the official docs.
• For other distributions, installation per distro guides are available on the official docs.
  • Install Docker Engine on Debian
  • Install Docker Engine on Fedora
  • Install Docker Engine on CentOS
• If you’re on a distribution that is not listed in the docs, you may follow the Install Docker Engine from binaries guide instead.
• Regardless of the procedure you follow, you’ll have to go through some Post-installation steps for Linux which are very important.
• Once you’re done with the docker installation, you’ll have to install another tool named Docker Compose. You may follow the Install Docker Compose guide from the official docs.

Once the installation is done, open up the terminal and execute docker --version and docker-compose --version to ensure the success of the installation.

Although Docker performs quite well regardless of the platform you’re on, I prefer Linux over the others. Throughout the book, I’ll be switching between my Ubuntu 20.10 and Fedora 33 workstations.

Another thing that I would like to clarify right from the get go, is that I won't be using any GUI tool for working with Docker throughout the entire book.

I'm aware of the nice GUI tools available for different platforms, but learning the common docker commands is one of the primary goals of this book.

Hello World in Docker – Intro to Docker Basics

Now that you have Docker up and running on your machine, it's time for you to run your first container. Open up the terminal and run the following command:

docker run hello-world

# Unable to find image 'hello-world:latest' locally
# latest: Pulling from library/hello-world
# 0e03bdcc26d7: Pull complete 
# Digest: sha256:4cf9c47f86df71d48364001ede3a4fcd85ae80ce02ebad74156906caff5378bc
# Status: Downloaded newer image for hello-world:latest
# 
# Hello from Docker!
# This message shows that your installation appears to be working correctly.
# 
# To generate this message, Docker took the following steps:
#  1. The Docker client contacted the Docker daemon.
#  2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
#     (amd64)
#  3. The Docker daemon created a new container from that image which runs the
#     executable that produces the output you are currently reading.
#  4. The Docker daemon streamed that output to the Docker client, which sent it
#     to your terminal.
#
# To try something more ambitious, you can run an Ubuntu container with:
#  $ docker run -it ubuntu bash
# 
# Share images, automate workflows, and more with a free Docker ID:
#  https://hub.docker.com/
#
# For more examples and ideas, visit:
#  https://docs.docker.com/get-started/

The hello-world image is an example of minimal containerization with Docker. It has a single program compiled from a hello.c file responsible for printing out the message you're seeing on your terminal.

Now in your terminal, you can use the docker ps -a command to have a look at all the containers that are currently running or have run in the past:

docker ps -a

# CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
# 128ec8ceab71        hello-world         "/hello"            14 seconds ago      Exited (0) 13 seconds ago                      exciting_chebyshev

In the output, a container named exciting_chebyshev was run with the container id of 128ec8ceab71 using the hello-world image. It has Exited (0) 13 seconds ago where the (0) exit code means no error was produced during the runtime of the container.

Now in order to understand what just happened behind the scenes, you'll have to get familiar with the Docker Architecture and three very fundamental concepts of containerization in general, which are as follows:

• Container
• Image
• Registry

I've listed the three concepts in alphabetical order and will begin my explanations with the first one on the list.

What is a Container?

In the world of containerization, there can not be anything more fundamental than the concept of a container.

The official Docker resources site says -

A container is an abstraction at the application layer that packages code and dependencies together. Instead of virtualizing the entire physical machine, containers virtualize the host operating system only.

You may consider containers to be the next generation of virtual machines.

Just like virtual machines, containers are completely isolated environments from the host system as well as from each other. They are also a lot lighter than the traditional virtual machine, so a large number of containers can be run simultaneously without affecting the performance of the host system.‌

Containers and virtual machines are actually different ways of virtualizing your physical hardware. The main difference between these two is the method of virtualization.

Virtual machines are usually created and managed by a program known as a hypervisor, like Oracle VM VirtualBox, VMware Workstation, KVM, Microsoft Hyper-V and so on. This hypervisor program usually sits between the host operating system and the virtual machines to act as a medium of communication.

Each virtual machine comes with its own guest operating system which is just as heavy as the host operating system.

The application running inside a virtual machine communicates with the guest operating system, which talks to the hypervisor, which then in turn talks to the host operating system to allocate necessary resources from the physical infrastructure to the running application.

As you can see, there is a long chain of communication between applications running inside virtual machines and the physical infrastructure. The application running inside the virtual machine may take only a small amount of resources, but the guest operating system adds a noticeable overhead.

Unlike a virtual machine, a container does the job of virtualization in a smarter way. Instead of having a complete guest operating system inside a container, it just utilizes the host operating system via the container runtime while maintaining isolation – just like a traditional virtual machine.

The container runtime, that is Docker, sits between the containers and the host operating system instead of a hypervisor. The containers then communicate with the container runtime which then communicates with the host operating system to get necessary resources from the physical infrastructure.

As a result of eliminating the entire guest operating system layer, containers are much lighter and less resource-hogging than traditional virtual machines.

As a demonstration of the point, look at the following code block:

uname -a
# Linux alpha-centauri 5.8.0-22-generic #23-Ubuntu SMP Fri Oct 9 00:34:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

docker run alpine uname -a
# Linux f08dbbe9199b 5.8.0-22-generic #23-Ubuntu SMP Fri Oct 9 00:34:40 UTC 2020 x86_64 Linux

In the code block above, I have executed the uname -a command on my host operating system to print out the kernel details. Then on the next line I've executed the same command inside a container running Alpine Linux.

As you can see in the output, the container is indeed using the kernel from my host operating system. This goes to prove the point that containers virtualize the host operating system instead of having an operating system of their own.

If you're on a Windows machine, you'll find out that all the containers use the WSL2 kernel. It happens because WSL2 acts as the back-end for Docker on Windows. On macOS the default back-end is a VM running on HyperKit hypervisor.

What is a Docker Image?

Images are multi-layered self-contained files that act as the template for creating containers. They are like a frozen, read-only copy of a container. Images can be exchanged through registries.

In the past, different container engines had different image formats. But later on, the Open Container Initiative (OCI) defined a standard specification for container images which is complied by the major containerization engines out there. This means that an image built with Docker can be used with another runtime like Podman without any additional hassle.

Containers are just images in running state. When you obtain an image from the internet and run a container using that image, you essentially create another temporary writable layer on top of the previous read-only ones.

This concept will become a lot clearer in upcoming sections of this book. But for now, just keep in mind that images are multi-layered read-only files carrying your application in a desired state inside them.

What is a Docker Registry?

You've already learned about two very important pieces of the puzzle, Containers and Images. The final piece is the Registry.

An image registry is a centralized place where you can upload your images and can also download images created by others. Docker Hub is the default public registry for Docker. Another very popular image registry is Quay by Red Hat.

Throughout this book I'll be using Docker Hub as my registry of choice.

You can share any number of public images on Docker Hub for free. People around the world will be able to download them and use them freely. Images that I've uploaded are available on my profile (fhsinchy) page.

Apart from Docker Hub or Quay, you can also create your own image registry for hosting private images. There is also a local registry that runs within your computer that caches images pulled from remote registries.

Docker Architecture Overview

Now that you've become familiar with most of the fundamental concepts regarding containerization and Docker, it's time for you to understand how Docker as a software was designed.

The engine consists of three major components:

1. Docker Daemon: The daemon (dockerd) is a process that keeps running in the background and waits for commands from the client. The daemon is capable of managing various Docker objects.
2. Docker Client: The client (docker) is a command-line interface program mostly responsible for transporting commands issued by users.
3. REST API: The REST API acts as a bridge between the daemon and the client. Any command issued using the client passes through the API to finally reach the daemon.

According to the official docs,

"Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers".

You as a user will usually execute commands using the client component. The client then use the REST API to reach out to the long running daemon and get your work done.

The Full Picture

Okay, enough talking. Now it's time for you to understand how all these pieces of the puzzle you just learned about work in harmony. Before I dive into the explanation of what really happens when you run the docker run hello-world command, let me show you a little diagram I've made:

This image is a slightly modified version of the one found in the official docs. The events that occur when you execute the command are as follows:

1. You execute docker run hello-world command where hello-world is the name of an image.
2. Docker client reaches out to the daemon, tells it to get the hello-world image and run a container from that.
3. Docker daemon looks for the image within your local repository and realizes that it's not there, resulting in the Unable to find image 'hello-world:latest' locally that's printed on your terminal.
4. The daemon then reaches out to the default public registry which is Docker Hub and pulls in the latest copy of the hello-world image, indicated by the latest: Pulling from library/hello-world line in your terminal.
5. Docker daemon then creates a new container from the freshly pulled image.
6. Finally Docker daemon runs the container created using the hello-world image outputting the wall of text on your terminal.

It's the default behavior of Docker daemon to look for images in the hub that are not present locally. But once an image has been fetched, it'll stay in the local cache. So if you execute the command again, you won't see the following lines in the output:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete
Digest: sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9
Status: Downloaded newer image for hello-world:latest

If there is a newer version of the image available on the public registry, the daemon will fetch the image again. That :latest is a tag. Images usually have meaningful tags to indicate versions or builds. You'll learn about this in greater detail later on.

Docker Container Manipulation Basics

In the previous sections, you've learned about the building blocks of Docker and have also run a container using the docker run command.

In this section, you'll be learning about container manipulation in a lot more detail. Container manipulation is one of the most common task you'll be performing every single day, so having a proper understanding of the various commands is crucial.

Keep in mind, though, that this is not an exhaustive list of all the commands you can execute on Docker. I'll be talking only about the most common ones. Anytime you want to learn more about the available commands, just visit the official reference for the Docker command-line.

How to Run a Container

Previously you've used docker run to create and start a container using the hello-world image. The generic syntax for this command is as follows:

docker run <image name>

Although this is a perfectly valid command, there is a better way of dispatching commands to the docker daemon.

Prior to version 1.13, Docker had only the previously mentioned command syntax. Later on, the command-line was restructured to have the following syntax:

docker <object> <command> <options>

In this syntax:

• object indicates the type of Docker object you'll be manipulating. This can be a container, image, network or volume object.
• command indicates the task to be carried out by the daemon, that is the run command.
• options can be any valid parameter that can override the default behavior of the command, like the --publish option for port mapping.

Now, following this syntax, the run command can be written as follows:

docker container run <image name>

The image name can be of any image from an online registry or your local system. As an example, you can try to run a container using the fhsinchy/hello-dock image. This image contains a simple Vue.js application that runs on port 80 inside the container.

To run a container using this image, execute following command on your terminal:

docker container run --publish 8080:80 fhsinchy/hello-dock

# /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
# /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
# /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
# 10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
# 10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
# /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
# /docker-entrypoint.sh: Configuration complete; ready for start up

The command is pretty self-explanatory. The only portion that may require some explanation is the --publish 8080:80 portion which will be explained in the next sub-section.

How to Publish a Port

Containers are isolated environments. Your host system doesn't know anything about what's going on inside a container. Hence, applications running inside a container remain inaccessible from the outside.

To allow access from outside of a container, you must publish the appropriate port inside the container to a port on your local network. The common syntax for the --publish or -p option is as follows:

--publish <host port>:<container port>

When you wrote --publish 8080:80 in the previous sub-section, it meant any request sent to port 8080 of your host system will be forwarded to port 80 inside the container‌.

Now to access the application on your browser, visit http://127.0.0.1:8080.

You can stop the container by simply hitting the ctrl + c key combination while the terminal window is in focus or closing off the terminal window completely.

How to Use Detached Mode

Another very popular option of the run command is the --detach or -d option. In the example above, in order for the container to keep running, you had to keep the terminal window open. Closing the terminal window also stopped the running container.

This is because, by default, containers run in the foreground and attach themselves to the terminal like any other normal program invoked from the terminal.

In order to override this behavior and keep a container running in background, you can include the --detach option with the run command as follows:

docker container run --detach --publish 8080:80 fhsinchy/hello-dock

# 9f21cb77705810797c4b847dbd330d9c732ffddba14fb435470567a7a3f46cdc

Unlike the previous example, you won't get a wall of text thrown at you this time. Instead what you'll get is the ID of the newly created container.

The order of the options you provide doesn't really matter. If you put the --publish option before the --detach option, it'll work just the same. One thing that you have to keep in mind in case of the run command is that the image name must come last. If you put anything after the image name then that'll be passed as an argument to the container entry-point (explained in the Executing Commands Inside a Container sub-section) and may result in unexpected situations.

How to List Containers

The container ls command can be used to list out containers that are currently running. To do so execute following command:

docker container ls

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                  NAMES
# 9f21cb777058        fhsinchy/hello-dock   "/docker-entrypoint.…"   5 seconds ago       Up 5 seconds        0.0.0.0:8080->80/tcp   gifted_sammet

A container named gifted_sammet is running. It was created 5 seconds ago and the status is Up 5 seconds, which indicates that the container has been running fine since its creation.

The CONTAINER ID is 9f21cb777058 which is the first 12 characters of the full container ID. The full container ID is 9f21cb77705810797c4b847dbd330d9c732ffddba14fb435470567a7a3f46cdc which is 64 characters long. This full container ID was printed as the output of the docker container run command in the previous section.

Listed under the PORTS column, port 8080 from your local network is pointing towards port 80 inside the container. The name gifted_sammet is generated by Docker and can be something completely different in your computer.

The container ls command only lists the containers that are currently running on your system. In order to list out the containers that have run in the past you can use the --all or -a option.

docker container ls --all

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                     PORTS                  NAMES
# 9f21cb777058        fhsinchy/hello-dock   "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes               0.0.0.0:8080->80/tcp   gifted_sammet
# 6cf52771dde1        fhsinchy/hello-dock   "/docker-entrypoint.…"   3 minutes ago       Exited (0) 3 minutes ago                          reverent_torvalds
# 128ec8ceab71        hello-world           "/hello"                 4 minutes ago       Exited (0) 4 minutes ago                          exciting_chebyshev

As you can see, the second container in the list reverent_torvalds was created earlier and has exited with the status code 0, which indicates that no error was produced during the runtime of the container.

How to Name or Rename a Container

By default, every container has two identifiers. They are as follows:

• CONTAINER ID - a random 64 character-long string.
• NAME - combination of two random words, joined with an underscore.

Referring to a container based on these two random identifiers is kind of inconvenient. It would be great if the containers could be referred to using a name defined by you.

Naming a container can be achieved using the --name option. To run another container using the fhsinchy/hello-dock image with the name hello-dock-container you can execute the following command:

docker container run --detach --publish 8888:80 --name hello-dock-container fhsinchy/hello-dock

# b1db06e400c4c5e81a93a64d30acc1bf821bed63af36cab5cdb95d25e114f5fb

The 8080 port on local network is occupied by the gifted_sammet container (the container created in the previous sub-section). That's why you'll have to use a different port number, like 8888. Now to verify, run the container ls command:

docker container ls

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                  NAMES
# b1db06e400c4        fhsinchy/hello-dock   "/docker-entrypoint.…"   28 seconds ago      Up 26 seconds       0.0.0.0:8888->80/tcp   hello-dock-container
# 9f21cb777058        fhsinchy/hello-dock   "/docker-entrypoint.…"   4 minutes ago       Up 4 minutes        0.0.0.0:8080->80/tcp   gifted_sammet

A new container with the name of hello-dock-container has been started.

You can even rename old containers using the container rename command. Syntax for the command is as follows:

docker container rename <container identifier> <new name>

To rename the gifted_sammet container to hello-dock-container-2, execute following command:

docker container rename gifted_sammet hello-dock-container-2

The command doesn't yield any output but you can verify that the changes have taken place using the container ls command. The rename command works for containers both in running state and stopped state.

How to Stop or Kill a Running Container

Containers running in the foreground can be stopped by simply closing the terminal window or hitting ctrl + c. Containers running in the background, however, can not be stopped in the same way.

There are two commands that deal with this task. The first one is the container stop command. Generic syntax for the command is as follows:

docker container stop <container identifier>

Where container identifier can either be the id or the name of the container.

I hope that you remember the container you started in the previous section. It's still running in the background. Get the identifier for that container using docker container ls (I'll be using hello-dock-container container for this demo). Now execute the following command to stop the container:

docker container stop hello-dock-container

# hello-dock-container

If you use the name as identifier, you'll get the name thrown back to you as output. The stop command shuts down a container gracefully by sending a SIGTERM signal. If the container doesn't stop within a certain period, a SIGKILL signal is sent which shuts down the container immediately.

In cases where you want to send a SIGKILL signal instead of a SIGTERM signal, you may use the container kill command instead. The container kill command follows the same syntax as the stop command.

docker container kill hello-dock-container-2

# hello-dock-container-2

How to Restart a Container

When I say restart I mean two scenarios specifically. They are as follows:

• Restarting a container that has been previously stopped or killed.
• Rebooting a running container.

As you've already learned from a previous sub-section, stopped containers remain in your system. If you want you can restart them. The container start command can be used to start any stopped or killed container. The syntax of the command is as follows:

docker container start <container identifier>

You can get the list of all containers by executing the container ls --all command. Then look for the containers with Exited status.

docker container ls --all

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                        PORTS               NAMES
# b1db06e400c4        fhsinchy/hello-dock   "/docker-entrypoint.…"   3 minutes ago       Exited (0) 47 seconds ago                         hello-dock-container
# 9f21cb777058        fhsinchy/hello-dock   "/docker-entrypoint.…"   7 minutes ago       Exited (137) 17 seconds ago                       hello-dock-container-2
# 6cf52771dde1        fhsinchy/hello-dock   "/docker-entrypoint.…"   7 minutes ago       Exited (0) 7 minutes ago                          reverent_torvalds
# 128ec8ceab71        hello-world           "/hello"                 9 minutes ago       Exited (0) 9 minutes ago                          exciting_chebyshev

Now to restart the hello-dock-container container, you may execute the following command:

docker container start hello-dock-container

# hello-dock-container

Now you can ensure that the container is running by looking at the list of running containers using the container ls command.

The container start command starts any container in detached mode by default and retains any port configurations made previously. So if you visit http://127.0.0.1:8080 now, you should be able to access the hello-dock application just like before.

Now, in scenarios where you would like to reboot a running container you may use the container restart command. The container restart command follows the exact syntax as the container start command.

docker container restart hello-dock-container-2

# hello-dock-container-2

The main difference between the two commands is that the container restart command attempts to stop the target container and then starts it back up again, whereas the start command just starts an already stopped container.

In case of a stopped container, both commands are exactly the same. But in case of a running container, you must use the container restart command.

How to Create a Container Without Running

So far in this section, you've started containers using the container run command which is in reality a combination of two separate commands. These commands are as follows:

• container create command creates a container from a given image.
• container start command starts a container that has been already created.

Now, to perform the demonstration shown in the "How to Run a Container" section using these two commands, you can do something like the following:

docker container create --publish 8080:80 fhsinchy/hello-dock

# 2e7ef5098bab92f4536eb9a372d9b99ed852a9a816c341127399f51a6d053856

docker container ls --all

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
# 2e7ef5098bab        fhsinchy/hello-dock   "/docker-entrypoint.…"   30 seconds ago      Created                                 hello-dock

Evident by the output of the container ls --all command, a container with the name of hello-dock has been created using the fhsinchy/hello-dock image. The STATUS of the container is Created at the moment, and, given that it's not running, it won't be listed without the use of the --all option.

Once the container has been created, it can be started using the container start command.

docker container start hello-dock

# hello-dock

docker container ls

# CONTAINER ID        IMAGE                 COMMAND                  CREATED              STATUS              PORTS                  NAMES
# 2e7ef5098bab        fhsinchy/hello-dock   "/docker-entrypoint.…"   About a minute ago   Up 29 seconds       0.0.0.0:8080->80/tcp   hello-dock

The container STATUS has changed from Created to Up 29 seconds which indicates that the container is now in running state. The port configuration has also shown up in the PORTS column which was previously empty.‌

Although you can get away with the container run command for the majority of the scenarios, there will be some situations later on in the book that require you to use this container create command.

How to Remove Dangling Containers

As you've already seen, containers that have been stopped or killed remain in the system. These dangling containers can take up space or can conflict with newer containers.

In order to remove a stopped container you can use the container rm command. The generic syntax is as follows:

docker container rm <container identifier>

To find out which containers are not running, use the container ls --all command and look for containers with Exited status.

docker container ls --all

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                      PORTS                  NAMES
# b1db06e400c4        fhsinchy/hello-dock   "/docker-entrypoint.…"   6 minutes ago       Up About a minute           0.0.0.0:8888->80/tcp   hello-dock-container
# 9f21cb777058        fhsinchy/hello-dock   "/docker-entrypoint.…"   10 minutes ago      Up About a minute           0.0.0.0:8080->80/tcp   hello-dock-container-2
# 6cf52771dde1        fhsinchy/hello-dock   "/docker-entrypoint.…"   10 minutes ago      Exited (0) 10 minutes ago                          reverent_torvalds
# 128ec8ceab71        hello-world           "/hello"                 12 minutes ago      Exited (0) 12 minutes ago                          exciting_chebyshev

As can be seen in the output, the containers with ID 6cf52771dde1 and 128ec8ceab71 are not running. To remove the 6cf52771dde1 you can execute the following command:

docker container rm 6cf52771dde1

# 6cf52771dde1

You can check if the container was deleted or not by using the container ls command. You can also remove multiple containers at once by passing their identifiers one after another separated by spaces.

Or, instead of removing individual containers, if you want to remove all dangling containers at one go, you can use the container prune command.

You can check the container list using the container ls --all command to make sure that the dangling containers have been removed:

docker container ls --all

# CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                  NAMES
# b1db06e400c4        fhsinchy/hello-dock   "/docker-entrypoint.…"   8 minutes ago       Up 3 minutes        0.0.0.0:8888->80/tcp   hello-dock-container
# 9f21cb777058        fhsinchy/hello-dock   "/docker-entrypoint.…"   12 minutes ago      Up 3 minutes        0.0.0.0:8080->80/tcp   hello-dock-container-2

If you are following the book exactly as written so far, you should only see the hello-dock-container and hello-dock-container-2 in the list. I would suggest stopping and removing both containers before going on to the next section.

There is also the --rm option for the container run and container start commands which indicates that you want the containers removed as soon as they're stopped. To start another hello-dock container with the --rm option, execute the following command:

docker container run --rm --detach --publish 8888:80 --name hello-dock-volatile fhsinchy/hello-dock

# 0d74e14091dc6262732bee226d95702c21894678efb4043663f7911c53fb79f3

You can use the container ls command to verify that the container is running:

docker container ls

# CONTAINER ID   IMAGE                 COMMAND                  CREATED              STATUS              PORTS                  NAMES
# 0d74e14091dc   fhsinchy/hello-dock   "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:8888->80/tcp   hello-dock-volatile

Now if you stop the container and then check again with the container ls --all command:

docker container stop hello-dock-volatile

# hello-dock-volatile

docker container ls --all

# CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

The container has been removed automatically. From now on I'll use the --rm option for most of the containers. I'll explicitly mention where it's not needed.

How to Run a Container in Interactive Mode

So far you've only run containers created from either the hello-world image or the fhsinchy/hello-dock image. These images are made for executing simple programs that are not interactive.

Well, all images are not that simple. Images can encapsulate an entire Linux distribution inside them.

Popular distributions such as Ubuntu, Fedora, and Debian all have official Docker images available in the hub. Programming languages such as python, php, go or run-times like node and deno all have their official images.

These images do not just run some pre-configured program. These are instead configured to run a shell by default. In case of the operating system images it can be something like sh or bash and in case of the programming languages or run-times, it is usually their default language shell.

As you may have already learned from your previous experiences with computers, shells are interactive programs. An image configured to run such a program is an interactive image. These images require a special -it option to be passed in the container run command.

As an example, if you run a container using the ubuntu image by executing docker container run ubuntu you'll see nothing happens. But if you execute the same command with the -it option, you should land directly on bash inside the Ubuntu container.

docker container run --rm -it ubuntu

# root@dbb1f56b9563:/# cat /etc/os-release
# NAME="Ubuntu"
# VERSION="20.04.1 LTS (Focal Fossa)"
# ID=ubuntu
# ID_LIKE=debian
# PRETTY_NAME="Ubuntu 20.04.1 LTS"
# VERSION_ID="20.04"
# HOME_URL="https://www.ubuntu.com/"
# SUPPORT_URL="https://help.ubuntu.com/"
# BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
# PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
# VERSION_CODENAME=focal
# UBUNTU_CODENAME=focal

As you can see from the output of the cat /etc/os-release command, I am indeed interacting with the bash running inside the Ubuntu container.

The -it option sets the stage for you to interact with any interactive program inside a container. This option is actually two separate options mashed together.

• The -i or --interactive option connects you to the input stream of the container, so that you can send inputs to bash.
• The -t or --tty option makes sure that you get some good formatting and a native terminal-like experience by allocating a pseudo-tty.

You need to use the -it option whenever you want to run a container in interactive mode. Another example can be running the node image as follows:

docker container run -it node

# Welcome to Node.js v15.0.0.
# Type ".help" for more information.
# > ['farhan', 'hasin', 'chowdhury'].map(name => name.toUpperCase())
# [ 'FARHAN', 'HASIN', 'CHOWDHURY' ]

Any valid JavaScript code can be executed in the node shell. Instead of writing -it you can be more verbose by writing --interactive --tty separately.

How to Execute Commands Inside a Container

In the Hello World in Docker section of this book, you've seen me executing a command inside an Alpine Linux container. It went something like this:

docker run alpine uname -a
# Linux f08dbbe9199b 5.8.0-22-generic #23-Ubuntu SMP Fri Oct 9 00:34:40 UTC 2020 x86_64 Linux

In this command, I've executed the uname -a command inside an Alpine Linux container. Scenarios like this (where all you want to do is to execute a certain command inside a certain container) are pretty common.

Assume that you want encode a string using the base64 program. This is something that's available in almost any Linux or Unix based operating system (but not on Windows).

In this situation you can quickly spin up a container using images like busybox and let it do the job.

The generic syntax for encoding a string using base64 is as follows:

echo -n my-secret | base64

# bXktc2VjcmV0

And the generic syntax for passing a command to a container that is not running is as follows:

docker container run <image name> <command>

To perform the base64 encoding using the busybox image, you can execute the following command:

docker container run --rm busybox sh -c "echo -n my-secret | base64

# bXktc2VjcmV0

What happens here is that, in a container run command, whatever you pass after the image name gets passed to the default entry point of the image.

An entry point is like a gateway to the image. Most of the images except the executable images (explained in the Working With Executable Images sub-section) use shell or sh as the default entry-point. So any valid shell command can be passed to them as arguments.

How to Work With Executable Images

In the previous section, I briefly mentioned executable images. These images are designed to behave like executable programs.

Take for example my rmbyext project. This is a simple Python script capable of recursively deleting files of given extensions. To learn more about the project, you can checkout the repository:

If you have both Git and Python installed, you can install this script by executing the following command:

pip install git+https://github.com/fhsinchy/rmbyext.git#egg=rmbyext

Assuming Python has been set up properly on your system, the script should be available anywhere through the terminal. The generic syntax for using this script is as follows:

rmbyext <file extension>

To test it out, open up your terminal inside an empty directory and create some files in it with different extensions. You can use the touch command to do so. Now, I have a directory on my computer with the following files:

touch a.pdf b.pdf c.txt d.pdf e.txt

ls

# a.pdf  b.pdf  c.txt  d.pdf  e.txt

To delete all the pdf files from this directory, you can execute the following command:

rmbyext pdf

# Removing: PDF
# b.pdf
# a.pdf
# d.pdf

An executable image for this program should be able to take extensions of files as arguments and delete them just like the rmbyext program did.

The fhsinchy/rmbyext image behaves in a similar manner. This image contains a copy of the rmbyext script and is configured to run the script on a directory /zone inside the container.

Now the problem is that containers are isolated from your local system, so the rmbyext program running inside the container doesn't have any access to your local file system. So, if somehow you can map the local directory containing the pdf files to the /zone directory inside the container, the files should be accessible to the container.

One way to grant a container direct access to your local file system is by using bind mounts.

A bind mount lets you form a two way data binding between the content of a local file system directory (source) and another directory inside a container (destination). This way any changes made in the destination directory will take effect on the source directory and vise versa.

Let's see a bind mount in action. To delete files using this image instead of the program itself, you can execute the following command:

docker container run --rm -v $(pwd):/zone fhsinchy/rmbyext pdf

# Removing: PDF
# b.pdf
# a.pdf
# d.pdf

As you may have already guessed by seeing the -v $(pwd):/zone part in the command, the -v or --volume option is used for creating a bind mount for a container. This option can take three fields separated by colons (:). The generic syntax for the option is as follows:

--volume <local file system directory absolute path>:<container file system directory absolute path>:<read write access>

The third field is optional but you must pass the absolute path of your local directory and the absolute path of the directory inside the container.

The source directory in my case is /home/fhsinchy/the-zone. Given that my terminal is opened inside the directory, $(pwd) will be replaced with /home/fhsinchy/the-zone which contains the previously mentioned .pdf and .txt files.

You can learn more about command substitution here if you want to.

The --volume or -v option is valid for the container run as well as the container create commands. We'll explore volumes in greater detail in the upcoming sections so don't worry if you didn't understand them very well here.

The difference between a regular image and an executable one is that the entry-point for an executable image is set to a custom program instead of sh, in this case the rmbyext program. And as you've learned in the previous sub-section, anything you write after the image name in a container run command gets passed to the entry-point of the image.

So in the end the docker container run --rm -v $(pwd):/zone fhsinchy/rmbyext pdf command translates to rmbyext pdf inside the container. Executable images are not that common in the wild but can be very useful in certain cases.

Docker Image Manipulation Basics

Now that you have a solid understanding of how to run containers using publicly available images, it's time for you to learn about creating your very own images.

In this section, you'll learn the fundamentals of creating images, running containers using them, and sharing them online.

I would suggest you to install Visual Studio Code with the official Docker Extension from the marketplace. This will greatly help your development experience.

How to Create a Docker Image

As I've already explained in the Hello World in Docker section, images are multi-layered self-contained files that act as the template for creating Docker containers. They are like a frozen, read-only copy of a container.

In order to create an image using one of your programs you must have a clear vision of what you want from the image. Take the official nginx image, for example. You can start a container using this image simply by executing the following command:

docker container run --rm --detach --name default-nginx --publish 8080:80 nginx

# b379ecd5b6b9ae27c144e4fa12bdc5d0635543666f75c14039eea8d5f38e3f56

docker container ls

# CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
# b379ecd5b6b9        nginx               "/docker-entrypoint.…"   8 seconds ago       Up 8 seconds        0.0.0.0:8080->80/tcp   default-nginx

Now, if you visit http://127.0.0.1:8080 in the browser, you'll see a default response page.

That's all nice and good, but what if you want to make a custom NGINX image which functions exactly like the official one, but that's built by you? That's a completely valid scenario to be honest. In fact, let's do that.‌

In order to make a custom NGINX image, you must have a clear picture of what the final state of the image will be. In my opinion the image should be as follows:

• The image should have NGINX pre-installed which can be done using a package manager or can be built from source.
• The image should start NGINX automatically upon running.

That's simple. If you've cloned the project repository linked in this book, go inside the project root and look for a directory named custom-nginx in there.

Now, create a new file named Dockerfile inside that directory. A Dockerfile is a collection of instructions that, once processed by the daemon, results in an image. Content for the Dockerfile is as follows:

FROM ubuntu:latest

EXPOSE 80

RUN apt-get update && \
    apt-get install nginx -y && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

CMD ["nginx", "-g", "daemon off;"]

Images are multi-layered files and in this file, each line (known as instructions) that you've written creates a layer for your image.

• Every valid Dockerfile starts with a FROM instruction. This instruction sets the base image for your resultant image. By setting ubuntu:latest as the base image here, you get all the goodness of Ubuntu already available in your custom image, so you can use things like the apt-get command for easy package installation.
• The EXPOSE instruction is used to indicate the port that needs to be published. Using this instruction doesn't mean that you won't need to --publish the port. You'll still need to use the --publish option explicitly. This EXPOSE instruction works like a documentation for someone who's trying to run a container using your image. It also has some other uses that I won't be discussing here.
• The RUN instruction in a Dockerfile executes a command inside the container shell. The apt-get update && apt-get install nginx -y command checks for updated package versions and installs NGINX. The apt-get clean && rm -rf /var/lib/apt/lists/* command is used for clearing the package cache because you don't want any unnecessary baggage in your image. These two commands are simple Ubuntu stuff, nothing fancy. The RUN instructions here are written in shell form. These can also be written in exec form. You can consult the official reference for more information.
• Finally the CMD instruction sets the default command for your image. This instruction is written in exec form here comprising of three separate parts. Here, nginx refers to the NGINX executable. The -g and daemon off are options for NGINX. Running NGINX as a single process inside containers is considered a best practice hence the usage of this option. The CMD instruction can also be written in shell form. You can consult the official reference for more information.

Now that you have a valid Dockerfile you can build an image out of it. Just like the container related commands, the image related commands can be issued using the following syntax:

docker image <command> <options>

To build an image using the Dockerfile you just wrote, open up your terminal inside the custom-nginx directory and execute the following command:

docker image build .

# Sending build context to Docker daemon  3.584kB
# Step 1/4 : FROM ubuntu:latest
#  ---> d70eaf7277ea
# Step 2/4 : EXPOSE 80
#  ---> Running in 9eae86582ec7
# Removing intermediate container 9eae86582ec7
#  ---> 8235bd799a56
# Step 3/4 : RUN apt-get update &&     apt-get install nginx -y &&     apt-get clean && rm -rf /var/lib/apt/lists/*
#  ---> Running in a44725cbb3fa
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container a44725cbb3fa
#  ---> 3066bd20292d
# Step 4/4 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in 4792e4691660
# Removing intermediate container 4792e4691660
#  ---> 3199372aa3fc
# Successfully built 3199372aa3fc

To perform an image build, the daemon needs two very specific pieces of information. These are the name of the Dockerfile and the build context. In the command issued above:

• docker image build is the command for building the image. The daemon finds any file named Dockerfile within the context.
• The . at the end sets the context for this build. The context means the directory accessible by the daemon during the build process.

Now to run a container using this image, you can use the container run command coupled with the image ID that you received as the result of the build process. In my case the id is 3199372aa3fc evident by the Successfully built 3199372aa3fc line in the previous code block.

docker container run --rm --detach --name custom-nginx-packaged --publish 8080:80 3199372aa3fc

# ec09d4e1f70c903c3b954c8d7958421cdd1ae3d079b57f929e44131fbf8069a0

docker container ls

# CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
# ec09d4e1f70c        3199372aa3fc        "nginx -g 'daemon of…"   23 seconds ago      Up 22 seconds       0.0.0.0:8080->80/tcp   custom-nginx-packaged

To verify, visit http://127.0.0.1:8080 and you should see the default response page.

How to Tag Docker Images

Just like containers, you can assign custom identifiers to your images instead of relying on the randomly generated ID. In case of an image, it's called tagging instead of naming. The --tag or -t option is used in such cases.

Generic syntax for the option is as follows:

--tag <image repository>:<image tag>

The repository is usually known as the image name and the tag indicates a certain build or version.

Take the official mysql image, for example. If you want to run a container using a specific version of MySQL, like 5.7, you can execute docker container run mysql:5.7 where mysql is the image repository and 5.7 is the tag.

In order to tag your custom NGINX image with custom-nginx:packaged you can execute the following command:

docker image build --tag custom-nginx:packaged .

# Sending build context to Docker daemon  1.055MB
# Step 1/4 : FROM ubuntu:latest
#  ---> f63181f19b2f
# Step 2/4 : EXPOSE 80
#  ---> Running in 53ab370b9efc
# Removing intermediate container 53ab370b9efc
#  ---> 6d6460a74447
# Step 3/4 : RUN apt-get update &&     apt-get install nginx -y &&     apt-get clean && rm -rf /var/lib/apt/lists/*
#  ---> Running in b4951b6b48bb
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container b4951b6b48bb
#  ---> fdc6cdd8925a
# Step 4/4 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in 3bdbd2af4f0e
# Removing intermediate container 3bdbd2af4f0e
#  ---> f8837621b99d
# Successfully built f8837621b99d
# Successfully tagged custom-nginx:packaged

Nothing will change except the fact that you can now refer to your image as custom-nginx:packaged instead of some long random string.

In cases where you forgot to tag an image during build time, or maybe you want to change the tag, you can use the image tag command to do that:

docker image tag <image id> <image repository>:<image tag>

## or ##

docker image tag <image repository>:<image tag> <new image repository>:<new image tag>

How to List and Remove Docker Images

Just like the container ls command, you can use the image ls command to list all the images in your local system:

docker image ls

# REPOSITORY     TAG        IMAGE ID       CREATED         SIZE
# <none>         <none>     3199372aa3fc   7 seconds ago   132MB
# custom-nginx   packaged   f8837621b99d   4 minutes ago   132MB

Images listed here can be deleted using the image rm command. The generic syntax is as follows:

docker image rm <image identifier>

The identifier can be the image ID or image repository. If you use the repository, you'll have to identify the tag as well. To delete the custom-nginx:packaged image, you may execute the following command:

docker image rm custom-nginx:packaged

# Untagged: custom-nginx:packaged
# Deleted: sha256:f8837621b99d3388a9e78d9ce49fbb773017f770eea80470fb85e0052beae242
# Deleted: sha256:fdc6cdd8925ac25b9e0ed1c8539f96ad89ba1b21793d061e2349b62dd517dadf
# Deleted: sha256:c20e4aa46615fe512a4133089a5cd66f9b7da76366c96548790d5bf865bd49c4
# Deleted: sha256:6d6460a744475a357a2b631a4098aa1862d04510f3625feb316358536fcd8641

You can also use the image prune command to cleanup all un-tagged dangling images as follows:

docker image prune --force

# Deleted Images:
# deleted: sha256:ba9558bdf2beda81b9acc652ce4931a85f0fc7f69dbc91b4efc4561ef7378aff
# deleted: sha256:ad9cc3ff27f0d192f8fa5fadebf813537e02e6ad472f6536847c4de183c02c81
# deleted: sha256:f1e9b82068d43c1bb04ff3e4f0085b9f8903a12b27196df7f1145aa9296c85e7
# deleted: sha256:ec16024aa036172544908ec4e5f842627d04ef99ee9b8d9aaa26b9c2a4b52baa

# Total reclaimed space: 59.19MB

The --force or -f option skips any confirmation questions. You can also use the --all or -a option to remove all cached images in your local registry.

How to Understand the Many Layers of a Docker Image

From the very beginning of this book, I've been saying that images are multi-layered files. In this sub-section I'll demonstrate the various layers of an image and how they play an important role in the build process of that image.

For this demonstration, I'll be using the custom-nginx:packaged image from the previous sub-section.

To visualize the many layers of an image, you can use the image history command. The various layers of the custom-nginx:packaged image can be visualized as follows:

docker image history custom-nginx:packaged

# IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
# 7f16387f7307        5 minutes ago       /bin/sh -c #(nop)  CMD ["nginx" "-g" "daemon…   0B                             
# 587c805fe8df        5 minutes ago       /bin/sh -c apt-get update &&     apt-get ins…   60MB                
# 6fe4e51e35c1        6 minutes ago       /bin/sh -c #(nop)  EXPOSE 80                    0B                  
# d70eaf7277ea        17 hours ago        /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
# <missing>           17 hours ago        /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B                  
# <missing>           17 hours ago        /bin/sh -c [ -z "$(apt-get indextargets)" ]     0B                  
# <missing>           17 hours ago        /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   811B                
# <missing>           17 hours ago        /bin/sh -c #(nop) ADD file:435d9776fdd3a1834…   72.9MB

There are eight layers of this image. The upper most layer is the latest one and as you go down the layers get older. The upper most layer is the one that you usually use for running containers.

Now, let's have a closer look at the images beginning from image d70eaf7277ea down to 7f16387f7307. I'll ignore the bottom four layers where the IMAGE is <missing> as they are not of our concern.

• d70eaf7277ea was created by /bin/sh -c #(nop) CMD ["/bin/bash"] which indicates that the default shell inside Ubuntu has been loaded successfully.
• 6fe4e51e35c1 was created by /bin/sh -c #(nop) EXPOSE 80 which was the second instruction in your code.
• 587c805fe8df was created by /bin/sh -c apt-get update && apt-get install nginx -y && apt-get clean && rm -rf /var/lib/apt/lists/* which was the third instruction in your code. You can also see that this image has a size of 60MB given all necessary packages were installed during the execution of this instruction.
• Finally the upper most layer 7f16387f7307 was created by /bin/sh -c #(nop) CMD ["nginx", "-g", "daemon off;"] which sets the default command for this image.

As you can see, the image comprises of many read-only layers, each recording a new set of changes to the state triggered by certain instructions. When you start a container using an image, you get a new writable layer on top of the other layers.

This layering phenomenon that happens every time you work with Docker has been made possible by an amazing technical concept called a union file system. Here, union means union in set theory. According to Wikipedia -

It allows files and directories of separate file systems, known as branches, to be transparently overlaid, forming a single coherent file system. Contents of directories which have the same path within the merged branches will be seen together in a single merged directory, within the new, virtual filesystem.

By utilizing this concept, Docker can avoid data duplication and can use previously created layers as a cache for later builds. This results in compact, efficient images that can be used everywhere.

How to Build NGINX from Source

In the previous sub-section, you learned about the FROM, EXPOSE, RUN and CMD instructions. In this sub-section you'll be learning a lot more about other instructions.

In this sub-section you'll again create a custom NGINX image. But the twist is that you'll be building NGINX from source instead of installing it using some package manager such as apt-get as in the previous example.

In order to build NGINX from source, you first need the source of NGINX. If you've cloned my projects repository you'll see a file named nginx-1.19.2.tar.gz inside the custom-nginx directory. You'll use this archive as the source for building NGINX.

Before diving into writing some code, let's plan out the process first. The image creation process this time can be done in seven steps. These are as follows:

• Get a good base image for building the application, like ubuntu.
• Install necessary build dependencies on the base image.
• Copy the nginx-1.19.2.tar.gz file inside the image.
• Extract the contents of the archive and get rid of it.
• Configure the build, compile and install the program using the make tool.
• Get rid of the extracted source code.
• Run nginx executable.

Now that you have a plan, let's begin by opening up old Dockerfile and updating its contents as follows:

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install build-essential\ 
                    libpcre3 \
                    libpcre3-dev \
                    zlib1g \
                    zlib1g-dev \
                    libssl1.1 \
                    libssl-dev \
                    -y && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

COPY nginx-1.19.2.tar.gz .

RUN tar -xvf nginx-1.19.2.tar.gz && rm nginx-1.19.2.tar.gz

RUN cd nginx-1.19.2 && \
    ./configure \
        --sbin-path=/usr/bin/nginx \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --with-pcre \
        --pid-path=/var/run/nginx.pid \
        --with-http_ssl_module && \
    make && make install

RUN rm -rf /nginx-1.19.2

CMD ["nginx", "-g", "daemon off;"]

As you can see, the code inside the Dockerfile reflects the seven steps I talked about above.

• The FROM instruction sets Ubuntu as the base image making an ideal environment for building any application.
• The RUN instruction installs standard packages necessary for building NGINX from source.
• The COPY instruction here is something new. This instruction is responsible for copying the the nginx-1.19.2.tar.gz file inside the image. The generic syntax for the COPY instruction is COPY <source> <destination> where source is in your local filesystem and the destination is inside your image. The . as the destination means the working directory inside the image which is by default / unless set otherwise.
• The second RUN instruction here extracts the contents from the archive using tar and gets rid of it afterwards.
• The archive file contains a directory called nginx-1.19.2 containing the source code. So on the next step, you'll have to cd inside that directory and perform the build process. You can read the How to Install Software from Source Code… and Remove it Afterwards article to learn more on the topic.
• Once the build and installation is complete, you remove the nginx-1.19.2 directory using rm command.
• On the final step you start NGINX in single process mode just like you did before.

Now to build an image using this code, execute the following command:

docker image build --tag custom-nginx:built .

# Step 1/7 : FROM ubuntu:latest
#  ---> d70eaf7277ea
# Step 2/7 : RUN apt-get update &&     apt-get install build-essential                    libpcre3                     libpcre3-dev                     zlib1g                     zlib1g-dev                     libssl-dev                     -y &&     apt-get clean && rm -rf /var/lib/apt/lists/*
#  ---> Running in 2d0aa912ea47
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container 2d0aa912ea47
#  ---> cbe1ced3da11
# Step 3/7 : COPY nginx-1.19.2.tar.gz .
#  ---> 7202902edf3f
# Step 4/7 : RUN tar -xvf nginx-1.19.2.tar.gz && rm nginx-1.19.2.tar.gz
 ---> Running in 4a4a95643020
### LONG EXTRACTION STUFF GOES HERE ###
# Removing intermediate container 4a4a95643020
#  ---> f9dec072d6d6
# Step 5/7 : RUN cd nginx-1.19.2 &&     ./configure         --sbin-path=/usr/bin/nginx         --conf-path=/etc/nginx/nginx.conf         --error-log-path=/var/log/nginx/error.log         --http-log-path=/var/log/nginx/access.log         --with-pcre         --pid-path=/var/run/nginx.pid         --with-http_ssl_module &&     make && make install
#  ---> Running in b07ba12f921e
### LONG CONFIGURATION AND BUILD STUFF GOES HERE ###
# Removing intermediate container b07ba12f921e
#  ---> 5a877edafd8b
# Step 6/7 : RUN rm -rf /nginx-1.19.2
#  ---> Running in 947e1d9ba828
# Removing intermediate container 947e1d9ba828
#  ---> a7702dc7abb7
# Step 7/7 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in 3110c7fdbd57
# Removing intermediate container 3110c7fdbd57
#  ---> eae55f7369d3
# Successfully built eae55f7369d3
# Successfully tagged custom-nginx:built

This code is alright but there are some places where we can make improvements.

• Instead of hard coding the filename like nginx-1.19.2.tar.gz, you can create an argument using the ARG instruction. This way, you'll be able to change the version or filename by just changing the argument.
• Instead of downloading the archive manually, you can let the daemon download the file during the build process. There is another instruction like COPY called the ADD instruction which is capable of adding files from the internet.

Open up the Dockerfile file and update its content as follows:

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install build-essential\ 
                    libpcre3 \
                    libpcre3-dev \
                    zlib1g \
                    zlib1g-dev \
                    libssl1.1 \
                    libssl-dev \
                    -y && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

ARG FILENAME="nginx-1.19.2"
ARG EXTENSION="tar.gz"

ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .

RUN tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION}

RUN cd ${FILENAME} && \
    ./configure \
        --sbin-path=/usr/bin/nginx \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --with-pcre \
        --pid-path=/var/run/nginx.pid \
        --with-http_ssl_module && \
    make && make install

RUN rm -rf /${FILENAME}}

CMD ["nginx", "-g", "daemon off;"]

The code is almost identical to the previous code block except for a new instruction called ARG on line 13, 14 and the usage of the ADD instruction on line 16. Explanation for the updated code is as follows:

• The ARG instruction lets you declare variables like in other languages. These variables or arguments can later be accessed using the ${argument name} syntax. Here, I've put the filename nginx-1.19.2 and the file extension tar.gz in two separate arguments. This way I can switch between newer versions of NGINX or the archive format by making a change in just one place. In the code above, I've added default values to the variables. Variable values can be passed as options of the image build command as well. You can consult the official reference for more details.
• In the ADD instruction, I've formed the download URL dynamically using the arguments declared above. The https://nginx.org/download/${FILENAME}.${EXTENSION} line will result in something like https://nginx.org/download/nginx-1.19.2.tar.gz during the build process. You can change the file version or the extension by changing it in just one place thanks to the ARG instruction.
• The ADD instruction doesn't extract files obtained from the internet by default, hence the usage of tar on line 18.

The rest of the code is almost unchanged. You should be able to understand the usage of the arguments by yourself now. Finally let's try to build an image from this updated code.

docker image build --tag custom-nginx:built .

# Step 1/9 : FROM ubuntu:latest
#  ---> d70eaf7277ea
# Step 2/9 : RUN apt-get update &&     apt-get install build-essential                    libpcre3                     libpcre3-dev                     zlib1g                     zlib1g-dev                     libssl-dev                     -y &&     apt-get clean && rm -rf /var/lib/apt/lists/*
#  ---> cbe1ced3da11
### LONG INSTALLATION STUFF GOES HERE ###
# Step 3/9 : ARG FILENAME="nginx-1.19.2"
#  ---> Running in 33b62a0e9ffb
# Removing intermediate container 33b62a0e9ffb
#  ---> fafc0aceb9c8
# Step 4/9 : ARG EXTENSION="tar.gz"
#  ---> Running in 5c32eeb1bb11
# Removing intermediate container 5c32eeb1bb11
#  ---> 36efdf6efacc
# Step 5/9 : ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .
# Downloading [==================================================>]  1.049MB/1.049MB
#  ---> dba252f8d609
# Step 6/9 : RUN tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION}
#  ---> Running in 2f5b091b2125
### LONG EXTRACTION STUFF GOES HERE ###
# Removing intermediate container 2f5b091b2125
#  ---> 2c9a325d74f1
# Step 7/9 : RUN cd ${FILENAME} &&     ./configure         --sbin-path=/usr/bin/nginx         --conf-path=/etc/nginx/nginx.conf         --error-log-path=/var/log/nginx/error.log         --http-log-path=/var/log/nginx/access.log         --with-pcre         --pid-path=/var/run/nginx.pid         --with-http_ssl_module &&     make && make install
#  ---> Running in 11cc82dd5186
### LONG CONFIGURATION AND BUILD STUFF GOES HERE ###
# Removing intermediate container 11cc82dd5186
#  ---> 6c122e485ec8
# Step 8/9 : RUN rm -rf /${FILENAME}}
#  ---> Running in 04102366960b
# Removing intermediate container 04102366960b
#  ---> 6bfa35420a73
# Step 9/9 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in 63ee44b571bb
# Removing intermediate container 63ee44b571bb
#  ---> 4ce79556db1b
# Successfully built 4ce79556db1b
# Successfully tagged custom-nginx:built

Now you should be able to run a container using the custom-nginx:built image.

docker container run --rm --detach --name custom-nginx-built --publish 8080:80 custom-nginx:built

# 90ccdbc0b598dddc4199451b2f30a942249d85a8ed21da3c8d14612f17eed0aa

docker container ls

# CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                  NAMES
# 90ccdbc0b598        custom-nginx:built   "nginx -g 'daemon of…"   2 minutes ago       Up 2 minutes        0.0.0.0:8080->80/tcp   custom-nginx-built

A container using the custom-nginx:built-v2 image has been successfully run. The container should be accessible at http://127.0.0.1:8080 now.

And here is the trusty default response page from NGINX. You can visit the official reference site to learn more about the available instructions.

How to Optimize Docker Images

The image we built in the last sub-section is functional but very unoptimized. To prove my point let's have a look at the size of the image using the image ls command:

docker image ls

# REPOSITORY         TAG       IMAGE ID       CREATED          SIZE
# custom-nginx       built     1f3aaf40bb54   16 minutes ago   343MB

For an image containing only NGINX, that's too much. If you pull the official image and check its size, you'll see how small it is:

docker image pull nginx:stable

# stable: Pulling from library/nginx
# a076a628af6f: Pull complete 
# 45d7b5d3927d: Pull complete 
# 5e326fece82e: Pull complete 
# 30c386181b68: Pull complete 
# b15158e9ebbe: Pull complete 
# Digest: sha256:ebd0fd56eb30543a9195280eb81af2a9a8e6143496accd6a217c14b06acd1419
# Status: Downloaded newer image for nginx:stable
# docker.io/library/nginx:stable

docker image ls

# REPOSITORY         TAG       IMAGE ID       CREATED          SIZE
# custom-nginx       built     1f3aaf40bb54   25 minutes ago   343MB
# nginx              stable    b9e1dc12387a   11 days ago      133MB

In order to find out the root cause, let's have a look at the Dockerfile first:

FROM ubuntu:latest

RUN apt-get update && \
    apt-get install build-essential\ 
                    libpcre3 \
                    libpcre3-dev \
                    zlib1g \
                    zlib1g-dev \
                    libssl1.1 \
                    libssl-dev \
                    -y && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

ARG FILENAME="nginx-1.19.2"
ARG EXTENSION="tar.gz"

ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .

RUN tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION}

RUN cd ${FILENAME} && \
    ./configure \
        --sbin-path=/usr/bin/nginx \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --with-pcre \
        --pid-path=/var/run/nginx.pid \
        --with-http_ssl_module && \
    make && make install

RUN rm -rf /${FILENAME}}

CMD ["nginx", "-g", "daemon off;"]

As you can see on line 3, the RUN instruction installs a lot of stuff. Although these packages are necessary for building NGINX from source, they are not necessary for running it.

Out of the 6 packages that we installed, only two are necessary for running NGINX. These are libpcre3 and zlib1g. So a better idea would be to uninstall the other packages once the build process is done.

To do so, update your Dockerfile as follows:

FROM ubuntu:latest

EXPOSE 80

ARG FILENAME="nginx-1.19.2"
ARG EXTENSION="tar.gz"

ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .

RUN apt-get update && \
    apt-get install build-essential \ 
                    libpcre3 \
                    libpcre3-dev \
                    zlib1g \
                    zlib1g-dev \
                    libssl1.1 \
                    libssl-dev \
                    -y && \
    tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION} && \
    cd ${FILENAME} && \
    ./configure \
        --sbin-path=/usr/bin/nginx \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --with-pcre \
        --pid-path=/var/run/nginx.pid \
        --with-http_ssl_module && \
    make && make install && \
    cd / && rm -rfv /${FILENAME} && \
    apt-get remove build-essential \ 
                    libpcre3-dev \
                    zlib1g-dev \
                    libssl-dev \
                    -y && \
    apt-get autoremove -y && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

CMD ["nginx", "-g", "daemon off;"]

As you can see, on line 10 a single RUN instruction is doing all the necessary heavy-lifting. The exact chain of events is as follows:

• From line 10 to line 17, all the necessary packages are being installed.
• On line 18, the source code is being extracted and the downloaded archive gets removed.
• From line 19 to line 28, NGINX is configured, built, and installed on the system.
• On line 29, the extracted files from the downloaded archive get removed.
• From line 30 to line 36, all the unnecessary packages are being uninstalled and cache cleared. The libpcre3 and zlib1g packages are needed for running NGINX so we keep them.

You may ask why am I doing so much work in a single RUN instruction instead of nicely splitting them into multiple instructions like we did previously. Well, splitting them up would be a mistake.

If you install packages and then remove them in separate RUN instructions, they'll live in separate layers of the image. Although the final image will not have the removed packages, their size will still be added to the final image since they exist in one of the layers consisting the image. So make sure you make these kind of changes on a single layer.

Let's build an image using this Dockerfile and see the differences.

docker image build --tag custom-nginx:built .

# Sending build context to Docker daemon  1.057MB
# Step 1/7 : FROM ubuntu:latest
#  ---> f63181f19b2f
# Step 2/7 : EXPOSE 80
#  ---> Running in 006f39b75964
# Removing intermediate container 006f39b75964
#  ---> 6943f7ef9376
# Step 3/7 : ARG FILENAME="nginx-1.19.2"
#  ---> Running in ffaf89078594
# Removing intermediate container ffaf89078594
#  ---> 91b5cdb6dabe
# Step 4/7 : ARG EXTENSION="tar.gz"
#  ---> Running in d0f5188444b6
# Removing intermediate container d0f5188444b6
#  ---> 9626f941ccb2
# Step 5/7 : ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .
# Downloading [==================================================>]  1.049MB/1.049MB
#  ---> a8e8dcca1be8
# Step 6/7 : RUN apt-get update &&     apt-get install build-essential                     libpcre3                     libpcre3-dev                     zlib1g                     zlib1g-dev                     libssl-dev                     -y &&     tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION} &&     cd ${FILENAME} &&     ./configure         --sbin-path=/usr/bin/nginx         --conf-path=/etc/nginx/nginx.conf         --error-log-path=/var/log/nginx/error.log         --http-log-path=/var/log/nginx/access.log         --with-pcre         --pid-path=/var/run/nginx.pid         --with-http_ssl_module &&     make && make install &&     cd / && rm -rfv /${FILENAME} &&     apt-get remove build-essential                     libpcre3-dev                     zlib1g-dev                     libssl-dev                     -y &&     apt-get autoremove -y &&     apt-get clean && rm -rf /var/lib/apt/lists/*
#  ---> Running in e5675cad1260
### LONG INSTALLATION AND BUILD STUFF GOES HERE ###
# Removing intermediate container e5675cad1260
#  ---> dc7e4161f975
# Step 7/7 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in b579e4600247
# Removing intermediate container b579e4600247
#  ---> 512aa6a95a93
# Successfully built 512aa6a95a93
# Successfully tagged custom-nginx:built

docker image ls

# REPOSITORY         TAG       IMAGE ID       CREATED              SIZE
# custom-nginx       built     512aa6a95a93   About a minute ago   81.6MB
# nginx              stable    b9e1dc12387a   11 days ago          133MB

As you can see, the image size has gone from being 343MB to 81.6MB. The official image is 133MB. This is a pretty optimized build, but we can go a bit further in the next sub-section.

Embracing Alpine Linux

If you've been fiddling around with containers for some time now, you may have heard about something called Alpine Linux. It's a full-featured Linux distribution like Ubuntu, Debian or Fedora.

But the good thing about Alpine is that it's built around musl libc and busybox and is lightweight. Where the latest ubuntu image weighs at around 28MB, alpine is 2.8MB.

Apart from the lightweight nature, Alpine is also secure and is a much better fit for creating containers than the other distributions.

Although not as user friendly as the other commercial distributions, the transition to Alpine is still very simple. In this sub-section you'll learn about recreating the custom-nginx image using the Alpine image as its base.

Open up your Dockerfile and update its content as follows:

FROM alpine:latest

EXPOSE 80

ARG FILENAME="nginx-1.19.2"
ARG EXTENSION="tar.gz"

ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .

RUN apk add --no-cache pcre zlib && \
    apk add --no-cache \
            --virtual .build-deps \
            build-base \ 
            pcre-dev \
            zlib-dev \
            openssl-dev && \
    tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION} && \
    cd ${FILENAME} && \
    ./configure \
        --sbin-path=/usr/bin/nginx \
        --conf-path=/etc/nginx/nginx.conf \
        --error-log-path=/var/log/nginx/error.log \
        --http-log-path=/var/log/nginx/access.log \
        --with-pcre \
        --pid-path=/var/run/nginx.pid \
        --with-http_ssl_module && \
    make && make install && \
    cd / && rm -rfv /${FILENAME} && \
    apk del .build-deps

CMD ["nginx", "-g", "daemon off;"]

The code is almost identical except for a few changes. I'll be listing the changes and explaining them as I go:

• Instead of using apt-get install for installing packages, we use apk add. The --no-cache option means that the downloaded package won't be cached. Likewise we'll use apk del instead of apt-get remove to uninstall packages.
• The --virtual option for the apk add command is used for bundling a bunch of packages into a single virtual package for easier management. Packages that are needed only for building the program are labeled as .build-deps which are then removed on line 29 by executing the apk del .build-deps command. You can learn more about virtuals in the official docs.
• The package names are a bit different here. Usually every Linux distribution has its package repository available to everyone where you can search for packages. If you know the packages required for a certain task, then you can just head over to the designated repository for a distribution and search for it. You can look up Alpine Linux packages here.

Now build a new image using this Dockerfile and see the difference in file size:

docker image build --tag custom-nginx:built .

# Sending build context to Docker daemon  1.055MB
# Step 1/7 : FROM alpine:latest
#  ---> 7731472c3f2a
# Step 2/7 : EXPOSE 80
#  ---> Running in 8336cfaaa48d
# Removing intermediate container 8336cfaaa48d
#  ---> d448a9049d01
# Step 3/7 : ARG FILENAME="nginx-1.19.2"
#  ---> Running in bb8b2eae9d74
# Removing intermediate container bb8b2eae9d74
#  ---> 87ca74f32fbe
# Step 4/7 : ARG EXTENSION="tar.gz"
#  ---> Running in aa09627fe48c
# Removing intermediate container aa09627fe48c
#  ---> 70cb557adb10
# Step 5/7 : ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .
# Downloading [==================================================>]  1.049MB/1.049MB
#  ---> b9790ce0c4d6
# Step 6/7 : RUN apk add --no-cache pcre zlib &&     apk add --no-cache             --virtual .build-deps             build-base             pcre-dev             zlib-dev             openssl-dev &&     tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION} &&     cd ${FILENAME} &&     ./configure         --sbin-path=/usr/bin/nginx         --conf-path=/etc/nginx/nginx.conf         --error-log-path=/var/log/nginx/error.log         --http-log-path=/var/log/nginx/access.log         --with-pcre         --pid-path=/var/run/nginx.pid         --with-http_ssl_module &&     make && make install &&     cd / && rm -rfv /${FILENAME} &&     apk del .build-deps
#  ---> Running in 0b301f64ffc1
### LONG INSTALLATION AND BUILD STUFF GOES HERE ###
# Removing intermediate container 0b301f64ffc1
#  ---> dc7e4161f975
# Step 7/7 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in b579e4600247
# Removing intermediate container b579e4600247
#  ---> 3e186a3c6830
# Successfully built 3e186a3c6830
# Successfully tagged custom-nginx:built

docker image ls

# REPOSITORY         TAG       IMAGE ID       CREATED         SIZE
# custom-nginx       built     3e186a3c6830   8 seconds ago   12.8MB

Where the ubuntu version was 81.6MB, the alpine one has come down to 12.8MB which is a massive gain. Apart from the apk package manager, there are some other things that differ in Alpine from Ubuntu but they're not that big a deal. You can just search the internet whenever you get stuck.

How to Create Executable Docker Images

In the previous section you worked with the fhsinchy/rmbyext image. In this section you'll learn how to make such an executable image.

To begin with, open up the directory where you've cloned the repository that came with this book. The code for the rmbyext application resides inside the sub-directory with the same name.

Before you start working on the Dockerfile take a moment to plan out what the final output should be. In my opinion it should be like something like this:

• The image should have Python pre-installed.
• It should contain a copy of my rmbyext script.
• A working directory should be set where the script will be executed.
• The rmbyext script should be set as the entry-point so the image can take extension names as arguments.

To build the above mentioned image, take the following steps:

• Get a good base image for running Python scripts, like python.
• Set-up the working directory to an easily accessible directory.
• Install Git so that the script can be installed from my GitHub repository.
• Install the script using Git and pip.
• Get rid of the build's unnecessary packages.
• Set rmbyext as the entry-point for this image.

Now create a new Dockerfile inside the rmbyext directory and put the following code in it:

FROM python:3-alpine

WORKDIR /zone

RUN apk add --no-cache git && \
    pip install git+https://github.com/fhsinchy/rmbyext.git#egg=rmbyext && \
    apk del git

ENTRYPOINT [ "rmbyext" ]

The explanation for the instructions in this file is as follows:

• The FROM instruction sets python as the base image, making an ideal environment for running Python scripts. The 3-alpine tag indicates that you want the Alpine variant of Python 3.
• The WORKDIR instruction sets the default working directory to /zone here. The name of the working directory is completely random here. I found zone to be a fitting name, you may use anything you want.
• Given the rmbyext script is installed from GitHub, git is an install time dependency. The RUN instruction on line 5 installs git then installs the rmbyext script using Git and pip. It also gets rid of git afterwards.
• Finally on line 9, the ENTRYPOINT instruction sets the rmbyext script as the entry-point for this image.

In this entire file, line 9 is the magic that turns this seemingly normal image into an executable one. Now to build the image you can execute following command:

docker image build --tag rmbyext .

# Sending build context to Docker daemon  2.048kB
# Step 1/4 : FROM python:3-alpine
# 3-alpine: Pulling from library/python
# 801bfaa63ef2: Already exists 
# 8723b2b92bec: Already exists 
# 4e07029ccd64: Already exists 
# 594990504179: Already exists 
# 140d7fec7322: Already exists 
# Digest: sha256:7492c1f615e3651629bd6c61777e9660caa3819cf3561a47d1d526dfeee02cf6
# Status: Downloaded newer image for python:3-alpine
#  ---> d4d4f50f871a
# Step 2/4 : WORKDIR /zone
#  ---> Running in 454374612a91
# Removing intermediate container 454374612a91
#  ---> 7f7e49bc98d2
# Step 3/4 : RUN apk add --no-cache git &&     pip install git+https://github.com/fhsinchy/rmbyext.git#egg=rmbyext &&     apk del git
#  ---> Running in 27e2e96dc95a
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container 27e2e96dc95a
#  ---> 3c7389432e36
# Step 4/4 : ENTRYPOINT [ "rmbyext" ]
#  ---> Running in f239bbea1ca6
# Removing intermediate container f239bbea1ca6
#  ---> 1746b0cedbc7
# Successfully built 1746b0cedbc7
# Successfully tagged rmbyext:latest

docker image ls

# REPOSITORY         TAG        IMAGE ID       CREATED         SIZE
# rmbyext            latest     1746b0cedbc7   4 minutes ago   50.9MB

Here I haven't provided any tag after the image name, so the image has been tagged as latest by default. You should be able to run the image as you saw in the previous section. Remember to refer to the actual image name you've set, instead of fhsinchy/rmbyext here.

How to Share Your Docker Images Online

Now that you know how to make images, it's time to share them with the world. Sharing images online is easy. All you need is an account at any of the online registries. I'll be using Docker Hub here.

Navigate to the Sign Up page and create a free account. A free account allows you to host unlimited public repositories and one private repository.

Once you've created the account, you'll have to sign in to it using the docker CLI. So open up your terminal and execute the following command to do so:

docker login

# Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
# Username: fhsinchy
# Password: 
# WARNING! Your password will be stored unencrypted in /home/fhsinchy/.docker/config.json.
# Configure a credential helper to remove this warning. See
# https://docs.docker.com/engine/reference/commandline/login/#credentials-store
#
# Login Succeeded

You'll be prompted for your username and password. If you input them properly, you should be logged in to your account successfully.

In order to share an image online, the image has to be tagged. You've already learned about tagging in a previous sub-section. Just to refresh your memory, the generic syntax for the --tag or -t option is as follows:

--tag <image repository>:<image tag>

As an example, let's share the custom-nginx image online. To do so, open up a new terminal window inside the custom-nginx project directory.

To share an image online, you'll have to tag it following the <docker hub username>/<image name>:<image tag> syntax. My username is fhsinchy so the command will look like this:

docker image build --tag fhsinchy/custom-nginx:latest --file Dockerfile.built .

# Step 1/9 : FROM ubuntu:latest
#  ---> d70eaf7277ea
# Step 2/9 : RUN apt-get update &&     apt-get install build-essential                    libpcre3                     libpcre3-dev                     zlib1g                     zlib1g-dev                     libssl-dev                     -y &&     apt-get clean && rm -rf /var/lib/apt/lists/*
#  ---> cbe1ced3da11
### LONG INSTALLATION STUFF GOES HERE ###
# Step 3/9 : ARG FILENAME="nginx-1.19.2"
#  ---> Running in 33b62a0e9ffb
# Removing intermediate container 33b62a0e9ffb
#  ---> fafc0aceb9c8
# Step 4/9 : ARG EXTENSION="tar.gz"
#  ---> Running in 5c32eeb1bb11
# Removing intermediate container 5c32eeb1bb11
#  ---> 36efdf6efacc
# Step 5/9 : ADD https://nginx.org/download/${FILENAME}.${EXTENSION} .
# Downloading [==================================================>]  1.049MB/1.049MB
#  ---> dba252f8d609
# Step 6/9 : RUN tar -xvf ${FILENAME}.${EXTENSION} && rm ${FILENAME}.${EXTENSION}
#  ---> Running in 2f5b091b2125
### LONG EXTRACTION STUFF GOES HERE ###
# Removing intermediate container 2f5b091b2125
#  ---> 2c9a325d74f1
# Step 7/9 : RUN cd ${FILENAME} &&     ./configure         --sbin-path=/usr/bin/nginx         --conf-path=/etc/nginx/nginx.conf         --error-log-path=/var/log/nginx/error.log         --http-log-path=/var/log/nginx/access.log         --with-pcre         --pid-path=/var/run/nginx.pid         --with-http_ssl_module &&     make && make install
#  ---> Running in 11cc82dd5186
### LONG CONFIGURATION AND BUILD STUFF GOES HERE ###
# Removing intermediate container 11cc82dd5186
#  ---> 6c122e485ec8
# Step 8/9 : RUN rm -rf /${FILENAME}}
#  ---> Running in 04102366960b
# Removing intermediate container 04102366960b
#  ---> 6bfa35420a73
# Step 9/9 : CMD ["nginx", "-g", "daemon off;"]
#  ---> Running in 63ee44b571bb
# Removing intermediate container 63ee44b571bb
#  ---> 4ce79556db1b
# Successfully built 4ce79556db1b
# Successfully tagged fhsinchy/custom-nginx:latest

In this command the fhsinchy/custom-nginx is the image repository and latest is the tag. The image name can be anything you want and can not be changed once you've uploaded the image. The tag can be changed whenever you want and usually reflects the version of the software or different kind of builds.

Take the node image as an example. The node:lts image refers to the long term support version of Node.js whereas the node:lts-alpine version refers to the Node.js version built for Alpine Linux, which is much smaller than the regular one.

If you do not give the image any tag, it'll be automatically tagged as latest. But that doesn't mean that the latest tag will always refer to the latest version. If, for some reason, you explicitly tag an older version of the image as latest, then Docker will not make any extra effort to cross check that.

Once the image has been built, you can them upload it by executing the following command:

docker image push <image repository>:<image tag>

So in my case the command will be as follows:

docker image push fhsinchy/custom-nginx:latest

# The push refers to repository [docker.io/fhsinchy/custom-nginx]
# 4352b1b1d9f5: Pushed 
# a4518dd720bd: Pushed 
# 1d756dc4e694: Pushed 
# d7a7e2b6321a: Pushed 
# f6253634dc78: Mounted from library/ubuntu 
# 9069f84dbbe9: Mounted from library/ubuntu 
# bacd3af13903: Mounted from library/ubuntu 
# latest: digest: sha256:ffe93440256c9edb2ed67bf3bba3c204fec3a46a36ac53358899ce1a9eee497a size: 1788

Depending on the image size, the upload may take some time. Once it's done you should able to find the image in your hub profile page.

How to Containerize a JavaScript Application

Now that you've got some idea of how to create images, it's time to work with something a bit more relevant.

In this sub-section, you'll be working with the source code of the fhsinchy/hello-dock image that you worked with on a previous section. In the process of containerizing this very simple application, you'll be introduced to volumes and multi-staged builds, two of the most important concepts in Docker.

How to Write the Development Dockerfile

To begin with, open up the directory where you've cloned the repository that came with this book. Code for the hello-dock application resides inside the sub-directory with the same name.

This is a very simple JavaScript project powered by the vitejs/vite project. Don't worry though, you don't need to know JavaScript or vite in order to go through this sub-section. Having a basic understanding of Node.js and npm will suffice.

Just like any other project you've done in the previous sub-section, you'll begin by making a plan of how you want this application to run. In my opinion, the plan should be as follows:

• Get a good base image for running JavaScript applications, like node.
• Set the default working directory inside the image.
• Copy the package.json file into the image.
• Install necessary dependencies.
• Copy the rest of the project files.
• Start the vite development server by executing npm run dev command.

This plan should always come from the developer of the application that you're containerizing. If you're the developer yourself, then you should already have a proper understanding of how this application needs to be run.

Now if you put the above mentioned plan inside Dockerfile.dev, the file should look like as follows:

FROM node:lts-alpine

EXPOSE 3000

USER node

RUN mkdir -p /home/node/app

WORKDIR /home/node/app

COPY ./package.json .
RUN npm install

COPY . .

CMD [ "npm", "run", "dev" ]

The explanation for this code is as follows:

• The FROM instruction here sets the official Node.js image as the base, giving you all the goodness of Node.js necessary to run any JavaScript application. The lts-alpine tag indicates that you want to use the Alpine variant, long term support version of the image. Available tags and necessary documentation for the image can be found on the node hub page.
• The USER instruction sets the default user for the image to node. By default Docker runs containers as the root user. But according to Docker and Node.js Best Practices this can pose a security threat. So it's a better idea to run as a non-root user whenever possible. The node image comes with a non-root user named node which you can set as the default user using the USER instruction.
• The RUN mkdir -p /home/node/app instruction creates a directory called app inside the home directory of the node user. The home directory for any non-root user in Linux is usually /home/<user name> by default.
• Then the WORKDIR instruction sets the default working directory to the newly created /home/node/app directory. By default the working directory of any image is the root. You don't want any unnecessary files sprayed all over your root directory, do you? Hence you change the default working directory to something more sensible like /home/node/app or whatever you like. This working directory will be applicable to any subsequent COPY, ADD, RUN and CMD instructions.
• The COPY instruction here copies the package.json file which contains information regarding all the necessary dependencies for this application. The RUN instruction executes the npm install command which is the default command for installing dependencies using a package.json file in Node.js projects. The . at the end represents the working directory.
• The second COPY instruction copies the rest of the content from the current directory (.) of the host filesystem to the working directory (.) inside the image.
• Finally, the CMD instruction here sets the default command for this image which is npm run dev written in exec form.
• The vite development server by default runs on port 3000 , and adding an EXPOSE command seemed like a good idea, so there you go.

Now, to build an image from this Dockerfile.dev you can execute the following command:

docker image build --file Dockerfile.dev --tag hello-dock:dev .

# Step 1/7 : FROM node:lts
#  ---> b90fa0d7cbd1
# Step 2/7 : EXPOSE 3000
#  ---> Running in 722d639badc7
# Removing intermediate container 722d639badc7
#  ---> e2a8aa88790e
# Step 3/7 : WORKDIR /app
#  ---> Running in 998e254b4d22
# Removing intermediate container 998e254b4d22
#  ---> 6bd4c42892a4
# Step 4/7 : COPY ./package.json .
#  ---> 24fc5164a1dc
# Step 5/7 : RUN npm install
#  ---> Running in 23b4de3f930b
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container 23b4de3f930b
#  ---> c17ecb19a210
# Step 6/7 : COPY . .
#  ---> afb6d9a1bc76
# Step 7/7 : CMD [ "npm", "run", "dev" ]
#  ---> Running in a7ff529c28fe
# Removing intermediate container a7ff529c28fe
#  ---> 1792250adb79
# Successfully built 1792250adb79
# Successfully tagged hello-dock:dev

Given the filename is not Dockerfile you have to explicitly pass the filename using the --file option. A container can be run using this image by executing the following command:

docker container run \
    --rm \
    --detach \
    --publish 3000:3000 \
    --name hello-dock-dev \
    hello-dock:dev

# 21b9b1499d195d85e81f0e8bce08f43a64b63d589c5f15cbbd0b9c0cb07ae268

Now visit http://127.0.0.1:3000 to see the hello-dock application in action.

Congratulations on running your first real-world application inside a container. The code you've just written is okay but there is one big issue with it and a few places where it can be improved. Let's begin with the issue first.

How to Work With Bind Mounts in Docker

If you've worked with any front-end JavaScript framework before, you should know that the development servers in these frameworks usually come with a hot reload feature. That is if you make a change in your code, the server will reload, automatically reflecting any changes you've made immediately.

But if you make any changes in your code right now, you'll see nothing happening to your application running in the browser. This is because you're making changes in the code that you have in your local file system but the application you're seeing in the browser resides inside the container file system.

To solve this issue, you can again make use of a bind mount. Using bind mounts, you can easily mount one of your local file system directories inside a container. Instead of making a copy of the local file system, the bind mount can reference the local file system directly from inside the container.

This way, any changes you make to your local source code will reflect immediately inside the container, triggering the hot reload feature of the vite development server. Changes made to the file system inside the container will be reflected on your local file system as well.

You've already learned in the Working With Executable Images sub-section, bind mounts can be created using the --volume or -v option for the container run or container start commands. Just to remind you, the generic syntax is as follows:

--volume <local file system directory absolute path>:<container file system directory absolute path>:<read write access>

Stop your previously started hello-dock-dev container, and start a new container by executing the following command:

docker container run \
    --rm \
    --publish 3000:3000 \
    --name hello-dock-dev \
    --volume $(pwd):/home/node/app \
    hello-dock:dev

# sh: 1: vite: not found
# npm ERR! code ELIFECYCLE
# npm ERR! syscall spawn
# npm ERR! file sh
# npm ERR! errno ENOENT
# npm ERR! hello-dock@0.0.0 dev: `vite`
# npm ERR! spawn ENOENT
# npm ERR!
# npm ERR! Failed at the hello-dock@0.0.0 dev script.
# npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
# npm WARN Local package.json exists, but node_modules missing, did you mean to install?

Keep in mind, I've omitted the --detach option and that's to demonstrate a very important point. As you can see, the application is not running at all now.

That's because although the usage of a volume solves the issue of hot reloads, it introduces another problem. If you have any previous experience with Node.js, you may know that the dependencies of a Node.js project live inside the node_modules directory on the project root.

Now that you're mounting the project root on your local file system as a volume inside the container, the content inside the container gets replaced along with the node_modules directory containing all the dependencies. This means that the vite package has gone missing.

How to Work With Anonymous Volumes in Docker

This problem can be solved using an anonymous volume. An anonymous volume is identical to a bind mount except that you don't need to specify the source directory here. The generic syntax for creating an anonymous volume is as follows:

--volume <container file system directory absolute path>:<read write access>

So the final command for starting the hello-dock container with both volumes should be as follows:

docker container run \
    --rm \
    --detach \
    --publish 3000:3000 \
    --name hello-dock-dev \
    --volume $(pwd):/home/node/app \
    --volume /home/node/app/node_modules \
    hello-dock:dev

# 53d1cfdb3ef148eb6370e338749836160f75f076d0fbec3c2a9b059a8992de8b

Here, Docker will take the entire node_modules directory from inside the container and tuck it away in some other directory managed by the Docker daemon on your host file system and will mount that directory as node_modules inside the container.

How to Perform Multi-Staged Builds in Docker

So far in this section, you've built an image for running a JavaScript application in development mode. Now if you want to build the image in production mode, some new challenges show up.

In development mode the npm run serve command starts a development server that serves the application to the user. That server not only serves the files but also provides the hot reload feature.

In production mode, the npm run build command compiles all your JavaScript code into some static HTML, CSS, and JavaScript files. To run these files you don't need node or any other runtime dependencies. All you need is a server like nginx for example.

To create an image where the application runs in production mode, you can take the following steps:

• Use node as the base image and build the application.
• Install nginx inside the node image and use that to serve the static files.

This approach is completely valid. But the problem is that the node image is big and most of the stuff it carries is unnecessary to serve your static files. A better approach to this scenario is as follows:

• Use node image as the base and build the application.
• Copy the files created using the node image to an nginx image.
• Create the final image based on nginx and discard all node related stuff.

This way your image only contains the files that are needed and becomes really handy.

This approach is a multi-staged build. To perform such a build, create a new Dockerfile inside your hello-dock project directory and put the following content in it:

FROM node:lts-alpine as builder

WORKDIR /app

COPY ./package.json ./
RUN npm install

COPY . .
RUN npm run build

FROM nginx:stable-alpine

EXPOSE 80

COPY --from=builder /app/dist /usr/share/nginx/html

As you can see the Dockerfile looks a lot like your previous ones with a few oddities. The explanation for this file is as follows:

• Line 1 starts the first stage of the build using node:lts-alpine as the base image. The as builder syntax assigns a name to this stage so that it can be referred to later on.
• From line 3 to line 9, it's standard stuff that you've seen many times before. The RUN npm run build command actually compiles the entire application and tucks it inside /app/dist directory where /app is the working directory and /dist is the default output directory for vite applications.
• Line 11 starts the second stage of the build using nginx:stable-alpine as the base image.
• The NGINX server runs on port 80 by default so the line EXPOSE 80 is added.
• The last line is a COPY instruction. The --from=builder part indicates that you want to copy some files from the builder stage. After that it's a standard copy instruction where /app/dist is the source and /usr/share/nginx/html is the destination. The destination used here is the default site path for NGINX so any static file you put inside there will be automatically served.

As you can see, the resulting image is a nginx base image containing only the files necessary for running the application. To build this image execute the following command:

docker image build --tag hello-dock:prod .

# Step 1/9 : FROM node:lts-alpine as builder
#  ---> 72aaced1868f
# Step 2/9 : WORKDIR /app
#  ---> Running in e361c5c866dd
# Removing intermediate container e361c5c866dd
#  ---> 241b4b97b34c
# Step 3/9 : COPY ./package.json ./
#  ---> 6c594c5d2300
# Step 4/9 : RUN npm install
#  ---> Running in 6dfabf0ee9f8
# npm WARN deprecated fsevents@2.1.3: Please update to v 2.2.x
#
# > esbuild@0.8.29 postinstall /app/node_modules/esbuild
# > node install.js
#
# npm notice created a lockfile as package-lock.json. You should commit this file.
# npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/chokidar/node_modules/fsevents):
# npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
# npm WARN hello-dock@0.0.0 No description
# npm WARN hello-dock@0.0.0 No repository field.
# npm WARN hello-dock@0.0.0 No license field.
#
# added 327 packages from 301 contributors and audited 329 packages in 35.971s
#
# 26 packages are looking for funding
#   run `npm fund` for details
#
# found 0 vulnerabilities
#
# Removing intermediate container 6dfabf0ee9f8
#  ---> 21fd1b065314
# Step 5/9 : COPY . .
#  ---> 43243f95bff7
# Step 6/9 : RUN npm run build
#  ---> Running in 4d918cf18584
#
# > hello-dock@0.0.0 build /app
# > vite build
#
# - Building production bundle...
#
# [write] dist/index.html 0.39kb, brotli: 0.15kb
# [write] dist/_assets/docker-handbook-github.3adb4865.webp 12.32kb
# [write] dist/_assets/index.eabcae90.js 42.56kb, brotli: 15.40kb
# [write] dist/_assets/style.0637ccc5.css 0.16kb, brotli: 0.10kb
# - Building production bundle...
#
# Build completed in 1.71s.
#
# Removing intermediate container 4d918cf18584
#  ---> 187fb3e82d0d
# Step 7/9 : EXPOSE 80
#  ---> Running in b3aab5cf5975
# Removing intermediate container b3aab5cf5975
#  ---> d6fcc058cfda
# Step 8/9 : FROM nginx:stable-alpine
# stable: Pulling from library/nginx
# 6ec7b7d162b2: Already exists 
# 43876acb2da3: Pull complete 
# 7a79edd1e27b: Pull complete 
# eea03077c87e: Pull complete 
# eba7631b45c5: Pull complete 
# Digest: sha256:2eea9f5d6fff078ad6cc6c961ab11b8314efd91fb8480b5d054c7057a619e0c3
# Status: Downloaded newer image for nginx:stable
#  ---> 05f64a802c26
# Step 9/9 : COPY --from=builder /app/dist /usr/share/nginx/html
#  ---> 8c6dfc34a10d
# Successfully built 8c6dfc34a10d
# Successfully tagged hello-dock:prod

Once the image has been built, you may run a new container by executing the following command:

docker container run \
    --rm \
    --detach \
    --name hello-dock-prod \
    --publish 8080:80 \
    hello-dock:prod

# 224aaba432bb09aca518fdd0365875895c2f5121eb668b2e7b2d5a99c019b953

The running application should be available on http://127.0.0.1:8080:

Here you can see my hello-dock application in all its glory. Multi-staged builds can be very useful if you're building large applications with a lot of dependencies. If configured properly, images built in multiple stages can be very optimized and compact.

How to Ignore Unnecessary Files

If you've been working with git for some time now, you may know about the .gitignore files in projects. These contain a list of files and directories to be excluded from the repository.

Well, Docker has a similar concept. The .dockerignore file contains a list of files and directories to be excluded from image builds. You can find a pre-created .dockerignore file in the hello-dock directory.

.git
*Dockerfile*
*docker-compose*
node_modules

This .dockerignore file has to be in the build context. Files and directories mentioned here will be ignored by the COPY instruction. But if you do a bind mount, the .dockerignore file will have no effect. I've added .dockerignore files where necessary in the project repository.

Network Manipulation Basics in Docker

So far in this book, you've only worked with single container projects. But in real life, the majority of projects that you'll have to work with will have more than one container. And to be honest, working with a bunch of containers can be a little difficult if you don't understand the nuances of container isolation.

So in this section of the book, you'll get familiar with basic networking with Docker and you'll work hands on with a small multi-container project.

Well you've already learned in the previous section that containers are isolated environments. Now consider a scenario where you have a notes-api application powered by Express.js and a PostgreSQL database server running in two separate containers.

These two containers are completely isolated from each other and are oblivious to each other's existence. So how do you connect the two? Won't that be a challenge?‌

You may think of two possible solutions to this problem. They are as follows:

• Accessing the database server using an exposed port.
• Accessing the database server using its IP address and default port.

The first one involves exposing a port from the postgres container and the notes-api will connect through that. Assume that the exposed port from the postgres container is 5432. Now if you try to connect to 127.0.0.1:5432 from inside the notes-api container, you'll find that the notes-api can't find the database server at all.

The reason is that when you're saying 127.0.0.1 inside the notes-api container, you're simply referring to the localhost of that container and that container only. The postgres server simply doesn't exist there. As a result the notes-api application failed to connect.

The second solution you may think of is finding the exact IP address of the postgres container using the container inspect command and using that with the port. Assuming the name of the postgres container is notes-api-db-server you can easily get the IP address by executing the following command:

docker container inspect --format='{{range .NetworkSettings.Networks}} {{.IPAddress}} {{end}}' notes-api-db-server

#  172.17.0.2

Now given that the default port for postgres is 5432, you can very easily access the database server by connecting to 172.17.0.2:5432 from the notes-api container.

There are problems in this approach as well. Using IP addresses to refer to a container is not recommended. Also, if the container gets destroyed and recreated, the IP address may change. Keeping track of these changing IP addresses can be pretty hectic.

Now that I've dismissed the possible wrong answers to the original question, the correct answer is, you connect them by putting them under a user-defined bridge network.

Docker Network Basics

A network in Docker is another logical object like a container and image. Just like the other two, there is a plethora of commands under the docker network group for manipulating networks.

To list out the networks in your system, execute the following command:

docker network ls

# NETWORK ID     NAME      DRIVER    SCOPE
# c2e59f2b96bd   bridge    bridge    local
# 124dccee067f   host      host      local
# 506e3822bf1f   none      null      local

You should see three networks in your system. Now look at the DRIVER column of the table here. These drivers are can be treated as the type of network.

By default, Docker has five networking drivers. They are as follows:

• bridge - The default networking driver in Docker. This can be used when multiple containers are running in standard mode and need to communicate with each other.
• host - Removes the network isolation completely. Any container running under a host network is basically attached to the network of the host system.
• none - This driver disables networking for containers altogether. I haven't found any use-case for this yet.
• overlay - This is used for connecting multiple Docker daemons across computers and is out of the scope of this book.
• macvlan - Allows assignment of MAC addresses to containers, making them function like physical devices in a network.

There are also third-party plugins that allow you to integrate Docker with specialized network stacks. Out of the five mentioned above, you'll only work with the bridge networking driver in this book.

How to Create a User-Defined Bridge in Docker

Before you start creating your own bridge, I would like to take some time to discuss the default bridge network that comes with Docker. Let's begin by listing all the networks on your system:

docker network ls

# NETWORK ID     NAME      DRIVER    SCOPE
# c2e59f2b96bd   bridge    bridge    local
# 124dccee067f   host      host      local
# 506e3822bf1f   none      null      local

As you can see, Docker comes with a default bridge network named bridge. Any container you run will be automatically attached to this bridge network:

docker container run --rm --detach --name hello-dock --publish 8080:80 fhsinchy/hello-dock
# a37f723dad3ae793ce40f97eb6bb236761baa92d72a2c27c24fc7fda0756657d

docker network inspect --format='{{range .Containers}}{{.Name}}{{end}}' bridge
# hello-dock

Containers attached to the default bridge network can communicate with each others using IP addresses which I have already discouraged in the previous sub-section.

A user-defined bridge, however, has some extra features over the default one. According to the official docs on this topic, some notable extra features are as follows:

• User-defined bridges provide automatic DNS resolution between containers: This means containers attached to the same network can communicate with each others using the container name. So if you have two containers named notes-api and notes-db the API container will be able to connect to the database container using the notes-db name.
• User-defined bridges provide better isolation: All containers are attached to the default bridge network by default which can cause conflicts among them. Attaching containers to a user-defined bridge can ensure better isolation.
• Containers can be attached and detached from user-defined networks on the fly: During a container’s lifetime, you can connect or disconnect it from user-defined networks on the fly. To remove a container from the default bridge network, you need to stop the container and recreate it with different network options.

Now that you've learned quite a lot about a user-defined network, it's time to create one for yourself. A network can be created using the network create command. The generic syntax for the command is as follows:

docker network create <network name>

To create a network with the name skynet execute the following command:

docker network create skynet

# 7bd5f351aa892ac6ec15fed8619fc3bbb95a7dcdd58980c28304627c8f7eb070

docker network ls

# NETWORK ID     NAME     DRIVER    SCOPE
# be0cab667c4b   bridge   bridge    local
# 124dccee067f   host     host      local
# 506e3822bf1f   none     null      local
# 7bd5f351aa89   skynet   bridge    local

As you can see a new network has been created with the given name. No container is currently attached to this network. In the next sub-section, you'll learn about attaching containers to a network.

How to Attach a Container to a Network in Docker

There are mostly two ways of attaching a container to a network. First, you can use the network connect command to attach a container to a network. The generic syntax for the command is as follows:

docker network connect <network identifier> <container identifier>

To connect the hello-dock container to the skynet network, you can execute the following command:

docker network connect skynet hello-dock

docker network inspect --format='{{range .Containers}} {{.Name}} {{end}}' skynet

#  hello-dock

docker network inspect --format='{{range .Containers}} {{.Name}} {{end}}' bridge

#  hello-dock

As you can see from the outputs of the two network inspect commands, the hello-dock container is now attached to both the skynet and the default bridge network.

The second way of attaching a container to a network is by using the --network option for the container run or container create commands. The generic syntax for the option is as follows:

--network <network identifier>

To run another hello-dock container attached to the same network, you can execute the following command:

docker container run --network skynet --rm --name alpine-box -it alpine sh

# lands you into alpine linux shell

/ # ping hello-dock

# PING hello-dock (172.18.0.2): 56 data bytes
# 64 bytes from 172.18.0.2: seq=0 ttl=64 time=0.191 ms
# 64 bytes from 172.18.0.2: seq=1 ttl=64 time=0.103 ms
# 64 bytes from 172.18.0.2: seq=2 ttl=64 time=0.139 ms
# 64 bytes from 172.18.0.2: seq=3 ttl=64 time=0.142 ms
# 64 bytes from 172.18.0.2: seq=4 ttl=64 time=0.146 ms
# 64 bytes from 172.18.0.2: seq=5 ttl=64 time=0.095 ms
# 64 bytes from 172.18.0.2: seq=6 ttl=64 time=0.181 ms
# 64 bytes from 172.18.0.2: seq=7 ttl=64 time=0.138 ms
# 64 bytes from 172.18.0.2: seq=8 ttl=64 time=0.158 ms
# 64 bytes from 172.18.0.2: seq=9 ttl=64 time=0.137 ms
# 64 bytes from 172.18.0.2: seq=10 ttl=64 time=0.145 ms
# 64 bytes from 172.18.0.2: seq=11 ttl=64 time=0.138 ms
# 64 bytes from 172.18.0.2: seq=12 ttl=64 time=0.085 ms

--- hello-dock ping statistics ---
13 packets transmitted, 13 packets received, 0% packet loss
round-trip min/avg/max = 0.085/0.138/0.191 ms

As you can see, running ping hello-dock from inside the alpine-box container works because both of the containers are under the same user-defined bridge network and automatic DNS resolution is working.

Keep in mind, though, that in order for the automatic DNS resolution to work you must assign custom names to the containers. Using the randomly generated name will not work.

How to Detach Containers from a Network in Docker

In the previous sub-section you learned about attaching containers to a network. In this sub-section, you'll learn about how to detach them.

You can use the network disconnect command for this task. The generic syntax for the command is as follows:

docker network disconnect <network identifier> <container identifier>

To detach the hello-dock container from the skynet network, you can execute the following command:

docker network disconnect skynet hello-dock

Just like the network connect command, the network disconnect command doesn't give any output.

How to Get Rid of Networks in Docker

Just like the other logical objects in Docker, networks can be removed using the network rm command. The generic syntax for the command is as follows:

docker network rm <network identifier>

To remove the skynet network from your system, you can execute the following command:

docker network rm skynet

You can also use the network prune command to remove any unused networks from your system. The command also has the -f or --force and -a or --all options.

How to Containerize a Multi-Container JavaScript Application

Now that you've learned enough about networks in Docker, in this section you'll learn to containerize a full-fledged multi-container project. The project you'll be working with is a simple notes-api powered by Express.js and PostgreSQL.

In this project there are two containers in total that you'll have to connect using a network. Apart from this, you'll also learn about concepts like environment variables and named volumes. So without further ado, let's jump right in.

How to Run the Database Server

The database server in this project is a simple PostgreSQL server and uses the official postgres image.

According to the official docs, in order to run a container with this image, you must provide the POSTGRES_PASSWORD environment variable. Apart from this one, I'll also provide a name for the default database using the POSTGRES_DB environment variable. PostgreSQL by default listens on port 5432, so you need to publish that as well.

To run the database server you can execute the following command:

docker container run \
    --detach \
    --name=notes-db \
    --env POSTGRES_DB=notesdb \
    --env POSTGRES_PASSWORD=secret \
    --network=notes-api-network \
    postgres:12

# a7b287d34d96c8e81a63949c57b83d7c1d71b5660c87f5172f074bd1606196dc

docker container ls

# CONTAINER ID   IMAGE         COMMAND                  CREATED              STATUS              PORTS      NAMES
# a7b287d34d96   postgres:12   "docker-entrypoint.s…"   About a minute ago   Up About a minute   5432/tcp   notes-db

The --env option for the container run and container create commands can be used for providing environment variables to a container. As you can see, the database container has been created successfully and is running now.

Although the container is running, there is a small problem. Databases like PostgreSQL, MongoDB, and MySQL persist their data in a directory. PostgreSQL uses the /var/lib/postgresql/data directory inside the container to persist data.

Now what if the container gets destroyed for some reason? You'll lose all your data. To solve this problem, a named volume can be used.

How to Work with Named Volumes in Docker

Previously you've worked with bind mounts and anonymous volumes. A named volume is very similar to an anonymous volume except that you can refer to a named volume using its name.

Volumes are also logical objects in Docker and can be manipulated using the command-line. The volume create command can be used for creating a named volume.

The generic syntax for the command is as follows:

docker volume create <volume name>

To create a volume named notes-db-data you can execute the following command:

docker volume create notes-db-data

# notes-db-data

docker volume ls

# DRIVER    VOLUME NAME
# local     notes-db-data

This volume can now be mounted to /var/lib/postgresql/data inside the notes-db container. To do so, stop and remove the notes-db container:

docker container stop notes-db

# notes-db

docker container rm notes-db

# notes-db

Now run a new container and assign the volume using the --volume or -v option.

docker container run \
    --detach \
    --volume notes-db-data:/var/lib/postgresql/data \
    --name=notes-db \
    --env POSTGRES_DB=notesdb \
    --env POSTGRES_PASSWORD=secret \
    --network=notes-api-network \
    postgres:12

# 37755e86d62794ed3e67c19d0cd1eba431e26ab56099b92a3456908c1d346791

Now inspect the notes-db container to make sure that the mounting was successful:

docker container inspect --format='{{range .Mounts}} {{ .Name }} {{end}}' notes-db

#  notes-db-data

Now the data will safely be stored inside the notes-db-data volume and can be reused in the future. A bind mount can also be used instead of a named volume here, but I prefer a named volume in such scenarios.

How to Access Logs from a Container in Docker

In order to see the logs from a container, you can use the container logs command. The generic syntax for the command is as follows:

docker container logs <container identifier>

To access the logs from the notes-db container, you can execute the following command:

docker container logs notes-db

# The files belonging to this database system will be owned by user "postgres".
# This user must also own the server process.

# The database cluster will be initialized with locale "en_US.utf8".
# The default database encoding has accordingly been set to "UTF8".
# The default text search configuration will be set to "english".
#
# Data page checksums are disabled.
#
# fixing permissions on existing directory /var/lib/postgresql/data ... ok
# creating subdirectories ... ok
# selecting dynamic shared memory implementation ... posix
# selecting default max_connections ... 100
# selecting default shared_buffers ... 128MB
# selecting default time zone ... Etc/UTC
# creating configuration files ... ok
# running bootstrap script ... ok
# performing post-bootstrap initialization ... ok
# syncing data to disk ... ok
#
#
# Success. You can now start the database server using:
#
#     pg_ctl -D /var/lib/postgresql/data -l logfile start
#
# initdb: warning: enabling "trust" authentication for local connections
# You can change this by editing pg_hba.conf or using the option -A, or
# --auth-local and --auth-host, the next time you run initdb.
# waiting for server to start....2021-01-25 13:39:21.613 UTC [47] LOG:  starting PostgreSQL 12.5 (Debian 12.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
# 2021-01-25 13:39:21.621 UTC [47] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
# 2021-01-25 13:39:21.675 UTC [48] LOG:  database system was shut down at 2021-01-25 13:39:21 UTC
# 2021-01-25 13:39:21.685 UTC [47] LOG:  database system is ready to accept connections
#  done
# server started
# CREATE DATABASE
#
#
# /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
#
# 2021-01-25 13:39:22.008 UTC [47] LOG:  received fast shutdown request
# waiting for server to shut down....2021-01-25 13:39:22.015 UTC [47] LOG:  aborting any active transactions
# 2021-01-25 13:39:22.017 UTC [47] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
# 2021-01-25 13:39:22.017 UTC [49] LOG:  shutting down
# 2021-01-25 13:39:22.056 UTC [47] LOG:  database system is shut down
#  done
# server stopped
#
# PostgreSQL init process complete; ready for start up.
#
# 2021-01-25 13:39:22.135 UTC [1] LOG:  starting PostgreSQL 12.5 (Debian 12.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
# 2021-01-25 13:39:22.136 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
# 2021-01-25 13:39:22.136 UTC [1] LOG:  listening on IPv6 address "::", port 5432
# 2021-01-25 13:39:22.147 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
# 2021-01-25 13:39:22.177 UTC [75] LOG:  database system was shut down at 2021-01-25 13:39:22 UTC
# 2021-01-25 13:39:22.190 UTC [1] LOG:  database system is ready to accept connections

Evident by the text in line 57, the database is up and ready to accept connections from the outside. There is also the --follow or -f option for the command which lets you attach the console to the logs output and get a continuous stream of text.

How to Create a Network and Attaching the Database Server in Docker

As you've learned in the previous section, the containers have to be attached to a user-defined bridge network in order to communicate with each other using container names. To do so, create a network named notes-api-network in your system:

docker network create notes-api-network

Now attach the notes-db container to this network by executing the following command:

docker network connect notes-api-network notes-db

How to Write the Dockerfile

Go to the directory where you've cloned the project code. Inside there, go inside the notes-api/api directory, and create a new Dockerfile. Put the following code in the file:

# stage one
FROM node:lts-alpine as builder

# install dependencies for node-gyp
RUN apk add --no-cache python make g++

WORKDIR /app

COPY ./package.json .
RUN npm install --only=prod

# stage two
FROM node:lts-alpine

EXPOSE 3000
ENV NODE_ENV=production

USER node
RUN mkdir -p /home/node/app
WORKDIR /home/node/app

COPY . .
COPY --from=builder /app/node_modules  /home/node/app/node_modules

CMD [ "node", "bin/www" ]

This is a multi-staged build. The first stage is used for building and installing the dependencies using node-gyp and the second stage is for running the application. I'll go through the steps briefly:

• Stage 1 uses node:lts-alpine as its base and uses builder as the stage name.
• On line 5, we install python, make, and g++. The node-gyp tool requires these three packages to run.
• On line 7, we set /app directory as the WORKDIR .
• On line 9 and 10, we copy the package.json file to the WORKDIR and install all the dependencies.
• Stage 2 also uses node-lts:alpine as the base.
• On line 16, we set the NODE_ENV environment variable to production. This is important for the API to run properly.
• From line 18 to line 20, we set the default user to node, create the /home/node/app directory, and set that as the WORKDIR.
• On line 22, we copy all the project files and on line 23 we copy the node_modules directory from the builder stage. This directory contains all the built dependencies necessary for running the application.
• On line 25, we set the default command.

To build an image from this Dockerfile, you can execute the following command:

docker image build --tag notes-api .

# Sending build context to Docker daemon  37.38kB
# Step 1/14 : FROM node:lts-alpine as builder
#  ---> 471e8b4eb0b2
# Step 2/14 : RUN apk add --no-cache python make g++
#  ---> Running in 5f20a0ecc04b
# fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
# fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
# (1/21) Installing binutils (2.33.1-r0)
# (2/21) Installing gmp (6.1.2-r1)
# (3/21) Installing isl (0.18-r0)
# (4/21) Installing libgomp (9.3.0-r0)
# (5/21) Installing libatomic (9.3.0-r0)
# (6/21) Installing mpfr4 (4.0.2-r1)
# (7/21) Installing mpc1 (1.1.0-r1)
# (8/21) Installing gcc (9.3.0-r0)
# (9/21) Installing musl-dev (1.1.24-r3)
# (10/21) Installing libc-dev (0.7.2-r0)
# (11/21) Installing g++ (9.3.0-r0)
# (12/21) Installing make (4.2.1-r2)
# (13/21) Installing libbz2 (1.0.8-r1)
# (14/21) Installing expat (2.2.9-r1)
# (15/21) Installing libffi (3.2.1-r6)
# (16/21) Installing gdbm (1.13-r1)
# (17/21) Installing ncurses-terminfo-base (6.1_p20200118-r4)
# (18/21) Installing ncurses-libs (6.1_p20200118-r4)
# (19/21) Installing readline (8.0.1-r0)
# (20/21) Installing sqlite-libs (3.30.1-r2)
# (21/21) Installing python2 (2.7.18-r0)
# Executing busybox-1.31.1-r9.trigger
# OK: 212 MiB in 37 packages
# Removing intermediate container 5f20a0ecc04b
#  ---> 637ca797d709
# Step 3/14 : WORKDIR /app
#  ---> Running in 846361b57599
# Removing intermediate container 846361b57599
#  ---> 3d58a482896e
# Step 4/14 : COPY ./package.json .
#  ---> 11b387794039
# Step 5/14 : RUN npm install --only=prod
#  ---> Running in 2e27e33f935d
#  added 269 packages from 220 contributors and audited 1137 packages in 140.322s
#
# 4 packages are looking for funding
#   run `npm fund` for details
#
# found 0 vulnerabilities
#
# Removing intermediate container 2e27e33f935d
#  ---> eb7cb2cb0b20
# Step 6/14 : FROM node:lts-alpine
#  ---> 471e8b4eb0b2
# Step 7/14 : EXPOSE 3000
#  ---> Running in 4ea24f871747
# Removing intermediate container 4ea24f871747
#  ---> 1f0206f2f050
# Step 8/14 : ENV NODE_ENV=production
#  ---> Running in 5d40d6ac3b7e
# Removing intermediate container 5d40d6ac3b7e
#  ---> 31f62da17929
# Step 9/14 : USER node
#  ---> Running in 0963e1fb19a0
# Removing intermediate container 0963e1fb19a0
#  ---> 0f4045152b1c
# Step 10/14 : RUN mkdir -p /home/node/app
#  ---> Running in 0ac591b3adbd
# Removing intermediate container 0ac591b3adbd
#  ---> 5908373dfc75
# Step 11/14 : WORKDIR /home/node/app
#  ---> Running in 55253b62ff57
# Removing intermediate container 55253b62ff57
#  ---> 2883cdb7c77a
# Step 12/14 : COPY . .
#  ---> 8e60893a7142
# Step 13/14 : COPY --from=builder /app/node_modules  /home/node/app/node_modules
#  ---> 27a85faa4342
# Step 14/14 : CMD [ "node", "bin/www" ]
#  ---> Running in 349c8ca6dd3e
# Removing intermediate container 349c8ca6dd3e
#  ---> 9ea100571585
# Successfully built 9ea100571585
# Successfully tagged notes-api:latest

Before you run a container using this image, make sure the database container is running, and is attached to the notes-api-network.

docker container inspect notes-db

# [
#     {
#         ...
#         "State": {
#             "Status": "running",
#             "Running": true,
#             "Paused": false,
#             "Restarting": false,
#             "OOMKilled": false,
#             "Dead": false,
#             "Pid": 11521,
#             "ExitCode": 0,
#             "Error": "",
#             "StartedAt": "2021-01-26T06:55:44.928510218Z",
#             "FinishedAt": "2021-01-25T14:19:31.316854657Z"
#         },
#         ...
#         "Mounts": [
#             {
#                 "Type": "volume",
#                 "Name": "notes-db-data",
#                 "Source": "/var/lib/docker/volumes/notes-db-data/_data",
#                 "Destination": "/var/lib/postgresql/data",
#                 "Driver": "local",
#                 "Mode": "z",
#                 "RW": true,
#                 "Propagation": ""
#             }
#         ],
#         ...
#         "NetworkSettings": {
#             ...
#             "Networks": {
#                 "bridge": {
#                     "IPAMConfig": null,
#                     "Links": null,
#                     "Aliases": null,
#                     "NetworkID": "e4c7ce50a5a2a49672155ff498597db336ecc2e3bbb6ee8baeebcf9fcfa0e1ab",
#                     "EndpointID": "2a2587f8285fa020878dd38bdc630cdfca0d769f76fc143d1b554237ce907371",
#                     "Gateway": "172.17.0.1",
#                     "IPAddress": "172.17.0.2",
#                     "IPPrefixLen": 16,
#                     "IPv6Gateway": "",
#                     "GlobalIPv6Address": "",
#                     "GlobalIPv6PrefixLen": 0,
#                     "MacAddress": "02:42:ac:11:00:02",
#                     "DriverOpts": null
#                 },
#                 "notes-api-network": {
#                     "IPAMConfig": {},
#                     "Links": null,
#                     "Aliases": [
#                         "37755e86d627"
#                     ],
#                     "NetworkID": "06579ad9f93d59fc3866ac628ed258dfac2ed7bc1a9cd6fe6e67220b15d203ea",
#                     "EndpointID": "5b8f8718ec9a5ec53e7a13cce3cb540fdf3556fb34242362a8da4cc08d37223c",
#                     "Gateway": "172.18.0.1",
#                     "IPAddress": "172.18.0.2",
#                     "IPPrefixLen": 16,
#                     "IPv6Gateway": "",
#                     "GlobalIPv6Address": "",
#                     "GlobalIPv6PrefixLen": 0,
#                     "MacAddress": "02:42:ac:12:00:02",
#                     "DriverOpts": {}
#                 }
#             }
#         }
#     }
# ]

I've shortened the output for easy viewing here. On my system, the notes-db container is running, uses the notes-db-data volume, and is attached to the notes-api-network bridge.

Once you're assured that everything is in place, you can run a new container by executing the following command:

docker container run \
    --detach \
    --name=notes-api \
    --env DB_HOST=notes-db \
    --env DB_DATABASE=notesdb \
    --env DB_PASSWORD=secret \
    --publish=3000:3000 \
    --network=notes-api-network \
    notes-api

# f9ece420872de99a060b954e3c236cbb1e23d468feffa7fed1e06985d99fb919

You should be able to understand this long command by yourself, so I'll go through the environment variables briefly.

The notes-api application requires three environment variables to be set. They are as follows:

• DB_HOST - This is the host of the database server. Given that both the database server and the API are attached to the same user-defined bridge network, the database server can be refereed to using its container name which is notes-db in this case.
• DB_DATABASE - The database that this API will use. On Running the Database Server we set the default database name to notesdb using the POSTGRES_DB environment variable. We'll use that here.
• DB_PASSWORD - Password for connecting to the database. This was also set on Running the Database Server sub-section using the POSTGRES_PASSWORD environment variable.

To check if the container is running properly or not, you can use the container ls command:

docker container ls

# CONTAINER ID   IMAGE         COMMAND                  CREATED          STATUS          PORTS                    NAMES
# f9ece420872d   notes-api     "docker-entrypoint.s…"   12 minutes ago   Up 12 minutes   0.0.0.0:3000->3000/tcp   notes-api
# 37755e86d627   postgres:12   "docker-entrypoint.s…"   17 hours ago     Up 14 minutes   5432/tcp                 notes-db

The container is running now. You can visit http://127.0.0.1:3000/ to see the API in action.

The API has five routes in total that you can see inside the /notes-api/api/api/routes/notes.js file.

Although the container is running, there is one last thing that you'll have to do before you can start using it. You'll have to run the database migration necessary for setting up the database tables, and you can do that by executing npm run db:migrate command inside the container.

How to Execute Commands in a Running Container

You've already learned about executing commands in a stopped container. Another scenario is executing a command inside a running container.

For this, you'll have to use the exec command to execute a custom command inside a running container.

The generic syntax for the exec command is as follows:

docker container exec <container identifier> <command>

To execute npm run db:migrate inside the notes-api container, you can execute the following command:

docker container exec notes-api npm run db:migrate

# > notes-api@ db:migrate /home/node/app
# > knex migrate:latest
#
# Using environment: production
# Batch 1 run: 1 migrations

In cases where you want to run an interactive command inside a running container, you'll have to use the -it flag. As an example, if you want to access the shell running inside the notes-api container, you can execute following the command:

docker container exec -it notes-api sh

# / # uname -a
# Linux b5b1367d6b31 5.10.9-201.fc33.x86_64 #1 SMP Wed Jan 20 16:56:23 UTC 2021 x86_64 Linux

How to Write Management Scripts in Docker

Managing a multi-container project along with the network and volumes and stuff means writing a lot of commands. To simplify the process, I usually have help from simple shell scripts and a Makefile.

You'll find four shell scripts in the notes-api directory. They are as follows:

• boot.sh - Used for starting the containers if they already exist.
• build.sh - Creates and runs the containers. It also creates the images, volumes, and networks if necessary.
• destroy.sh - Removes all containers, volumes and networks associated with this project.
• stop.sh - Stops all running containers.

There is also a Makefile that contains four targets named start, stop, build and destroy, each invoking the previously mentioned shell scripts.

If the container is in a running state in your system, executing make stop should stop all the containers. Executing make destroy should stop the containers and remove everything. Make sure you're running the scripts inside the notes-api directory:

make destroy

# ./shutdown.sh
# stopping api container --->
# notes-api
# api container stopped --->

# stopping db container --->
# notes-db
# db container stopped --->

# shutdown script finished

# ./destroy.sh
# removing api container --->
# notes-api
# api container removed --->

# removing db container --->
# notes-db
# db container removed --->

# removing db data volume --->
# notes-db-data
# db data volume removed --->

# removing network --->
# notes-api-network
# network removed --->

# destroy script finished

If you're getting a permission denied error, than execute chmod +x on the scripts:

chmod +x boot.sh build.sh destroy.sh shutdown.sh

I'm not going to explain these scripts because they're simple if-else statements along with some Docker commands that you've already seen many times. If you have some understanding of the Linux shell, you should be able to understand the scripts as well.

How to Compose Projects Using Docker-Compose

In the previous section, you've learned about managing a multi-container project and the difficulties of it. Instead of writing so many commands, there is an easier way to manage multi-container projects, a tool called Docker Compose.

According to the Docker documentation -

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Although Compose works in all environments, it's more focused on development and testing. Using Compose on a production environment is not recommended at all.

Docker Compose Basics

Go the directory where you've cloned the repository that came with this book. Go inside the notes-api/api directory and create a Dockerfile.dev file. Put the following code in it:

# stage one
FROM node:lts-alpine as builder

# install dependencies for node-gyp
RUN apk add --no-cache python make g++

WORKDIR /app

COPY ./package.json .
RUN npm install

# stage two
FROM node:lts-alpine

ENV NODE_ENV=development

USER node
RUN mkdir -p /home/node/app
WORKDIR /home/node/app

COPY . .
COPY --from=builder /app/node_modules /home/node/app/node_modules

CMD [ "./node_modules/.bin/nodemon", "--config", "nodemon.json", "bin/www" ]

The code is almost identical to the Dockerfile that you worked with in the previous section. The three differences in this file are as follows:

• On line 10, we run npm install instead of npm run install --only=prod because we want the development dependencies also.
• On line 15, we set the NODE_ENV environment variable to development instead of production.
• On line 24, we use a tool called nodemon to get the hot-reload feature for the API.

You already know that this project has two containers:

• notes-db - A database server powered by PostgreSQL.
• notes-api - A REST API powered by Express.js

In the world of Compose, each container that makes up the application is known as a service. The first step in composing a multi-container project is to define these services.

Just like the Docker daemon uses a Dockerfile for building images, Docker Compose uses a docker-compose.yaml file to read service definitions from.

Head to the notes-api directory and create a new docker-compose.yaml file. Put the following code into the newly created file:

version: "3.8"

services: 
    db:
        image: postgres:12
        container_name: notes-db-dev
        volumes: 
            - notes-db-dev-data:/var/lib/postgresql/data
        environment:
            POSTGRES_DB: notesdb
            POSTGRES_PASSWORD: secret
    api:
        build:
            context: ./api
            dockerfile: Dockerfile.dev
        image: notes-api:dev
        container_name: notes-api-dev
        environment: 
            DB_HOST: db ## same as the database service name
            DB_DATABASE: notesdb
            DB_PASSWORD: secret
        volumes: 
            - /home/node/app/node_modules
            - ./api:/home/node/app
        ports: 
            - 3000:3000

volumes:
    notes-db-dev-data:
        name: notes-db-dev-data

Every valid docker-compose.yaml file starts by defining the file version. At the time of writing, 3.8 is the latest version. You can look up the latest version here.

Blocks in an YAML file are defined by indentation. I will go through each of the blocks and will explain what they do.

• The services block holds the definitions for each of the services or containers in the application. db and api are the two services that comprise this project.
• The db block defines a new service in the application and holds necessary information to start the container. Every service requires either a pre-built image or a Dockerfile to run a container. For the db service we're using the official PostgreSQL image.
• Unlike the db service, a pre-built image for the api service doesn't exist. So we'll use the Dockerfile.dev file.
• The volumes block defines any name volume needed by any of the services. At the time it only enlists notes-db-dev-data volume used by the db service.

Now that have a high level overview of the docker-compose.yaml file, let's have a closer look at the individual services.

The definition code for the db service is as follows:

db:
    image: postgres:12
    container_name: notes-db-dev
    volumes: 
        - db-data:/var/lib/postgresql/data
    environment:
        POSTGRES_DB: notesdb
        POSTGRES_PASSWORD: secret

• The image key holds the image repository and tag used for this container. We're using the postgres:12 image for running the database container.
• The container_name indicates the name of the container. By default containers are named following <project directory name>_<service name> syntax. You can override that using container_name.
• The volumes array holds the volume mappings for the service and supports named volumes, anonymous volumes, and bind mounts. The syntax <source>:<destination> is identical to what you've seen before.
• The environment map holds the values of the various environment variables needed for the service.

Definition code for the api service is as follows:

api:
    build:
        context: ./api
        dockerfile: Dockerfile.dev
    image: notes-api:dev
    container_name: notes-api-dev
    environment: 
        DB_HOST: db ## same as the database service name
        DB_DATABASE: notesdb
        DB_PASSWORD: secret
    volumes: 
        - /home/node/app/node_modules
        - ./api:/home/node/app
    ports: 
        - 3000:3000

• The api service doesn't come with a pre-built image. Instead it has a build configuration. Under the build block we define the context and the name of the Dockerfile for building an image. You should have an understanding of context and Dockerfile by now so I won't spend time explaining those.
• The image key holds the name of the image to be built. If not assigned, the image will be named following the <project directory name>_<service name> syntax.
• Inside the environment map, the DB_HOST variable demonstrates a feature of Compose. That is, you can refer to another service in the same application by using its name. So the db here, will be replaced by the IP address of the api service container. The DB_DATABASE and DB_PASSWORD variables have to match up with POSTGRES_DB and POSTGRES_PASSWORD respectively from the db service definition.
• In the volumes map, you can see an anonymous volume and a bind mount described. The syntax is identical to what you've seen in previous sections.
• The ports map defines any port mapping. The syntax, <host port>:<container port> is identical to the --publish option you used before.

Finally, the code for the volumes is as follows:

volumes:
    db-data:
        name: notes-db-dev-data

Any named volume used in any of the services has to be defined here. If you don't define a name, the volume will be named following the <project directory name>_<volume key> and the key here is db-data.

You can learn about the different options for volume configuration in the official docs.

How to Start Services in Docker Compose

There are a few ways of starting services defined in a YAML file. The first command that you'll learn about is the up command. The up command builds any missing images, creates containers, and starts them in one go.

Before you execute the command, though, make sure you've opened your terminal in the same directory where the docker-compose.yaml file is. This is very important for every docker-compose command you execute.

docker-compose --file docker-compose.yaml up --detach

# Creating network "notes-api_default" with the default driver
# Creating volume "notes-db-dev-data" with default driver
# Building api
# Sending build context to Docker daemon  37.38kB
#
# Step 1/13 : FROM node:lts-alpine as builder
#  ---> 471e8b4eb0b2
# Step 2/13 : RUN apk add --no-cache python make g++
#  ---> Running in 197056ec1964
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container 197056ec1964
#  ---> 6609935fe50b
# Step 3/13 : WORKDIR /app
#  ---> Running in 17010f65c5e7
# Removing intermediate container 17010f65c5e7
#  ---> b10d12e676ad
# Step 4/13 : COPY ./package.json .
#  ---> 600d31d9362e
# Step 5/13 : RUN npm install
#  ---> Running in a14afc8c0743
### LONG INSTALLATION STUFF GOES HERE ###
#  Removing intermediate container a14afc8c0743
#  ---> 952d5d86e361
# Step 6/13 : FROM node:lts-alpine
#  ---> 471e8b4eb0b2
# Step 7/13 : ENV NODE_ENV=development
#  ---> Running in 0d5376a9e78a
# Removing intermediate container 0d5376a9e78a
#  ---> 910c081ce5f5
# Step 8/13 : USER node
#  ---> Running in cfaefceb1eff
# Removing intermediate container cfaefceb1eff
#  ---> 1480176a1058
# Step 9/13 : RUN mkdir -p /home/node/app
#  ---> Running in 3ae30e6fb8b8
# Removing intermediate container 3ae30e6fb8b8
#  ---> c391cee4b92c
# Step 10/13 : WORKDIR /home/node/app
#  ---> Running in 6aa27f6b50c1
# Removing intermediate container 6aa27f6b50c1
#  ---> 761a7435dbca
# Step 11/13 : COPY . .
#  ---> b5d5c5bdf3a6
# Step 12/13 : COPY --from=builder /app/node_modules /home/node/app/node_modules
#  ---> 9e1a19960420
# Step 13/13 : CMD [ "./node_modules/.bin/nodemon", "--config", "nodemon.json", "bin/www" ]
#  ---> Running in 5bdd62236994
# Removing intermediate container 5bdd62236994
#  ---> 548e178f1386
# Successfully built 548e178f1386
# Successfully tagged notes-api:dev
# Creating notes-api-dev ... done
# Creating notes-db-dev  ... done

The --detach or -d option here functions the same as the one you've seen before. The --file or -f option is only needed if the YAML file is not named docker-compose.yaml (but I've used here for demonstration purposes).

Apart from the the up command there is the start command. The main difference between these two is that the start command doesn't create missing containers, only starts existing containers. It's basically the same as the container start command.

The --build option for the up command forces a rebuild of the images. There are some other options for the up command that you can see in the official docs.

How to List Services in Docker Compose

Although service containers started by Compose can be listed using the container ls command, there is the ps command for listing containers defined in the YAML only.

docker-compose ps

#     Name                   Command               State           Ports         
# -------------------------------------------------------------------------------
# notes-api-dev   docker-entrypoint.sh ./nod ...   Up      0.0.0.0:3000->3000/tcp
# notes-db-dev    docker-entrypoint.sh postgres    Up      5432/tcp

It's not as informative as the container ls output, but it's useful when you have tons of containers running simultaneously.

How to Execute Commands Inside a Running Service in Docker Compose

I hope you remember from the previous section that you have to run some migration scripts to create the database tables for this API.

Just like the container exec command, there is an exec command for docker-compose. Generic syntax for the command is as follows:

docker-compose exec <service name> <command>

To execute the npm run db:migrate command inside the api service, you can execute the following command:

docker-compose exec api npm run db:migrate

# > notes-api@ db:migrate /home/node/app
# > knex migrate:latest
# 
# Using environment: development
# Batch 1 run: 1 migrations

Unlike the container exec command, you don't need to pass the -it flag for interactive sessions. docker-compose does that automatically.

How to Access Logs from a Running Service in Docker Compose

You can also use the logs command to retrieve logs from a running service. The generic syntax for the command is as follows:

docker-compose logs <service name>

To access the logs from the api service, execute the following command:

docker-compose logs api

# Attaching to notes-api-dev
# notes-api-dev | [nodemon] 2.0.7
# notes-api-dev | [nodemon] reading config ./nodemon.json
# notes-api-dev | [nodemon] to restart at any time, enter `rs`
# notes-api-dev | [nodemon] or send SIGHUP to 1 to restart
# notes-api-dev | [nodemon] ignoring: *.test.js
# notes-api-dev | [nodemon] watching path(s): *.*
# notes-api-dev | [nodemon] watching extensions: js,mjs,json
# notes-api-dev | [nodemon] starting `node bin/www`
# notes-api-dev | [nodemon] forking
# notes-api-dev | [nodemon] child pid: 19
# notes-api-dev | [nodemon] watching 18 files
# notes-api-dev | app running -> http://127.0.0.1:3000

This is just a portion from the log output. You can kind of hook into the output stream of the service and get the logs in real-time by using the -f or --follow option. Any later log will show up instantly in the terminal as long as you don't exit by pressing ctrl + c or closing the window. The container will keep running even if you exit out of the log window.

How to Stop Services in Docker Compose

To stop services, there are two approaches that you can take. The first one is the down command. The down command stops all running containers and removes them from the system. It also removes any networks:

docker-compose down --volumes

# Stopping notes-api-dev ... done
# Stopping notes-db-dev  ... done
# Removing notes-api-dev ... done
# Removing notes-db-dev  ... done
# Removing network notes-api_default
# Removing volume notes-db-dev-data

The --volumes option indicates that you want to remove any named volume(s) defined in the volumes block. You can learn about the additional options for the down command in the official docs.

Another command for stopping services is the stop command which functions identically to the container stop command. It stops all the containers for the application and keeps them. These containers can later be started with the start or up command.

How to Compose a Full-stack Application in Docker Compose

In this sub-section, we'll be adding a front-end to our notes API and turning it into a complete full-stack application. I won't be explaining any of the Dockerfile.dev files in this sub-section (except the one for the nginx service) as they are identical to some of the others you've already seen in previous sub-sections.‌

If you've cloned the project code repository, then go inside the fullstack-notes-application directory. Each directory inside the project root contains the code for each service and the corresponding Dockerfile.‌

Before we start with the docker-compose.yaml file let's look at a diagram of how the application is going to work:

Instead of accepting requests directly like we previously did, in this application all the requests will be first received by an NGINX (lets call it router) service.

The router will then see if the requested end-point has /api in it. If yes, the router will route the request to the back-end or if not, the router will route the request to the front-end.

You do this because when you run a front-end application it doesn't run inside a container. It runs on the browser, served from a container. As a result, Compose networking doesn't work as expected and the front-end application fails to find the api service.

NGINX, on the other hand, runs inside a container and can communicate with the different services across the entire application.

I will not get into the configuration of NGINX here. That topic is kinda out of the scope of this book. But if you want to have a look at it, go ahead and check out the /notes-api/nginx/development.conf and /notes-api/nginx/production.conf files. Code for the /notes-api/nginx/Dockerfile.dev is as follows:

FROM nginx:stable-alpine

COPY ./development.conf /etc/nginx/conf.d/default.conf

All it does is copy the configuration file to /etc/nginx/conf.d/default.conf inside the container.

Let's start writing the docker-compose.yaml file. Apart from the api and db services there will be the client and nginx services. There will also be some network definitions that I'll get into shortly.

version: "3.8"

services: 
    db:
        image: postgres:12
        container_name: notes-db-dev
        volumes: 
            - db-data:/var/lib/postgresql/data
        environment:
            POSTGRES_DB: notesdb
            POSTGRES_PASSWORD: secret
        networks:
            - backend
    api:
        build: 
            context: ./api
            dockerfile: Dockerfile.dev
        image: notes-api:dev
        container_name: notes-api-dev
        volumes: 
            - /home/node/app/node_modules
            - ./api:/home/node/app
        environment: 
            DB_HOST: db ## same as the database service name
            DB_PORT: 5432
            DB_USER: postgres
            DB_DATABASE: notesdb
            DB_PASSWORD: secret
        networks:
            - backend
    client:
        build:
            context: ./client
            dockerfile: Dockerfile.dev
        image: notes-client:dev
        container_name: notes-client-dev
        volumes: 
            - /home/node/app/node_modules
            - ./client:/home/node/app
        networks:
            - frontend
    nginx:
        build:
            context: ./nginx
            dockerfile: Dockerfile.dev
        image: notes-router:dev
        container_name: notes-router-dev
        restart: unless-stopped
        ports: 
            - 8080:80
        networks:
            - backend
            - frontend

volumes:
    db-data:
        name: notes-db-dev-data

networks: 
    frontend:
        name: fullstack-notes-application-network-frontend
        driver: bridge
    backend:
        name: fullstack-notes-application-network-backend
        driver: bridge

The file is almost identical to the previous one you worked with. The only thing that needs some explanation is the network configuration. The code for the networks block is as follows:

networks: 
    frontend:
        name: fullstack-notes-application-network-frontend
        driver: bridge
    backend:
        name: fullstack-notes-application-network-backend
        driver: bridge

I've defined two bridge networks. By default, Compose creates a bridge network and attaches all containers to that. In this project, however, I wanted proper network isolation. So I defined two networks, one for the front-end services and one for the back-end services.

I've also added networks block in each of the service definitions. This way the the api and db service will be attached to one network and the client service will be attached to a separate network. But the nginx service will be attached to both the networks so that it can perform as router between the front-end and back-end services.

Start all the services by executing the following command:

docker-compose --file docker-compose.yaml up --detach

# Creating network "fullstack-notes-application-network-backend" with driver "bridge"
# Creating network "fullstack-notes-application-network-frontend" with driver "bridge"
# Creating volume "notes-db-dev-data" with default driver
# Building api
# Sending build context to Docker daemon  37.38kB
# 
# Step 1/13 : FROM node:lts-alpine as builder
#  ---> 471e8b4eb0b2
# Step 2/13 : RUN apk add --no-cache python make g++
#  ---> Running in 8a4485388fd3
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container 8a4485388fd3
#  ---> 47fb1ab07cc0
# Step 3/13 : WORKDIR /app
#  ---> Running in bc76cc41f1da
# Removing intermediate container bc76cc41f1da
#  ---> 8c03fdb920f9
# Step 4/13 : COPY ./package.json .
#  ---> a1d5715db999
# Step 5/13 : RUN npm install
#  ---> Running in fabd33cc0986
### LONG INSTALLATION STUFF GOES HERE ###
# Removing intermediate container fabd33cc0986
#  ---> e09913debbd1
# Step 6/13 : FROM node:lts-alpine
#  ---> 471e8b4eb0b2
# Step 7/13 : ENV NODE_ENV=development
#  ---> Using cache
#  ---> b7c12361b3e5
# Step 8/13 : USER node
#  ---> Using cache
#  ---> f5ac66ca07a4
# Step 9/13 : RUN mkdir -p /home/node/app
#  ---> Using cache
#  ---> 60094b9a6183
# Step 10/13 : WORKDIR /home/node/app
#  ---> Using cache
#  ---> 316a252e6e3e
# Step 11/13 : COPY . .
#  ---> Using cache
#  ---> 3a083622b753
# Step 12/13 : COPY --from=builder /app/node_modules /home/node/app/node_modules
#  ---> Using cache
#  ---> 707979b3371c
# Step 13/13 : CMD [ "./node_modules/.bin/nodemon", "--config", "nodemon.json", "bin/www" ]
#  ---> Using cache
#  ---> f2da08a5f59b
# Successfully built f2da08a5f59b
# Successfully tagged notes-api:dev
# Building client
# Sending build context to Docker daemon  43.01kB
# 
# Step 1/7 : FROM node:lts-alpine
#  ---> 471e8b4eb0b2
# Step 2/7 : USER node
#  ---> Using cache
#  ---> 4be5fb31f862
# Step 3/7 : RUN mkdir -p /home/node/app
#  ---> Using cache
#  ---> 1fefc7412723
# Step 4/7 : WORKDIR /home/node/app
#  ---> Using cache
#  ---> d1470d878aa7
# Step 5/7 : COPY ./package.json .
#  ---> Using cache
#  ---> bbcc49475077
# Step 6/7 : RUN npm install
#  ---> Using cache
#  ---> 860a4a2af447
# Step 7/7 : CMD [ "npm", "run", "serve" ]
#  ---> Using cache
#  ---> 11db51d5bee7
# Successfully built 11db51d5bee7
# Successfully tagged notes-client:dev
# Building nginx
# Sending build context to Docker daemon   5.12kB
# 
# Step 1/2 : FROM nginx:stable-alpine
#  ---> f2343e2e2507
# Step 2/2 : COPY ./development.conf /etc/nginx/conf.d/default.conf
#  ---> Using cache
#  ---> 02a55d005a98
# Successfully built 02a55d005a98
# Successfully tagged notes-router:dev
# Creating notes-client-dev ... done
# Creating notes-api-dev    ... done
# Creating notes-router-dev ... done
# Creating notes-db-dev     ... done

Now visit http://localhost:8080 and voilà!

Try adding and deleting notes to see if the application works properly. The project also comes with shell scripts and a Makefile. Explore them to see how you can run this project without the help of docker-compose like you did in the previous section.

Conclusion

I would like to thank you from the bottom of my heart for the time you've spent reading this book. I hope you've enjoyed it and have learned all the essentials of Docker.

Apart from this one, I've written full-length handbooks on other complicated topics available for free on freeCodeCamp.

These handbooks are part of my mission to simplify hard to understand technologies for everyone. Each of these handbooks takes a lot of time and effort to write.

If you've enjoyed my writing and want to keep me motivated, consider leaving starts on GitHub and endorse me for relevant skills on LinkedIn. I also accept sponsorship so you may consider buying me a coffee if you want to.

I'm always open to suggestions and discussions on Twitter or LinkedIn. Hit me with direct messages.

In the end, consider sharing the resources with others, because

Sharing knowledge is the most fundamental act of friendship. Because it is a way you can give something without loosing something. — Richard Stallman

Till the next one, stay safe and keep learning.

In this post you'll learn how to configure a development environment with live-reload enabled. This will allow you to convert a legacy application so it uses Docker, Docker volumes, and docker-compose.

Some developers turn up their noses when talking about using Docker for their development environment. They say that Docker isn't good for development because it always needs to rebuild the entire image to reflect all new modifications. This makes it unproductive and slow.

In this article, our goal is to tackle this mindset by demonstrating how simple configurations can result in many benefits such as a reliable environment over production and development environments.

By the end of this post you will have learned how to:

• Convert a legacy application to run within a Docker container;
• Enable dependency caching on Node.js modules;
• Enable live-reload by using docker volumes;
• Aggregate all services within docker-compose.

Requirements

In the next steps, you'll clone an existing project to execute all examples in this article. Before starting to code make sure you have the following tools installed on your machine:

• Docker and Docker compose
• Node.js 10+
• Git

Why use docker?

More and more cutting-edge technologies are being released for the internet all the time. They're stable, and they're fun to develop and release, but they're not predictable when working over different environments. So developers created Docker to help reduce the chances of possible errors.

Docker is one of my favorite tools that I work with every day on desktop, web, and IoT apps. It has given me the power to not only move applications through different environments, but also to keep my local environment as clean as possible.

Developers working with cutting-edge technologies are always working with something new. But what about legacy applications? Should we just rewrite everything with new tech? I know this is not as simple as it seems. We should work on new stuff, but also make improvements to existing applications.

Let's say you have decided to move from Windows Servers to Unix servers. How would you do it? Do you know every dependency your app requires to work?

What should a development environment look like?

Developers have always tried to be more productive by adding plugins, boilerplates, and codebases on their editors/IDEs/terminals. The best environment in my opinion should be:

1. Easy to run and test;
2. Environment agnostic;
3. Fast to evaluate modifications;
4. Easy to replicate on any machine.

Following these principles, we'll configure an application over the next sections of this article. Also, if you've never heard about live-reload (or hot reload), it is a feature that watches for changes in your code and restarts the server if needed. So you don't need to go back and forth, restarting your app or even rebuilding the system.

Getting started

First, you'll need to have an empty folder called post-docker-livereload which you'll use as a workspace. Go to the Github repository and clone it on your post-docker-live-reload folder.

Secondly, let's analyse what the application requires. If you take a look at the README.md file, there are a few instructions demonstrating how to run this app as shown in the image below:

It requires Node.js version 10 or higher and MongoDB. Instead of installing MongoDB on your local environment machine, you'll install it using Docker. You'll also expose it on localhost:27017 so applications that are not running through Docker can access it without knowing the internal Docker IP address.

Copy the command below and paste it in your terminal:

docker run --name mongodb -p 27017:27017 -d mongo:4

Using the command above, it'll download and run the MongoDB instance. Notice that if you already have an instance with this name it'll throw an error about invalid name.

If you see the error, run docker rm mongodb and it will remove any previous instance so you can run the docker run command again.

Digging into the application

The README.md file says that you need a MongoDB instance running before starting your app, along with Node.js.

If you have Node.js installed, go to the nodejs-with-mongodb-api-example folder and run the following commands:

npm i 
npm run build 
npm start

After running these commands, you can go to a browser on http://localhost:3000 and see the application running as shown in the image below:

Keep in mind that the app already has a command to enable live reload which is npm run dev:watch. The pipeline should reflect the following steps:

1. Developer changes Typescript files;
2. Typescript transpiles files to Javascript;
3. Server notices changes to Javascript and restarts the Node.js server.

So mirroring files to Docker containers will reflect all changes in the container. The npm run build:watch from the application will catch the changes and generate output files in the lib folder so the npm run dev:run will restart the server every time it has been triggered.

Dockerizing applications

If Docker is a completely new world to you, don't be afraid! You will configure it from scratch. You'll need to create a few files to start:

1. Dockerfile - a receipt file that lists how to install and run your app;
2. .dockerignore - a file that lists what file(s) won't go within the Docker container instance.

Creating the Dockerfile

The Dockerfile is the key concept here. There you specify the steps and dependencies to prepare and run the application. As long as you have read the README.md file, it will be easy to implement the receipt file.

I'm going to put the whole file below and dig into it later. In your nodejs-with-mongodb-api-example folder create a Dockerfile file and paste the code below:

FROM node:14-alpine

WORKDIR /src

ADD package.json /src 

RUN npm i --silent

ADD . /src 

RUN npm run build 

CMD npm start

What's happening there?

• On line 1 - It uses as its image base Node.js 14 - alpine version;
• From lines 2 to 4 - It copies and installs Node.js dependencies from host to container. Note that the order there is important. Adding package.json to the src folder before restoring dependencies will cache it and prevent it from installing packages every time you need to build your image;
• From lines 6 to 7 - It runs commands for the compilation process and then for starting the program as mentioned in the README.md file.

Ignoring unnecessary files with .dockerignore

Also, I'm working on an OSX-based system and the Docker container will run on a Linux Alpine-based system. When you run npm install it will restore dependencies for specific environments.

You will now create a file to ignore the generated code from your local machine such as nodemodules and lib._ So when you copy all files from the current directory to the container it won't get invalid package versions.

In the nodejs-with-mongodb-api-example folder create a .dockerignore file and paste the code below:

node_modules/
lib/

Building the docker image

I prefer running this app from the rootFolder. Go back to the post-docker-live-reload folder and run the following commands to prepare an image for further use:

docker build -t app nodejs-with-mongodb-api-example

Notice that the command above uses the -t flag to tell you the image name and, right after that, the folder which contains the Dockerfile file.

Working with volumes

Before running the app, let's do a few hacks to improve our experience in the Docker containers.

Docker volumes is a feature to mirror files through your local machine and Docker environment. You can also share volumes over containers and reuse them to cache dependencies.

Your goal is to watch any changes on local .ts files and mirror those changes in the container. Even though the files and the node_modules folder are in the same path.

Do you remember when I said that the dependencies in each operating system would be different? To make sure our local environment won't affect the Docker environment when mirroring files, we'll isolate the container's node_modules folder on a distinct volume.

Consequently, when creating the node_modules folder on the container, it won't create the folder on local machine environment. Run the command below in your terminal to create it:

docker volume create --name nodemodules

Running and enabling live-reload

As you know, the npm run dev:watch specified in the README.md shows how to enable live-reload. The problem is that you're coding on a local machine and it must reflect directly your container.

By running the following commands you will link your local environment with the Docker container so any change in nodejs-with-mongodb-api-example will affect the container's src folder.

docker run \
    --name app \
    --link mongodb \
    -e MONGO_URL=mongodb \
    -e PORT=4000 \
    -p 4000:4000 \
    -v `pwd`/nodejs-with-mongodb-api-example:/src \
    -v nodemodules:/src/node_modules \
    app npm run dev:watch

Let's dig into what's happening there:

• --link is giving permission to the app to access the MongoDB instance;
• -e - is the environment variables. As mentioned in the README.md file you can specify the MongoDB connection string you want to connect to by overriding the MONGO_URL variable. Override the PORT variable if you want to run it on a different port. Note that the value mongodb is the same name we used to create our MongoDB instance in the previous sections. This value is also an alias for the internal MongoDB instance's IP;
• -v - maps the current directory to the Docker container by using a virtual volume. Using the pwd command you can get the absolute path for your current working directory and then the folder you want to mirror on the Docker container. There's the :/src. The src path is the WORKDIR instruction defined on Dockerfile so we mirror the local folder to the Docker container's src;
• -v - the second volume there will mirror the individual volume we created on the container's node_modules folder;
• app - the image name;
• npm run dev:watch - this last command will override the CMD instruction from the Dockerfile.

After running the command below, you can trigger the browser after changing the local index.ts file to see the results. The video below demonstrates these steps in practice:

Wrapping up

You know that working with shell commands works. But is not that common to use them here, and it's not productive for running all those commands, building images, and managing instances by hand. So compose it!

Docker compose is a way to simplify the aggregation and linking of services. You can specify the databases, logs, application, volumes, networks, and so on.

First, you need to remove all active instances to avoid conflict on exposed ports. Run the following commands on your terminal to remove volumes, services and containers:

docker rm app 
docker volume rm nodemodules
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

The docker-compose file

Create a docker-compose.yml file on your post-docker-livereload folder using the data below:

version: '3'
services:
    mongodb:
        image: mongo:4
        ports:
            - 27017:27017
    app:
        build: nodejs-with-mongodb-api-example
        command: npm run dev:watch
        ports:
            - 4000:4000
        environment: 
            MONGO_URL: mongodb
            PORT: 4000
        volumes:
            - ./nodejs-with-mongodb-api-example:/src/
            - nodemodules:/src/node_modules
        links:
            - mongodb
        depends_on: 
            - mongodb

volumes:
    nodemodules: {}

The file above specifies resources by sections. Notice that you have links and depends_on sections there. The links field is the same flag you've used in your shell command. The depends_on will make sure that the MongoDB is a dependency for running your app. It'll run the MongoDB before your app like magic!

Going back to your terminal, to start all services and build the Docker image and create volumes and link services run the following command:

docker-compose up --build

If you need to remove all services created before by that Dockerfile you can also run docker-compose down.

Docker is your friend!

That's right, my friend. Docker can help you prevent many possible errors before they happen. You can use it for front and back end applications. Even for IoT when you need to control hardware, you can specify policies for using it.

As your next steps, I highly recommend that you take a look at container orchestrators such as Kubernetes and Docker swarm. They could improve even more your existing applications and help you go to the next level.

Thank you for reading

I really appreciate the time we spent together. I hope this content will be more than just text. I hope it will help make you a better thinker and also a better programmer. Follow me on Twitter and check out my personal blog where I share all my valuable content.

See ya! ?

Automated end to end testing is the most effective way to test your app. It also requires the least effort to get the benefit of tests if you currently have no tests at all. And you don’t need a ton of infrastructure or cloud servers to get there. In this guide we’ll see how we can easily overcome the two main hurdles of end to end testing.

The first hurdle is Selenium. The API you use to write tests is ugly and non-intuitive. But it is not that difficult to use, and with a few convenient functions it can become a breeze to write Selenium tests. The effort is well-rewarded because you can automatically test your end users' flows end to end.

The second hurdle is putting components together in an isolated environment. We want the frontend, the backend, the database and everything else your app uses. We will use Docker compose to put things together and automate the tests. It is easy even if you have your components in different Git repositories.

Writing end to end tests in Selenium

Even if you are an API only business, you have a frontend, and an admin or back office frontend. So end to end tests ultimately talk to a frontend application.

The industry standard tool is Selenium. Selenium provides an API to talk to the web browser and interact with the DOM. You can check what elements are displayed, fill inputs and click around. Anything a real user would do with your application, you can automate.

Selenium uses something called the WebDriver API. It is not very handy to use at first glance. But the learning curve is not steep. Creating a few convenience functions will get you productive in no time. I won’t go into the details of the WebDriver API here. You can have a look at this excellent article to dig deeper.

There are also libraries to make your life easier. Nightwatch is the most popular.

If you have an Angular application, Protractor is your best friend. It integrates with the Angular event loop and allows you to use selectors based on your model. That is gold.

Writing a test for your most critical user feature or your app should take only a few hours, so go ahead. It will run automatically ever after. Let's see how.

Running your tests in Docker

We need to run our tests in an isolated environment so the outcome is predictable. And so we can enable Continuous Integration easily. We'll use Docker compose.

Selenium provides Docker images out of the box to test with one or several browsers. The images spawn a Selenium server and a browser underneath. It can work with different browsers.

Let’s start with one browser for now: headless-chrome. You can see the docker-compose.yml file below (the commands are from an Angular example).

Note: If you've never used Docker you can easily install it on your computer. Docker has the troublesome tendency of forcing you to sign up for an account just to download the thing. But you actually don't have to. Go to the release notes (link for Windows and link for Mac) and download not the latest version but the one right before. This is a direct download link.

version: '3.1'

services:
 app-serve:
   build: .
   image: myapp
   command: npm run serve:production
   expose:
    - 4200

 app-e2e-tests:
   image: myapp
   command: dockerize -wait tcp://app-serve:4200 
             -wait tcp://selenium-chrome-standalone:4444 
             -timeout 10s -wait-retry-interval 1s bash -c "npm run e2e"
   depends_on:
     - app-serve
     - selenium-chrome-standalone

 selenium-chrome-standalone:
   image: selenium/standalone-chrome
   expose:
    - 44444

The file above tells Docker to spin up an environment with 3 containers:

• Our app to test: the container uses the myapp image which we’ll build right below
• A container running the tests: the container uses the same myapp image. It uses dockerize to wait for the servers to be up before running the tests
• The Selenium server: the container uses the official Selenium image. Nothing to do here. We could run the tests from the same container as the app. Splitting it makes things clearer. It also allows you to separate outputs from the 2 containers in the result logs.

The containers will live inside a private virtual network and see each other as http://the-container-name (more here on networking in Docker).

We need a Dockerfile to build the myapp image used for the containers. It should turn your frontend code into a bundle as close to production as possible. Running unit tests and linting is a good idea at that stage. After all, there's no need to run end to end tests if the basics do not work.

In the Dockerfile below we use a node image as base, install dockerize, and bundle the application. It is important to build for production. You don’t want to test a development build that is pre-optimizations. Many things can go wrong there.

FROM node:12.13.0 AS base

ENV DOCKERIZE_VERSION v0.6.0

RUN wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
   && tar -C /usr/local/bin -xzvf dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz \
   && rm dockerize-alpine-linux-amd64-$DOCKERIZE_VERSION.tar.gz

RUN mkdir -p ~/app
WORKDIR ~/app

COPY package.json .
COPY package-lock.json .

FROM base AS dependencies

RUN npm install

FROM dependencies AS runtime

COPY . .
RUN npm run lint
RUN npm run test:ci
RUN npm run build --production

Now that we have all the pieces together let's run the tests using this command:

docker-compose up --build --abort-on-container-exit

It is long-ish, so script it in your project somehow. It will build the myapp image based on the provided Dockerfile and then start all the containers. Dockerize makes sure your app and Selenium are up before executing the tests.

The --abort-on-container-exit option will kill the environment when one container exists. Since only our testing container is meant to exit (the others are servers), that is what we want.

The docker-compose command will have the same exit code as the exiting container. It means you can easily detect from the command line if the tests succeeded or not.

You are now ready to run end to end tests locally and on any server supporting Docker. That's pretty good!

The tests run with only one browser for now, though. Let’s add more.

Testing on different browsers

The standalone Selenium image spins up a Selenium server with the browser you want. To run the tests on different browsers you need to update your tests' configuration and use the selenium/hub Docker image.

The image creates a hub between your application and the standalone Selenium images. Replace the selenium container section in your docker-compose.yml as follows:

  selenium-hub:
    image: selenium/hub
    container_name: selenium-hub
    expose:
      - 4444
  chrome:
    image: selenium/node-chrome
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444
  firefox:
    image: selenium/node-firefox
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - HUB_HOST=selenium-hub
      - HUB_PORT=4444

We now have 3 containers: Chrome, Firefox and the Selenium hub.

All the Docker images provided by Selenium are in this repository.

Careful! There is a tricky timing effect to consider. We use dockerize to have our test container wait for the Selenium hub to be up. That is not enough because we need to wait for the standalone images to be ready – in fact to register themselves to the hub.

We can do this by waiting for the standalone images to expose a port, but that is not a guarantee. It is easier to wait a few seconds before starting the tests. Update your test script to wait 5 seconds before the tests start (you can add a sleep command after the dockerize call).

Now you can be sure that your tests won’t start until all the browsers are ready. Waiting is not ideal but a few seconds are worth it. There is nothing more annoying than failing tests because of unstable automations.

Good. We have now covered the front end part. Let’s add the back end.

Add the back end as containers or git modules

The above might seem overkill to test only the front end part of your app. But we're aiming for much more. We want to test the whole system.

Let’s add a database and a back end to our Docker compose environment.

If you are a front end developer you might think "we are the front end team – we don’t care about testing the back end." Are you sure?

The front end is always the last part to integrate all the other pieces. That means crunch time. Crunch time that would no longer exist if you were able to test the front end with the back end continuously and catch errors sooner.

The technique I describe here is very easy to apply even if your back end is in a different repository.

This is what the docker-compose.yml looks like:

version: '3.1'

services:
 db:
   image: postgres
   environment:
     POSTGRES_USER: john
     POSTGRES_PASSWORD: mysecretpassword
   expose:
     - 5432
 backend:
   context: ./backend
   dockerfile: ./backend/Dockerfile
   image:mybackend
   command: dockerize
       -wait tcp://db:5432 -timeout 10s
       bash -c "./seed_db.sh && ./start_server.sh"
   environment:
     APP_DB_HOST: db
     APP_DB_USER: john
     APP_DB_PASSWORD: mysecretpassword
   expose:
     - 8000
   depends_on:
     - db
 app-serve:
   build: .
   image: myapp
   command: npm run serve:sw
   expose:
    - 4200
 app-e2e-tests:
   image: myapp
   command: dockerize -wait tcp://app-serve:4200 
        -wait tcp://backend:8000 
        -wait tcp://selenium-chrome-standalone:4444 -timeout 10s 
        -wait-retry-interval 1s bash -c "npm run e2e:docker"
   depends_on:
     - app-serve
     - selenium-chrome-standalone
 selenium-chrome-standalone:
   image: selenium/standalone-chrome
   expose:
    - 44444

In this example we added a postgres database and a container for the back end to run. Dockerize synchronizes the containers' commands.

If your system has more than one back end component, add as many containers as you need. You need to wire the container dependencies properly. This means proper hostnames as environment variables on your components. And order of startup if some components depend on others.

The Selenium tests you have written should not need any modifications. You might need to put test data in the database. To keep this step in the testing area we added the seeding script before the backend startup script. This way we are sure that things happen in the proper order:

• The DB starts and is ready to accept connections
• A script seeds the DB data
• The back end and the front end start – so the tests can start

Monorepository

If you look at the back end container you can see there is a catch. It uses an image called mybackend built from a file located at backend/Dockerfile. This implies that your back end is in the same git repository in a folder called backend. The name is just an example of course.

If your back end and front end are in the same repository you are good. Define the Dockerfile to build your back end and adjust the startup command to what you need.

That’s all good but usually the back end is not in the same repository. Or you can have many back end components in different repositories. What do you do then?

Multiple repositories

The super clean solution is to have a CI process on each back end component repository.

If you don’t have any you can check out the API end to end testing with Docker guide to get started. It uses the same techniques as this article so you have a consistent setup across your whole project.

The CI process for each component runs automated tests. Upon success it pushes a docker image with the component to a private Docker registry. The back end container in our docker-compose.yml file above would use this image.

This solution requires a private Docker registry to store your images. You can use Docker Hub but then it becomes public. If you don't have one already and don’t plan to do so, it is not worth the effort.

The other solution is to use the submodules feature in git. Your back end repository becomes a virtual child of your front end repository. You just need to add the file .gitmodules like this to your front end repository:

[submodule "backend"]
  path = backend
  url = git@your:backend/repository.git
  branch = develop

Run the command git submodule update --remote which will pull the specified branch of the back end repository into a folder called "backend". Add as many submodules as you need if you have more than one back end component.

That’s it. Have your CI run the submodule command and from a file system perspective it's as if you're in a monorepository.

If you don’t want the back end code locally while developing the front end just don’t run the command. You’ll have an empty back end folder.

Versioning and back end/front end incompatibilities

The 2 techniques above test the front end with the latest “CI tests passed” version of your back end. That may lead to broken builds if your components are not compatible at times.

If they are compatible more often than not, stick to the “always test with the latest versions” approach. You’ll fix the occasional incompatibilities on the fly.

That won’t work, though, if incompatibilities are business as usual. In this case you need to manually control version updates. That is very easy to do.

You can lock the version of a component in the docker-compose.yml file or in the .gitmodules file. When pushing to the Docker registry you would tag the component image with the commit number of the corresponding code. The relevant docker-compose.yml file section becomes:

backend:
  image: backendapp:34028fc

Similarly the .gitmodules file would not target a branch head but a given commit:

[submodule "backend"]
  path = backend
  url = git@your:backend/repository.git
  branch = 34028fc

Bonus: version updates are versioned with your code. You can track which version was used for each build. This is useful when fixing failed builds or trying to reproduce old bugs.

We could push the approach to the next level. You could have a dedicated repository that would wire all your components as git modules. Bumping the versions could be a form of delivery and handover to the test/QA team.

In theory it is best to keep the latest versions of components working together more often than not. And drop the need for manual versioning. If that is not the case that is OK. Ignore the purists who will tell you that you are not following best practices and so on.

If you are just starting don't aim for the stars at first. Pick what works best for you to enjoy the benefits of automated testing right now. Then keep improving your process along the way.

Bonus on writing maintainable Selenium tests

Back to Selenium and three important bits of advice to help you write good UI tests.

First, avoid CSS selectors if you can. Selenium works on the DOM and can identify elements by IDs or CSS or XPath. Use IDs as much as possible even if you have to add them to your app code for only this purpose. CSS and XPath selectors are shaky. As soon as your application structure changes, they will be broken.

Second, use the Page Objects approach. It is about encapsulating your application so selectors are not directly used in tests. If your page HTML/CSS changes, your tests will have to be rewritten to use new selectors. Page Objects abstract selectors and turn them into user actions. Here is a great article on how to use Page Objects properly.

Third, don’t build long user journeys in your tests. If your tests fail at the 50th action it’s going to be difficult to reproduce and fix. Create test suites that play part of the scenarios starting from the login page. This way you are always a few clicks away from the bug your tests will catch.

Also don’t risk writing tests that rely on state from previous actions. Test suite coupling is something you want to avoid.

Let's take a practical example. Say you are testing a SaaS application for schools. The use cases could be:

• Create a class
• Register kids' and parents' data
• Setup the weekly plan for the class
• Check absences
• Input grades

Along the way you will have the login process and some navigation checks.

You could write a test that goes through the whole chain as described above. And this would be convenient because to declare kids you need a class to exist. To check absences you need a weekly plan in place. And you need kids to input grades. It’s a quick win to build one test suite that does all these things at first.

If you have nothing at the moment and want to achieve good test coverage quickly: go for it! Done is better than perfect if it allows you to catch errors in your application now.

The cleaner solution would be to use a baseline scenario to start smaller test suites. In the example above the baseline scenario should be to create a class and register kids' data.

Create a test suite that does exactly that: create a class and registered kids' and parents' data. Always run it first. If this stops working then you don’t need to move further on. This version of the code will never reach end users anyway.

Then create a function that encapsulates the baseline scenario. It will be duplicate code to some extent with the previous test suite. But it will allow you to have a one line function to use as a setup hook for all the other test suites. This is the best of both worlds: test scenarios starting from a fresh state in the application with minimal effort.

Conclusion

I hope this article gave you some good insight into how you can quickly set up end to end tests for a complex system. Multiple components in multiple repositories should not be a barrier. Docker compose makes it easy to put things together.

End to end tests are the best way to avoid crunch time. In complex systems, late delivery of some components puts a burden on other teams. Integrations are done in a rush. Code quality drops. That's a vicious circle. Testing often and catching cross component errors early is the solution.

Selenium tests can be done quick and dirty to get going fast. That is perfectly OK. Automate things. Then improve. Remember:

Done is better than perfect any day of the year.

Thanks for reading!

If you want more of my articles like this, you can find them on The Fire CI Blog.

This post is originally from www.jaredwolff.com

Privacy.

Performance.

Brilliant looks.

Can you have all three?

(Of course!)

Having a statically generated blog is great. Many folks use services like Disqus and Google Analytics to make them even better. Not surprising if you were one of them! Privacy concerns are are the forefront of everyone’s attention. So, rather than keeping the status quo, it’s time to do something about it!

If you've been looking to protect your site visitor's privacy and improve performance this blog post is for you.

In this article we'll be using DigitalOcean’s Docker droplet. It allows you to host several different applications/services on one (virtual) machine. By the end of it you'll know how to run your own comments server using Commento. Plus i’ll share a few tricks i’ve learned along the way to make it much easier for you.

Leeeets go!

Reverse Proxy

One of the most important aspects of this setup is the reverse proxy. A reverse proxy acts like a router. Requests come in for a certain domain. That request is then routed to the service associated with that domain.

Here’s a diagram from the Nginx Reverse Proxy + Let’s Encrypt Helper documentation. It'll help illustrate the idea.

Another benefit is that there’s an extra layer of protection to the outside world. Your websites run in a private network and the only access is through the Nginx reverse proxy. Point your DNS to the server and Nginx handles all the magic.

Here's how to get it setup:

1. Go ahead and set up your Digital Ocean Droplet. All the info you need is right here. The $5 version is more than sufficient.

2. Go here to clone the repository. You can also run this in your terminal. Make sure you SSH into your Digital Ocean droplet first!

   git clone git@github.com:evertramos/docker-compose-letsencrypt-nginx-proxy-companion.git

3. Change directories to the cloned repository.

4. Copy .env.sample to .env and update the values inside. I had to change the IP value to the IP of my Digital Ocean Droplet. I left all the other ones alone.
5. Run docker-compose up -d to start everything. (you can run without the -d option to make sure everything starts ok. Or you can attach the log output using docker container logs -f <container name

When pointing your sub-domains to this server, make sure you use an A record. Here's an example of mine:

Depending on your DNS provider, you'll have to figure out how to set an A record. That is beyond the purpose of this article though!

Setting Up Commento with Docker Compose

Here is the current docker compose file i'm using for Commento. It includes a few more environment variables for configuring Github, Gitlab and Google. It also includes the environment variables for setting the SMTP settings. These parameters are important. Otherwise you can't receive password reset or moderation emails!

version: '3'

services: commento: image: registry.gitlab.com/commento/commento container_name: commento restart: always environment: COMMENTO_ORIGIN: https://${COMMENTS_URL} COMMENTO_PORT: 8080 COMMENTO_POSTGRES: postgres://postgres:postgres@postgres:5432/commento?sslmode=disable COMMENTO_SMTP_HOST: ${SMTP_HOST} COMMENTO_SMTP_PORT: ${SMTP_PORT} COMMENTO_SMTP_USERNAME: ${SMTP_USERNAME} COMMENTO_SMTP_PASSWORD: ${SMTP_PASSWORD} COMMENTO_SMTP_FROM_ADDRESS: ${SMTP_FROM_ADDRESS} COMMENTO_GITHUB_KEY: ${COMMENTO_GITHUB_KEY} COMMENTO_GITHUB_SECRET: ${COMMENTO_GITHUB_SECRET} COMMENTO_GITLAB_KEY: ${COMMENTO_GITLAB_KEY} COMMENTO_GITLAB_SECRET: ${COMMENTO_GITLAB_SECRET} COMMENTO_GOOGLE_KEY: ${COMMENTO_GOOGLE_KEY} COMMENTO_GOOGLE_SECRET: ${COMMENTO_GOOGLE_SECRET} COMMENTO_TWITTER_KEY: ${COMMENTO_TWITTER_KEY} COMMENTO_TWITTER_SECRET: ${COMMENTO_TWITTER_SECRET} VIRTUAL_HOST: ${COMMENTS_URL} VIRTUAL_PORT: 8080 LETSENCRYPT_HOST: ${COMMENTS_URL} LETSENCRYPT_EMAIL: ${EMAIL} depends_on:

• postgres networks:
• db_network

• webproxy

  postgres: image: postgres container_name: postgres environment: POSTGRES_DB: commento POSTGRES_USER: postgres POSTGRES_PASSWORD: postgres networks:

• db_network volumes:

• postgres_data_volume:/var/lib/postgresql/data

  networks: db_network: webproxy: external: true

  volumes: postgres_data_volume:

To set the environment variables, put them inside an .env file. Make sure the .env file is in the same directory as docker-compose.yml. When you run docker-compose up it will apply the variables set in the .env file. Nothing gets set if they're left blank.

Set the required COMMENTS_URL and EMAIL or you may run into problems. The best way to set these is by pacing them in the .env file. Here is an example:

COMMENTS_URL=comments.your.url EMAIL=you@your.url

Getting OAuth Key & Secret

Commento works with most popular OAuth providers. Thus visitors can leave comments without making an account.

The instructions are similar for each. I've outlined the steps for all of them below.

Twitter

1. Login to Twitter.com and apply for a developer account: https://developer.twitter.com/en/application/use-case

   

2. Describe how you'll use the API. You can use what I wrote.

   

3. Double check your entry and click Looks Good!

   

4. Agree to the terms of service.

   

   

5. They'll tell you to check your email for a confirmation. Confirm your email and you should be able to create your first app!

6. Once approved to to Get started click Create an app.

   

7. Next screen, again click Create an app

   

8. Enter all the appropriate details. For the callback URL, use https://<your URL>/api/oauth/github/callback where <your URL> is your Commento subdomain.

   

9. Finally, once you're done filling out the information to go the Keys and Tokens area. Save both the key and token. Enter them into the .env file. You can use COMMENTO_TWITTER_KEY and COMMENTO_TWITTER_SECRET

   

Gitlab

1. Login to Gitlab.com and go to to top right and click Settings

2. Then click on Applications

   

3. Enter a name for your app. I put Commento.

4. Set the Redirect URI to https://<your URL>/api/oauth/gitlab/callback

5. Select the read_user scope.

   

6. Click the green Save Application button

7. Copy the Application ID and Secret and place them in your .env file using COMMENTO_GITLAB_KEY and COMMENTO_GITLAB_SECRET

   

Github

1. To get your OAuth key and secret, you'll need to go to this URL: https://github.com/settings/developers

2. Once there, click on New OAuth App

   

3. Enter your details. For the callback URL, use https://<your URL>/api/oauth/github/callback where <your URL> is your Commento subdomain.

   

   Note: Make sure you include https in your URLs.

4. Grab the Client ID and Client secret and put that into your .env file using COMMENTO_GITHUB_KEY and COMMENTO_GITHUB_SECRET

   

Google

Setting up Google is just about as tedious to set up as Twitter. Despite how scary I just made it out to be, it's completely doable. Here are the steps.

1. Go to this URL: Google Developer Console

2. Create a new project

   

3. Click the GoogleAPIs logo in the top left corner to go back once you have a project. (Make sure the dropdown next to the GoogleAPIs logo is the same as your new project!)

4. Then, click Credentials on the left side.

5. Update the Application Name and Authorized Domains in the OAuth consent screen

   

6. Click Create credentials then OAuth client ID

   

7. On the Create OAuth client ID enter the subdomain associated with Commento to Authorized Javascript origins. Then, enter the full callback URL. For example https://comments.jaredwolff.com/api/oauth/google/callback. Make it yours by replacing comments.jaredwolff.com with your URL.

   

   Once entered, click the create button.

8. Grab the client ID and client secret

   

9. Update your .env file using COMMENTO_GOOGLE_KEY and COMMENTO_GOOGLE_SECRET

Install your application

You've entered your OAuth Credentials email, domain and SMTP credentials. It's time to wrap this show up!

1. Once you're done editing your .env file. Run docker-compose up (For files not named docker-compose.yml, use the -f flag. Example: docker-compose -f commento.yml up
2. Watch the output for errors. If it looks good you may want to kill it (CTRL+C) and run with the -d flag

3. On first start, Commento will prompt you with a login screen.

   

4. Create a new account by clicking Don't have an account yet? Sign up.

5. Enter your information and click Sign Up

6. Check your email and click the included link:

   

7. Log in with your freshly made account.

8. Then, click Add a New Domain.

   

9. Once created go to Installation Guide. Copy the snippet and place it where ever you want your comments to live. In my case, I put the snippet in an area just after my <article> tag.

   

10. Re-compile your site and check for success!

    

    Checkmark! Finally, I recommend you try logging in with each individual OAuth configuration. That way you know it working for your website visitors. ?

Alternatives

I spent a good chunk playing around with some of the alternatives. This is by no means a definitive guide on what will work best for your site. Here are some of the top ones as of this writing:

https://utteranc.es/#configuration

https://github.com/netlify/gotell

https://github.com/eduardoboucas/staticman

https://posativ.org/isso/

https://www.remarkbox.com

https://www.vis4.net/blog/2017/10/hello-schnack/

https://github.com/gka/schnack

There's also a huge thread over at the Hugo blog which has a ton more links and resources as well:

https://discourse.gohugo.io/t/alternative-to-disqus-needed-more-than-ever/5516

Conclusion

Congrats! You are now hosting your own comments server! ?

In this article you've learned how to harness the power of Docker and a Nginx Reverse Proxy. As an added bonus, you know how to set up OAuth credentials! That way future setup will be easy peasy.

By the way, this is only the tip of the iceberg. You can set up the same server for analytics, data collection and more. All the example code including code for other applications can be found here.

Finally, if you're looking pay for Commento head to www.commento.io and sign up for the service. You'll be supporting awesome open source software!

If you have comments and questions let's hear em'. Start the conversation down below. ???

Docker containers opened a world of possibilities for the tech community, hassles in setting up new software were decreased unlike old times when a mess was to be sorted by a grievous format, it reduced the time to set up and use new software which eventually played a big part for techies to learn new things, roll it out in a container and scrap it when done. Things became easy, and the best thing its open source anyone and everyone can use it, comes with a little learning curve though.

Out of the myriad possibilities was the possibility of implementing complex technology stacks for our applications, which previously would have been the domain of experts. Today with the help of containers software engineers with sound understanding of the underlying systems can implement a complex stack and why not it’s the need of the hour, the figure of speech “Jack of all trades” got a fancy upgrade; “Master of some” based on the needs of the age. Simply put “T” shaped skills.

The possibility of defining a complex stack in a file and running it with a single command, pretty tempting huh. The guys at Docker Inc. choose to call it Docker compose.

In this article, we will use Docker’s example Voting App and deploy it using Docker compose.

Docker Compose

In the words of Docker Inc.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

The Voting App

Introducing the most favourite demonstration app for the Docker community “The Voting App”, as if it needs an introduction at all. This is a simple application based on micro-services architecture, consisting of 5 simple services.

Voting app architecture: https://github.com/docker/example-voting-app

1. Voting-App: Frontend of the application written in Python, used by users to cast their votes.

2. Redis: In-memory database, used as intermediate storage.

3. Worker: .Net service, used to fetch votes from Redis and store in Postres database.

4. DB: PostgreSQL database, used as database.

5. Result-App: Frontend of the application written in Node.js, displays the voting results.

The Voting repo has a file called docker-compose.yml this file contains the configuration for creating the containers, exposing ports, binding volumes and connecting containers through networks required for the voting app to work. Sounds like a lot of pretty long docker run and docker network create commands otherwise, docker compose allows us to put all of that stuff in a single docker-compose file in yaml format.

version: "3"

services:
  vote:
    build: ./vote
    command: python app.py
    volumes:
     - ./vote:/app
    ports:
      - "5000:80"
    networks:
      - front-tier
      - back-tier

  result:
    build: ./result
    command: nodemon server.js
    volumes:
      - ./result:/app
    ports:
      - "5001:80"
      - "5858:5858"
    networks:
      - front-tier
      - back-tier

  worker:
    build:
      context: ./worker
    depends_on:
      - "redis"
    networks:
      - back-tier

  redis:
    image: redis:alpine
    container_name: redis
    ports: ["6379"]
    networks:
      - back-tier

  db:
    image: postgres:9.4
    container_name: db
    volumes:
      - "db-data:/var/lib/postgresql/data"
    networks:
      - back-tier

volumes:
  db-data:

networks:
  front-tier:
  back-tier:

Git clone and cd into the voting app repo.

dockersamples/example-voting-app
_example-voting-app - Example Docker Compose app_github.com

Compose Time

With all of our application defined in a single compose file we can take a sigh of relief, chill and simply run the application. The beauty of compose lies in the fact that a single command creates all the services, wires up the networks(literally), mounts all volumes and exposes the ports. Its time to welcome the up command, its performs all of the aforementioned tasks.$ docker-compose up

After lots of “Pull complete”, hundreds of megabytes and few minutes (maybe more). . .

Voila, we have the voting app up and running.

Command docker ps lists all the running containers

$ docker ps -a --format="table {{.Names}}\t{{.Image}}\t{{.Ports}}" 
NAMES               IMAGE               PORTS
voting_worker_1     voting_worker      
db                  postgres:9.4        5432/tcp
voting_vote_1       voting_vote         0.0.0.0:5000->80/tcp
voting_result_1     voting_result       0.0.0.0:5858->5858/tcp, 0.0.0.0:5001->80/tcp
redis               redis:alpine        0.0.0.0:32768->6379/tcp

The above command displays all the running containers, respective images and the exposed port numbers.

The Voting app can be accessed on http://localhost:5000

Likewise the Voting results app can be accessed on http://localhost:5001

Each vote cast on the Voting app is first stored in the Redis in-memory database, the .Net worker service fetches the vote and stores it in the Postgres DB which is accessed by the Node.js frontend.

Compose Features

Compose provide the flexibility to use a project name to isolate the environments from each other, the project name is the base name of the directory that contains the project. In our voting app this is signified by the name of the containers voting_worker_1 where voting is the base name of the directory. We can set a custom project name using the -p flag followed by the custom name.

Compose preserves all volumes used by the services defined in the compose file, thus no data is lost when the containers are recreated using docker-compose up. Another cool feature is that only the containers which have changed are recreated, the containers whose state did not change remain untouched.

Another cool feature is the support for variables in the compose file, we can define variables in a .env file and use them in the docker-compose file. Here the POSTGRES_VERSION=9.4 can defined in the environment file or can be defined in the shell. It is used in the compose file in the following manner:

db:  
  image: "postgres:${POSTGRES_VERSION}"

Command Cheat Sheet

It's easy as breeze to start, stop and play around with compose.

$ docker-compose up -d
$ docker-compose down
$ docker-compose start
$ docker-compose stop
$ docker-compose build
$ docker-compose logs -f db
$ docker-compose scale db=4
$ docker-compose events
$ docker-compose exec db bash

Summary

Docker Compose is a great tool to quickly deploy and scrap containers, the compose file can run seamlessly on any machine installed with docker-compose. Experimentation and learning technologies is just a Compose file away ;).

I hope this article helped in the understanding of Docker Compose. I’d love to hear about how you use Docker Compose in your projects. Share the knowledge, help it reach more people.