Yet most developers have never built one.

That's a missed opportunity. CLI tools are one of the most practical things a developer can ship. They automate repetitive workflows, standardise processes across teams, and, when published, become tangible artifacts that the developer community can discover, install, and use.

In this handbook, you'll go from zero to building a fully distributed Dart CLI tool. We'll start with the fundamentals – how CLIs work, how Dart receives and processes terminal input, and the core syntax you need to know. Then we'll build three progressively complex CLIs, starting with the basics and finishing with a real-world API request runner. Finally, we will cover every distribution path available, from pub.dev to compiled binaries, Homebrew taps, Docker, and local team activation.

By the end of the guide, you'll understand both how to build a CLI tool in Dart as well as how to ship it so other developers can actually use it.

Table of Contents

• Prerequisites

• What is a CLI and Why Should You Build One?

• CLI Syntax Anatomy

• How Dart Receives Terminal Input

• Core CLI Concepts in Dart

  • stdout, stderr, and stdin

  • Exit Codes

  • Environment Variables

  • File and Directory Operations

  • Running External Processes

  • Platform Detection

  • Async in CLI

• Setting Up Your Dart CLI Project

• CLI 1 — Hello CLI: The Fundamentals

• CLI 2 — dart_todo: A Terminal Task Manager

  • Introducing the args Package

  • Building dart_todo

• CLI 3 — dart_http: A Lightweight API Request Runner

  • Building dart_http

• Adding Color and Polish to Your CLI

• Testing Your CLI Tool

• Deploying and Distributing Your CLI

  • Mode 1: pub.dev — Public Package Distribution

  • Mode 2: Local Path Activation

  • Mode 3: Compiled Binary via GitHub Releases

  • Mode 4: Homebrew Tap

  • Mode 5: Docker

• Choosing the Right Distribution Mode

• Conclusion

Prerequisites

Before starting, you should have:

• Dart SDK installed (dart --version should work in your terminal)

• Basic familiarity with Dart syntax

• Comfort with the terminal and running commands

• A pub.dev account (for the publishing section)

• A GitHub account (for the binary distribution section)

What is a CLI and Why Should You Build One?

A CLI (or Command Line Interface) is a program you interact with entirely through text commands in a terminal, rather than through buttons and screens in a graphical interface.

Many of the tools you likely already rely on as a developer are CLI tools:

flutter build apk
git commit -m "fix: auth flow"
dart pub get
npm install

Flutter, Git, Dart, npm – all CLIs. You are already a CLI user every single day. This article is about becoming a CLI builder.

There are three strong reasons to build CLI tools as a developer:

1. Automating repetitive work: Anything you type more than twice a week is a candidate for automation. Generating boilerplate folder structures, running sequences of commands, scaffolding files, checking environments before a build a CLI turns a seven-step manual process into a single command.

2. Standardising team workflows: Instead of a README that says "run these commands in this order," you ship one command that does all of it – consistently, every time, with no room for human error or a missed step.

3. Building and publishing tooling. A published Dart CLI package is a tangible artifact. It shows up on pub.dev, gets installed and used by other developers, and communicates real engineering depth in a way that a portfolio or resume cannot.

CLI Syntax Anatomy

Before writing a single line of code, it helps to understand the structure of a CLI command. Every command follows a consistent pattern:

tool [subcommand] [arguments] [options/flags]

Breaking down a real example:

flutter build apk --release --obfuscate
│       │     │   │
tool    sub   arg  flags

• Tool — the program itself (flutter, dart, git)

• Subcommand — the action being performed (build, run, pub)

• Arguments — what the action operates on (apk, main.dart, a filename)

• Flags and Options — modifiers that change behaviour

There are two types of options:

--release              # Boolean flag — either present or absent

--output=build/app     # Key-value option — name and a value
-v                     # Short flag — single hyphen, single character

This is the anatomy your CLIs will follow. Understanding it before writing any code means you will design your commands intentionally rather than stumbling into structure by accident.

How Dart Receives Terminal Input

In Dart, everything the user types after your tool name is passed into your program through the main function:

void main(List<String> args) {
  print(args);
}

Run it:

dart run bin/mytool.dart hello world --name=Seyi
# [hello, world, --name=Seyi]

That List<String> args is just a list of strings. Each word or flag the user typed becomes an element in that list. Everything else you build on top of a CLI subcommands, flags, validation — is ultimately just processing this list.

Core CLI Concepts in Dart

Before building anything, there's a set of foundational concepts that every CLI developer needs to understand. These are the building blocks that everything else sits on top of.

stdout, stderr, and stdin

Most developers use print() for all output when they start building CLIs. That works for learning but it's incorrect in production.

There are two separate output streams in a terminal program:

• stdout — regular output, meant for the user

• stderr — error output, meant for diagnostic messages and failures

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Error: no arguments provided');
    exit(1);
  }

  stdout.writeln('Processing: ${args[0]}');
}

Keeping these separate matters because users can redirect stdout to a file without errors polluting it:

dart run bin/tool.dart > output.txt
# Errors still appear in the terminal
# Normal output goes cleanly to the file

Tools like git, flutter, and curl all do this correctly. Your CLI should too.

stdin is the third stream — reading input from the user interactively at runtime:

import 'dart:io';

void main() {
  stdout.write('Enter your name: ');
  final name = stdin.readLineSync();

  if (name == null || name.trim().isEmpty) {
    stderr.writeln('Error: no name provided');
    exit(1);
  }

  stdout.writeln('Hello, $name!');
}

stdout.write (without ln) keeps the cursor on the same line so the user types right after the prompt. stdin.readLineSync() blocks until the user presses Enter and returns the typed string, or null if the stream closes unexpectedly. Always handle the null case.

Exit Codes

Every program returns an exit code when it finishes. This is how the shell – and any script or CI system calling your tool – knows whether it succeeded or failed.

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Error: please provide an argument');
    exit(1); // failure
  }

  stdout.writeln('Done');
  exit(0); // success — also the default if you don't call exit()
}

The conventions are:

• 0 — success

• 1 — general failure

• 2 — incorrect usage (wrong arguments, missing flags)

Exit codes are critical when your CLI is called inside shell scripts or GitHub Actions workflows. A non-zero exit code stops a pipeline immediately. That's exactly the behaviour you want from a quality gate or a validation step.

Environment Variables

Your CLI can read environment variables set in the user's shell:

import 'dart:io';

void main() {
  final token = Platform.environment['API_TOKEN'];

  if (token == null) {
    stderr.writeln('Error: API_TOKEN environment variable is not set');
    exit(1);
  }

  stdout.writeln('Token found — proceeding...');
}

Set it in the terminal and run:

export API_TOKEN=mytoken123
dart run bin/tool.dart
# Token found — proceeding...

This pattern is essential for CLI tools that interact with APIs, cloud services, or CI environments where credentials should never be hardcoded.

File and Directory Operations

Many CLI tools read from or write to the file system. Dart's dart:io library covers everything you need:

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: tool <filename>');
    exit(2);
  }

  final file = File(args[0]);

  if (!file.existsSync()) {
    stderr.writeln('Error: "${args[0]}" not found');
    exit(1);
  }

  final contents = file.readAsStringSync();
  stdout.writeln(contents);

  final output = File('output.txt');
  output.writeAsStringSync('Processed:\n$contents');
  stdout.writeln('Written to output.txt');
}

Working with directories:

import 'dart:io';

void main() {
  // Where the command was run from
  final cwd = Directory.current.path;
  stdout.writeln('Working directory: $cwd');

  // Create a directory relative to current location
  final dir = Directory('$cwd/generated');

  if (!dir.existsSync()) {
    dir.createSync(recursive: true);
    stdout.writeln('Created: ${dir.path}');
  } else {
    stdout.writeln('Already exists: ${dir.path}');
  }
}

The recursive: true flag on createSync means it creates all intermediate directories — equivalent to mkdir -p in bash.

Running External Processes

One of the most powerful things a CLI can do is call other programs. Your Dart CLI can run git, flutter, dart, or any shell command programmatically:

import 'dart:io';

void main() async {
  // Run a command and wait for it to finish
  final result = await Process.run('dart', ['pub', 'get']);

  stdout.write(result.stdout);

  if (result.exitCode != 0) {
    stderr.write(result.stderr);
    exit(result.exitCode);
  }

  stdout.writeln('Dependencies installed successfully');
}

For long-running commands where you want output to stream live as it happens:

import 'dart:io';

void main() async {
  final process = await Process.start('flutter', ['build', 'apk']);

  // Pipe output directly to the terminal in real time
  process.stdout.pipe(stdout);
  process.stderr.pipe(stderr);

  final exitCode = await process.exitCode;
  exit(exitCode);
}

Process.run — waits for completion, returns all output at once. Use for short commands.

Process.start — streams output live as it arrives. Use for long-running commands where the user needs to see progress.

Platform Detection

Sometimes your CLI needs to behave differently depending on the operating system it is running on:

import 'dart:io';

void main() {
  if (Platform.isWindows) {
    stdout.writeln('Running on Windows');
  } else if (Platform.isMacOS) {
    stdout.writeln('Running on macOS');
  } else if (Platform.isLinux) {
    stdout.writeln('Running on Linux');
  }

  // Useful for path handling across operating systems
  stdout.writeln(Platform.pathSeparator); // \ on Windows, / elsewhere
  stdout.writeln(Platform.operatingSystem); // 'macos', 'linux', 'windows'
}

This matters when your CLI creates files, resolves paths, or calls shell commands that differ between operating systems.

Async in CLI

Dart CLIs support async/await natively. Any main function can be made async:

import 'dart:io';

void main() async {
  stdout.writeln('Starting...');

  await Future.delayed(const Duration(seconds: 1)); // simulating async work

  stdout.writeln('Done');
}

Any operation involving file I/O, HTTP requests, or spawning processes will be asynchronous. Get comfortable with async main functions early — you'll use them constantly.

Setting Up Your Dart CLI Project

Create a new Dart console project:

dart create -t console my_cli_tool
cd my_cli_tool

This generates a clean structure:

my_cli_tool/
  bin/
    my_cli_tool.dart    ← entry point
  lib/                  ← shared library code
  test/                 ← tests
  pubspec.yaml
  README.md

The bin/ directory is where your executable entry point lives. The lib/ directory is where you put everything else — commands, utilities, models — that bin/ imports and uses.

Open pubspec.yaml. You'll need to add an executables block before publishing:

name: my_cli_tool
description: A sample CLI tool built with Dart
version: 1.0.0

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  my_cli_tool: my_cli_tool  # executable name: bin file name

dependencies:
  args: ^2.4.2

dev_dependencies:
  lints: ^3.0.0
  test: ^1.24.0

The executables block is what makes dart pub global activate my_cli_tool work. It tells Dart which script in bin/ to expose as a runnable command after installation.

CLI 1 — Hello CLI: The Fundamentals

This first CLI uses pure Dart — no packages. The goal is to get comfortable with args, subcommands, input validation, and exit codes before introducing any external dependencies.

Replace the contents of bin/my_cli_tool.dart:

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    printHelp();
    exit(0);
  }

  final command = args[0];

  switch (command) {
    case 'greet':
      handleGreet(args.sublist(1));
    case 'time':
      handleTime();
    case 'echo':
      handleEcho(args.sublist(1));
    case 'help':
      printHelp();
    default:
      stderr.writeln('Unknown command: "$command"');
      stderr.writeln('Run "mytool help" to see available commands.');
      exit(1);
  }
}

void handleGreet(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: mytool greet <name>');
    exit(2);
  }

  final name = args[0];
  stdout.writeln('Hello, $name! Welcome to your first Dart CLI.');
}

void handleTime() {
  final now = DateTime.now();
  stdout.writeln(
    'Current time: ${now.hour.toString().padLeft(2, '0')}:'
    '${now.minute.toString().padLeft(2, '0')}:'
    '${now.second.toString().padLeft(2, '0')}',
  );
}

void handleEcho(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: mytool echo <message>');
    exit(2);
  }

  stdout.writeln(args.join(' '));
}

void printHelp() {
  stdout.writeln('''
mytool — a simple Dart CLI

Usage:
  mytool <command> [arguments]

Commands:
  greet <name>      Greet someone by name
  time              Show the current time
  echo <message>    Echo a message back to the terminal
  help              Show this help message

Examples:
  mytool greet Seyi
  mytool echo "Hello from the terminal"
  mytool time
  ''');
}

Run it:

dart run bin/my_cli_tool.dart help

dart run bin/my_cli_tool.dart greet Seyi
# Hello, Seyi! Welcome to your first Dart CLI.

dart run bin/my_cli_tool.dart time
# Current time: 14:32:10

dart run bin/my_cli_tool.dart echo "Dart CLIs are powerful"
# Dart CLIs are powerful

dart run bin/my_cli_tool.dart unknown
# Unknown command: "unknown"
# Run "mytool help" to see available commands.

Three things this CLI demonstrates that are worth internalising:

1. Subcommands are just a switch on args[0]. The pattern is simple and scalable — add a new case to add a new command.

2. args.sublist(1) passes remaining args to the handler. When greet receives ['greet', 'Seyi'], it calls handleGreet(['Seyi']) — clean and isolated.

3. Every error path has a message and a non-zero exit code. The user always knows what went wrong and what to do next.

CLI 2 — dart_todo: A Terminal Task Manager

This CLI introduces the args package, JSON file persistence, and structured terminal output. It's meaningfully more complex than CLI 1 and reflects real patterns you will use in production tools.

Introducing the args Package

Manually parsing List<String> args works for simple cases, but breaks down quickly when you add flags like --priority=high, boolean options like --done, or commands with multiple optional arguments.

The args package handles all of that cleanly.

Add it to your pubspec.yaml:

dependencies:
  args: ^2.4.2

Run:

dart pub get

The core concept in args is the ArgParser. You define what your CLI accepts, and args handles parsing, validation, and generating help text automatically:

import 'package:args/args.dart';

void main(List<String> arguments) {
  final parser = ArgParser()
    ..addCommand('add')
    ..addCommand('list')
    ..addFlag('help', abbr: 'h', negatable: false);

  final results = parser.parse(arguments);

  if (results['help'] as bool) {
    print(parser.usage);
    return;
  }
}

For more complex CLIs with subcommands that each have their own flags, use ArgParser per command:

final parser = ArgParser();

final addCommand = ArgParser()
  ..addOption('priority', abbr: 'p', defaultsTo: 'normal');

parser.addCommand('add', addCommand);

Building dart_todo

Create a fresh project:

dart create -t console dart_todo
cd dart_todo

Update pubspec.yaml:

name: dart_todo
description: A terminal task manager built with Dart
version: 1.0.0

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  dart_todo: dart_todo

dependencies:
  args: ^2.4.2

dev_dependencies:
  lints: ^3.0.0
  test: ^1.24.0

Run dart pub get.

Create the folder structure:

dart_todo/
  bin/
    dart_todo.dart
  lib/
    models/
      task.dart
    storage/
      task_storage.dart
    commands/
      add_command.dart
      list_command.dart
      complete_command.dart
      delete_command.dart
      clear_command.dart
  pubspec.yaml

Step 1 — The Task Model (lib/models/task.dart)

class Task {
  final int id;
  final String title;
  final String priority;
  final bool isComplete;
  final DateTime createdAt;

  Task({
    required this.id,
    required this.title,
    required this.priority,
    this.isComplete = false,
    required this.createdAt,
  });

  Task copyWith({bool? isComplete}) {
    return Task(
      id: id,
      title: title,
      priority: priority,
      isComplete: isComplete ?? this.isComplete,
      createdAt: createdAt,
    );
  }

  Map<String, dynamic> toJson() => {
        'id': id,
        'title': title,
        'priority': priority,
        'isComplete': isComplete,
        'createdAt': createdAt.toIso8601String(),
      };

  factory Task.fromJson(Map<String, dynamic> json) => Task(
        id: json['id'] as int,
        title: json['title'] as String,
        priority: json['priority'] as String,
        isComplete: json['isComplete'] as bool,
        createdAt: DateTime.parse(json['createdAt'] as String),
      );
}

Step 2 — Storage (lib/storage/task_storage.dart)

This class handles reading and writing tasks to a local JSON file so they persist between CLI runs:

import 'dart:convert';
import 'dart:io';

import '../models/task.dart';

class TaskStorage {
  static final _file = File(
    '${Platform.environment['HOME'] ?? Directory.current.path}/.dart_todo.json',
  );

  static List<Task> loadAll() {
    if (!_file.existsSync()) return [];

    try {
      final content = _file.readAsStringSync();
      final List<dynamic> json = jsonDecode(content) as List<dynamic>;
      return json
          .map((e) => Task.fromJson(e as Map<String, dynamic>))
          .toList();
    } catch (_) {
      return [];
    }
  }

  static void saveAll(List<Task> tasks) {
    final json = jsonEncode(tasks.map((t) => t.toJson()).toList());
    _file.writeAsStringSync(json);
  }
}

Tasks are stored in a hidden JSON file in the user's home directory — a common pattern for CLI tools that need lightweight local persistence.

Step 3 — Commands

lib/commands/add_command.dart:

import 'dart:io';

import '../models/task.dart';
import '../storage/task_storage.dart';

void runAdd(List<String> args, String priority) {
  if (args.isEmpty) {
    stderr.writeln('Usage: dart_todo add <title> [--priority=high|normal|low]');
    exit(2);
  }

  final title = args.join(' ');
  final tasks = TaskStorage.loadAll();

  final newTask = Task(
    id: tasks.isEmpty ? 1 : tasks.last.id + 1,
    title: title,
    priority: priority,
    createdAt: DateTime.now(),
  );

  tasks.add(newTask);
  TaskStorage.saveAll(tasks);

  stdout.writeln('Added task #\({newTask.id}: "\)title" [$priority]');
}

lib/commands/list_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runList() {
  final tasks = TaskStorage.loadAll();

  if (tasks.isEmpty) {
    stdout.writeln('No tasks yet. Add one with: dart_todo add <title>');
    return;
  }

  stdout.writeln('');
  stdout.writeln('  ID   Status      Priority   Title');
  stdout.writeln('  ───  ──────────  ─────────  ────────────────────────');

  for (final task in tasks) {
    final status = task.isComplete ? 'done  ' : 'pending';
    final id = task.id.toString().padRight(4);
    final priority = task.priority.padRight(9);
    stdout.writeln('  \(id \)status  \(priority  \){task.title}');
  }

  stdout.writeln('');
}

lib/commands/complete_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runComplete(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: dart_todo complete <id>');
    exit(2);
  }

  final id = int.tryParse(args[0]);
  if (id == null) {
    stderr.writeln('Error: "${args[0]}" is not a valid task ID');
    exit(1);
  }

  final tasks = TaskStorage.loadAll();
  final index = tasks.indexWhere((t) => t.id == id);

  if (index == -1) {
    stderr.writeln('Error: No task found with ID $id');
    exit(1);
  }

  if (tasks[index].isComplete) {
    stdout.writeln('Task #$id is already complete.');
    return;
  }

  tasks[index] = tasks[index].copyWith(isComplete: true);
  TaskStorage.saveAll(tasks);

  stdout.writeln('Task #\(id marked as complete: "\){tasks[index].title}"');
}

lib/commands/delete_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runDelete(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: dart_todo delete <id>');
    exit(2);
  }

  final id = int.tryParse(args[0]);
  if (id == null) {
    stderr.writeln('Error: "${args[0]}" is not a valid task ID');
    exit(1);
  }

  final tasks = TaskStorage.loadAll();
  final index = tasks.indexWhere((t) => t.id == id);

  if (index == -1) {
    stderr.writeln('Error: No task found with ID $id');
    exit(1);
  }

  final title = tasks[index].title;
  tasks.removeAt(index);
  TaskStorage.saveAll(tasks);

  stdout.writeln('Deleted task #\(id: "\)title"');
}

lib/commands/clear_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runClear() {
  stdout.write('Are you sure you want to delete all tasks? (y/N): ');
  final input = stdin.readLineSync()?.trim().toLowerCase();

  if (input != 'y') {
    stdout.writeln('Cancelled.');
    return;
  }

  TaskStorage.saveAll([]);
  stdout.writeln('All tasks cleared.');
}

Step 4 — Entry Point (bin/dart_todo.dart)

import 'dart:io';

import 'package:args/args.dart';

import '../lib/commands/add_command.dart';
import '../lib/commands/clear_command.dart';
import '../lib/commands/complete_command.dart';
import '../lib/commands/delete_command.dart';
import '../lib/commands/list_command.dart';

void main(List<String> arguments) {
  final parser = ArgParser();

  // Add subcommand parsers
  final addParser = ArgParser()
    ..addOption(
      'priority',
      abbr: 'p',
      defaultsTo: 'normal',
      allowed: ['high', 'normal', 'low'],
      help: 'Task priority level',
    );

  parser
    ..addCommand('add', addParser)
    ..addCommand('list')
    ..addCommand('complete')
    ..addCommand('delete')
    ..addCommand('clear')
    ..addFlag('help', abbr: 'h', negatable: false, help: 'Show help');

  ArgResults results;

  try {
    results = parser.parse(arguments);
  } catch (e) {
    stderr.writeln('Error: $e');
    stderr.writeln(parser.usage);
    exit(2);
  }

  if (results['help'] as bool || results.command == null) {
    printHelp(parser);
    exit(0);
  }

  final command = results.command!;

  switch (command.name) {
    case 'add':
      runAdd(command.rest, command['priority'] as String);
    case 'list':
      runList();
    case 'complete':
      runComplete(command.rest);
    case 'delete':
      runDelete(command.rest);
    case 'clear':
      runClear();
    default:
      stderr.writeln('Unknown command: "${command.name}"');
      exit(1);
  }
}

void printHelp(ArgParser parser) {
  stdout.writeln('''
dart_todo — a terminal task manager

Usage:
  dart_todo <command> [arguments]

Commands:
  add <title>        Add a new task
    -p, --priority   Priority: high, normal, low (default: normal)
  list               List all tasks
  complete <id>      Mark a task as complete
  delete <id>        Delete a task
  clear              Delete all tasks

Examples:
  dart_todo add "Write the CLI article" --priority=high
  dart_todo list
  dart_todo complete 1
  dart_todo delete 2
  dart_todo clear
  ''');
}

Run it:

dart run bin/dart_todo.dart add "Write the CLI article" --priority=high
# Added task #1: "Write the CLI article" [high]

dart run bin/dart_todo.dart add "Review PR comments"
# Added task #2: "Review PR comments" [normal]

dart run bin/dart_todo.dart list
#   ID   Status      Priority   Title
#   ───  ──────────  ─────────  ────────────────────────
#   1    ⬜ pending  high       Write the CLI article
#   2    ⬜ pending  normal     Review PR comments

dart run bin/dart_todo.dart complete 1
# Task #1 marked as complete: "Write the CLI article"

dart run bin/dart_todo.dart delete 2
# Deleted task #2: "Review PR comments"

dart_todo demonstrates the patterns that form the backbone of almost every real CLI tool — argument parsing with args, JSON persistence, interactive prompts, structured output, and clean error handling across every command.

CLI 3 — dart_http: A Lightweight API Request Runner

This is the most complex CLI in this article – and the most immediately useful. dart_http lets developers make HTTP requests directly from the terminal, with pretty-printed JSON responses, response metadata, header support, and the ability to save responses to a file.

dart_http get https://jsonplaceholder.typicode.com/users/1
dart_http post https://jsonplaceholder.typicode.com/posts --body='{"title":"Hello"}'
dart_http get https://jsonplaceholder.typicode.com/users --save=users.json
dart_http get https://api.example.com/me --header="Authorization: Bearer mytoken"

Building dart_http

Create the project:

dart create -t console dart_http
cd dart_http

Update pubspec.yaml:

name: dart_http
description: A lightweight API request runner for the terminal
version: 1.0.0

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  dart_http: dart_http

dependencies:
  args: ^2.4.2
  http: ^1.2.1

dev_dependencies:
  lints: ^3.0.0
  test: ^1.24.0

Run dart pub get.

Project structure:

dart_http/
  bin/
    dart_http.dart
  lib/
    runner/
      request_runner.dart
    printer/
      response_printer.dart
    utils/
      headers_parser.dart
  pubspec.yaml

Step 1 — Headers Parser (lib/utils/headers_parser.dart)

Map<String, String> parseHeaders(List<String> rawHeaders) {
  final headers = <String, String>{};

  for (final header in rawHeaders) {
    final index = header.indexOf(':');
    if (index == -1) continue;

    final key = header.substring(0, index).trim();
    final value = header.substring(index + 1).trim();
    headers[key] = value;
  }

  return headers;
}

Step 2 — Response Printer (lib/printer/response_printer.dart)

import 'dart:convert';
import 'dart:io';

void printResponse({
  required int statusCode,
  required String body,
  required int durationMs,
  required int bodyBytes,
}) {
  final statusLabel = _statusLabel(statusCode);
  final size = _formatSize(bodyBytes);

  stdout.writeln('');
  stdout.writeln('\(statusLabel | \){durationMs}ms | $size');
  stdout.writeln('─' * 50);

  try {
    final decoded = jsonDecode(body);
    const encoder = JsonEncoder.withIndent('  ');
    stdout.writeln(encoder.convert(decoded));
  } catch (_) {
    // Not JSON — print as plain text
    stdout.writeln(body);
  }

  stdout.writeln('');
}

String _statusLabel(int code) {
  if (code >= 200 && code < 300) return '✅ $code';
  if (code >= 300 && code < 400) return '↪️  $code';
  if (code >= 400 && code < 500) return '❌ $code';
  return '$code';
}

String _formatSize(int bytes) {
  if (bytes < 1024) return '${bytes}b';
  if (bytes < 1024 * 1024) return '${(bytes / 1024).toStringAsFixed(1)}kb';
  return '${(bytes / (1024 * 1024)).toStringAsFixed(1)}mb';
}

Step 3 — Request Runner (lib/runner/request_runner.dart)

import 'dart:io';

import 'package:http/http.dart' as http;

import '../printer/response_printer.dart';

Future<void> runRequest({
  required String method,
  required String url,
  required Map<String, String> headers,
  String? body,
  String? saveToFile,
}) async {
  final uri = Uri.tryParse(url);

  if (uri == null) {
    stderr.writeln('Error: "$url" is not a valid URL');
    exit(1);
  }

  stdout.writeln('→ \({method.toUpperCase()} \)url');

  http.Response response;
  final stopwatch = Stopwatch()..start();

  try {
    switch (method.toLowerCase()) {
      case 'get':
        response = await http.get(uri, headers: headers);
      case 'post':
        response = await http.post(uri, headers: headers, body: body);
      case 'put':
        response = await http.put(uri, headers: headers, body: body);
      case 'patch':
        response = await http.patch(uri, headers: headers, body: body);
      case 'delete':
        response = await http.delete(uri, headers: headers);
      default:
        stderr.writeln('Error: unsupported method "$method"');
        exit(2);
    }
  } catch (e) {
    stderr.writeln('Error: request failed — $e');
    exit(1);
  }

  stopwatch.stop();

  printResponse(
    statusCode: response.statusCode,
    body: response.body,
    durationMs: stopwatch.elapsedMilliseconds,
    bodyBytes: response.bodyBytes.length,
  );

  if (saveToFile != null) {
    final file = File(saveToFile);
    file.writeAsStringSync(response.body);
    stdout.writeln('Response saved to $saveToFile');
  }
}

Step 4 — Entry Point (bin/dart_http.dart)

import 'dart:io';

import 'package:args/args.dart';

import '../lib/runner/request_runner.dart';
import '../lib/utils/headers_parser.dart';

void main(List<String> arguments) async {
  final parser = ArgParser();

  for (final method in ['get', 'post', 'put', 'patch', 'delete']) {
    final commandParser = ArgParser()
      ..addMultiOption('header', abbr: 'H', help: 'Request header (repeatable)')
      ..addOption('body', abbr: 'b', help: 'Request body (for POST/PUT/PATCH)')
      ..addOption('save', abbr: 's', help: 'Save response body to a file');

    parser.addCommand(method, commandParser);
  }

  parser.addFlag('help', abbr: 'h', negatable: false, help: 'Show help');

  ArgResults results;

  try {
    results = parser.parse(arguments);
  } catch (e) {
    stderr.writeln('Error: $e');
    printHelp();
    exit(2);
  }

  if (results['help'] as bool || results.command == null) {
    printHelp();
    exit(0);
  }

  final command = results.command!;
  final method = command.name!;
  final rest = command.rest;

  if (rest.isEmpty) {
    stderr.writeln('Error: please provide a URL');
    stderr.writeln('Usage: dart_http $method <url>');
    exit(2);
  }

  final url = rest[0];
  final rawHeaders = command['header'] as List<String>;
  final body = command['body'] as String?;
  final saveToFile = command['save'] as String?;

  final headers = parseHeaders(rawHeaders);

  // Default Content-Type for requests with a body
  if (body != null && !headers.containsKey('Content-Type')) {
    headers['Content-Type'] = 'application/json';
  }

  await runRequest(
    method: method,
    url: url,
    headers: headers,
    body: body,
    saveToFile: saveToFile,
  );
}

void printHelp() {
  stdout.writeln('''
dart_http — a lightweight API request runner

Usage:
  dart_http <method> <url> [options]

Methods:
  get       Send a GET request
  post      Send a POST request
  put       Send a PUT request
  patch     Send a PATCH request
  delete    Send a DELETE request

Options:
  -H, --header    Add a request header (repeatable)
  -b, --body      Request body (JSON string)
  -s, --save      Save response body to a file
  -h, --help      Show this help message

Examples:
  dart_http get https://jsonplaceholder.typicode.com/users
  dart_http get https://api.example.com/me --header="Authorization: Bearer token"
  dart_http post https://api.example.com/posts --body=\'{"title":"Hello"}\'
  dart_http get https://api.example.com/users --save=users.json
  ''');
}

Run it:

dart run bin/dart_http.dart get https://jsonplaceholder.typicode.com/users/1

# → GET https://jsonplaceholder.typicode.com/users/1
# 200 | 87ms | 510b
# ──────────────────────────────────────────────────
# {
#   "id": 1,
#   "name": "Leanne Graham",
#   "username": "Bret",
#   "email": "Sincere@april.biz"
# }

dart run bin/dart_http.dart get https://jsonplaceholder.typicode.com/users --save=users.json
# → GET https://jsonplaceholder.typicode.com/users
# 200 | 143ms | 5.3kb
# ──────────────────────────────────────────────────
# [ ... ]
# Response saved to users.json

dart run bin/dart_http.dart post https://jsonplaceholder.typicode.com/posts \
  --body='{"title":"Hello from dart_http","userId":1}'
# → POST https://jsonplaceholder.typicode.com/posts
# 201 | 312ms | 72b

Adding Color and Polish to Your CLI

The CLIs above are functional, but terminal output can be made significantly more readable with color. The ansi_styles package provides ANSI escape code support for coloring text in the terminal.

Add it to pubspec.yaml:

dependencies:
  ansi_styles: ^0.3.0

Using it:

import 'package:ansi_styles/ansi_styles.dart';

stdout.writeln(AnsiStyles.green('✅ Success'));
stdout.writeln(AnsiStyles.red('❌ Error: something went wrong'));
stdout.writeln(AnsiStyles.yellow('⚠️  Warning: check your config'));
stdout.writeln(AnsiStyles.bold('dart_http — API request runner'));
stdout.writeln(AnsiStyles.cyan('→ GET https://api.example.com/users'));

Apply color intentionally and consistently:

• Green — success states, completed operations

• Red — errors and failures

• Yellow — warnings and non-blocking issues

• Cyan — informational output, URLs, paths

• Bold — headers, tool names, important values

Avoid coloring everything. Color loses meaning when it is everywhere. Use it to draw the user's eye to what actually matters.

Testing Your CLI Tool

CLI tools are testable, and they should be tested. The most reliable approach is to test the logic inside your commands directly — not the terminal output formatting, but the behaviour.

Add test to your dev dependencies if it's not already there:

dev_dependencies:
  test: ^1.24.0

Testing command logic:

import 'package:test/test.dart';

import '../lib/models/task.dart';

void main() {
  group('Task model', () {
    test('copyWith updates isComplete correctly', () {
      final task = Task(
        id: 1,
        title: 'Write tests',
        priority: 'high',
        createdAt: DateTime.now(),
      );

      final completed = task.copyWith(isComplete: true);

      expect(completed.isComplete, isTrue);
      expect(completed.title, equals('Write tests'));
      expect(completed.id, equals(1));
    });

    test('toJson and fromJson round-trips correctly', () {
      final task = Task(
        id: 2,
        title: 'Ship the tool',
        priority: 'normal',
        createdAt: DateTime.parse('2025-01-01T00:00:00.000'),
      );

      final json = task.toJson();
      final restored = Task.fromJson(json);

      expect(restored.id, equals(task.id));
      expect(restored.title, equals(task.title));
      expect(restored.priority, equals(task.priority));
    });
  });
}

Testing the headers parser:

import 'package:test/test.dart';

import '../lib/utils/headers_parser.dart';

void main() {
  group('parseHeaders', () {
    test('parses a single header correctly', () {
      final result = parseHeaders(['Authorization: Bearer mytoken']);
      expect(result['Authorization'], equals('Bearer mytoken'));
    });

    test('parses multiple headers', () {
      final result = parseHeaders([
        'Authorization: Bearer token',
        'Accept: application/json',
      ]);
      expect(result.length, equals(2));
      expect(result['Accept'], equals('application/json'));
    });

    test('ignores malformed headers without a colon', () {
      final result = parseHeaders(['malformed-header']);
      expect(result.isEmpty, isTrue);
    });
  });
}

Run your tests:

dart test

Deploying and Distributing Your CLI

Building a CLI tool is half the work. Getting it into the hands of developers is the other half. There are five distribution paths available, each suited to a different use case.

Mode 1: pub.dev — Public Package Distribution

Publishing to pub.dev makes your tool installable by anyone in the Dart and Flutter community with a single command.

Prepare your package:

Your pubspec.yaml needs to be complete:

name: dart_http
description: A lightweight API request runner for Dart developers.
version: 1.0.0
homepage: https://github.com/yourname/dart_http

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  dart_http: dart_http

The executables block is critical. It tells pub.dev which script in bin/ to expose as a runnable command.

You also need:

• README.md — what the tool does, how to install it, usage examples

• CHANGELOG.md — version history

• LICENSE — an open source license (MIT is standard)

Validate before publishing:

dart pub publish --dry-run

This runs all validation checks without actually publishing. Fix any warnings before proceeding.

Publish:

dart pub publish

You will be prompted to authenticate with your pub.dev account. Once published, your tool is available globally:

dart pub global activate dart_http
dart_http get https://api.example.com/users

Mode 2: Local Path Activation

For internal team tools that you don't want to publish publicly, activate directly from a local or cloned repository:

dart pub global activate --source path /path/to/dart_http

Any developer on the team clones the repo and runs this command once. The tool is then available globally in their terminal without needing a pub.dev publish.

This is the right distribution mode for:

• Internal company tooling

• Tools that depend on private packages

• Work-in-progress tools shared within a team before a public release

Mode 3: Compiled Binary via GitHub Releases

Dart can compile to a self-contained native executable — no Dart SDK required on the target machine. This makes your tool accessible to developers outside the Dart ecosystem.

Compile:

# macOS
dart compile exe bin/dart_http.dart -o dist/dart_http-macos

# Linux
dart compile exe bin/dart_http.dart -o dist/dart_http-linux

# Windows
dart compile exe bin/dart_http.dart -o dist/dart_http-windows.exe

The compiled binary is fully self-contained. Copy it to any machine and run it — no Dart installation needed.

Automate with GitHub Actions:

Create .github/workflows/release.yml:

name: Release

on:
  push:
    tags:
      - 'v*'

jobs:
  build:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v3

      - uses: dart-lang/setup-dart@v1
        with:
          sdk: stable

      - name: Install dependencies
        run: dart pub get

      - name: Compile binary
        run: |
          mkdir -p dist
          dart compile exe bin/dart_http.dart -o dist/dart_http-${{ runner.os }}

      - name: Upload binary to release
        uses: softprops/action-gh-release@v1
        with:
          files: dist/dart_http-${{ runner.os }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Every time you push a version tag (v1.0.0), GitHub Actions compiles binaries for all three platforms and attaches them to the GitHub Release automatically.

Write an install script:

#!/usr/bin/env bash
set -euo pipefail

VERSION="1.0.0"
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
BINARY="dart_http-$OS"
INSTALL_DIR="/usr/local/bin"

curl -L "https://github.com/yourname/dart_http/releases/download/v\(VERSION/\)BINARY" \
  -o "$INSTALL_DIR/dart_http"

chmod +x "$INSTALL_DIR/dart_http"
echo "dart_http installed successfully"

Developers install it with:

curl -fsSL https://raw.githubusercontent.com/yourname/dart_http/main/install.sh | bash

Mode 4: Homebrew Tap

Homebrew is the standard package manager for macOS and is widely used on Linux. A Homebrew tap makes your tool installable with brew install — the most familiar installation pattern for macOS developers.

Create your tap repository:

Create a new GitHub repository named homebrew-tools (the homebrew- prefix is required by Homebrew's naming convention).

Write the formula:

Create Formula/dart_http.rb in that repository:

class DartHttp < Formula
  desc "A lightweight API request runner for the terminal"
  homepage "https://github.com/yourname/dart_http"
  version "1.0.0"

  on_macos do
    url "https://github.com/yourname/dart_http/releases/download/v1.0.0/dart_http-macOS"
    sha256 "YOUR_SHA256_HASH_HERE"
  end

  on_linux do
    url "https://github.com/yourname/dart_http/releases/download/v1.0.0/dart_http-Linux"
    sha256 "YOUR_SHA256_HASH_HERE"
  end

  def install
    bin.install "dart_http-#{OS.mac? ? 'macOS' : 'Linux'}" => "dart_http"
  end

  test do
    system "#{bin}/dart_http", "--help"
  end
end

Generate the SHA256 hash for each binary:

shasum -a 256 dist/dart_http-macOS

Install from the tap:

brew tap yourname/tools
brew install dart_http

When you release a new version, update the url and sha256 values in the formula and push the change. Users run brew upgrade dart_http to update.

Mode 5: Docker

Docker distribution is best suited for CI environments, teams that standardise on containers, or tools with complex dependencies.

Write a Dockerfile:

FROM dart:stable AS build

WORKDIR /app
COPY pubspec.* ./
RUN dart pub get

COPY . .
RUN dart compile exe bin/dart_http.dart -o /app/dart_http

FROM debian:stable-slim
COPY --from=build /app/dart_http /usr/local/bin/dart_http

ENTRYPOINT ["dart_http"]

This uses a multi-stage build: the first stage compiles the binary using the Dart SDK image, and the second stage copies only the binary into a minimal Debian image. The final image has no Dart SDK — just the compiled binary.

Build and run:

docker build -t dart_http .
docker run dart_http get https://jsonplaceholder.typicode.com/users/1

Publish to Docker Hub:

docker tag dart_http yourname/dart_http:1.0.0
docker push yourname/dart_http:1.0.0

Users can then run your tool without installing anything locally:

docker run yourname/dart_http get https://api.example.com/users

Choosing the Right Distribution Mode

For most tools, the practical recommendation is:

• Start with pub.dev if your audience is Dart developers

• Add compiled binary + GitHub Releases once you want broader adoption

• Add a Homebrew tap when macOS developers start asking for it

• Use Docker only when it is already part of your team's workflow

Conclusion

You've gone from understanding what a CLI is to building three progressively complex tools and distributing them across five different channels.

The foundational skills – args, stdin, stdout, stderr, exit codes, file I/O, and process spawning – are the same building blocks that tools like flutter, git, and dart themselves are built on. Everything else is composition.

The three CLIs we built (Hello CLI, dart_todo, and dart_http) each introduced a new layer: raw Dart fundamentals, the args package with JSON persistence, and real-world HTTP interaction. The distribution section ensures that whatever you build next, you have a clear path to getting it in front of the developers who will use it.

Dart is a powerful language for CLI development. Its strong typing, async support, native compilation, and pub.dev ecosystem make it a serious choice for building developer tooling, not just mobile apps.

The next step is building something that solves a real problem for you or your team, and shipping it.

Happy coding!!

The problem only showed up when I checked which authentication middleware the agent had imported.

Our codebase had two: a v1 middleware backed by MongoDB and a v2 middleware backed by MySQL, which we had spent the previous quarter migrating.

New endpoints were supposed to use v2. The agent had used v1 for all three. Tests passed because user records still existed in both databases (that was the point of the migration), and the v1 middleware happily authenticated them. The code worked. But every new endpoint we shipped was reinforcing the legacy auth path we had just spent a quarter trying to retire.

I caught it on the second read. Twenty minutes after the comments, the engineer fixed it and reopened the PR. The third reviewer probably wouldn't have caught it. The migration timeline lived in a Slack thread from six months earlier. The rule that "new endpoints use v2" lived in my head.

This kind of catch is the slow-burn version of why AI changed my job as a tech lead. Code generation got faster. My review queue got longer. The hardest reviews were the ones where everything looked right, and the only thing wrong was something that lived in the team's collective memory rather than in the diff.

This handbook is about what we did to fix that. It's the story of how we went from drowning in clean-looking PRs to running a custom AI PR reviewer that catches a meaningful share of these mistakes before any human is pulled in. The fix turned out to be less about buying a better tool and more about moving the team's memory into a place the AI could actually read.

The lessons should transfer whether your team uses Claude Code, Cursor, Cline, GitHub Copilot, or any combination. The structure matters more than the tool.

Table of Contents

• The Old Bottleneck, and the One AI Created

• What the New Review Work Actually Looks Like

• Why I Did Not Just Buy a Tool

• The Realisation: Move the Rules Into the Codebase

• Two Files That Changed Everything: AGENTS.md and CLAUDE.md

• Where Per-Service Memory Files Earn Their Keep

• What This Looks Like on Disk

• Generated Documentation as a Side Effect

• Building the PR Review Command

• Guardrails: Read-Only by Default

• The Compounding Loop That Made the Real Difference

• Starting From Zero on an Existing Project

• What Still Needs Human Review

• A Two-Week Setup Plan

• What Is Working, What I Am Still Improving

• Sources

The Old Bottleneck, and the One AI Created

To understand why this fix was needed, it helps to remember what reviewing code looked like a couple of years ago.

Back then, the slow part was upstream of the PR. A ticket would land, and before anyone could open a branch, there was a long preamble of context-gathering.

Junior engineers needed time to understand what the change was for. Senior engineers had to explain business rules and architectural decisions. Tickets sat in "ready" columns for days while someone with the right context made themselves available. Then the writing itself took time, because typing real code is slower than typing comments about it.

That bottleneck mostly dissolved when the team got serious about AI-assisted development. Engineers used the agent to read the codebase, ask clarifying questions, draft an implementation plan, and produce a working branch in hours instead of days. Tickets moved through the queue faster. Junior engineers shipped more without blocking on senior availability. From the outside, this looked like an unambiguous win.

But the bottleneck didn't disappear. It moved.

Within a few weeks of widespread AI adoption, my review queue had doubled. Then tripled. Engineers were opening PRs faster than I could read them.

The PRs themselves looked clean: well-formatted, with sensible variable names, passing tests, and AI-generated descriptions that read better than most human-written ones.

On the surface, this was great. In practice, it was creating a different kind of pain. I was the senior engineer who knew which patterns mattered and which paths through the codebase were the right ones, and I was the bottleneck. The team's velocity was now capped by my reading speed.

The CircleCI 2026 State of Software Delivery report confirmed I was not alone. Drawing on more than 28 million CI workflow runs across over 22,000 organisations, the report showed feature branch throughput had grown 59% year over year, the largest jump CircleCI had ever measured. Main branch throughput, where code actually gets promoted to production, fell by 7% for the median team in the same period. Build success rates dropped to 70.8%, the lowest in five years.

The pattern was consistent across the industry. AI accelerated writing. The rest of the system absorbed the cost.

So the question for me, as a tech lead, became concrete: how do I unblock myself without lowering the bar?

What the New Review Work Actually Looks Like

Before I explain the fix, it helps to know what kinds of issues were actually piling up. They weren't the dramatic kind. None of them would crash production. They were small, recurring, and looked plausible at a glance.

Take the simplest case I kept catching. An engineer would ask the agent to add a delete button on a new screen. The button needed to call our existing backend delete endpoint. Instead of reusing the hook the team already had for that endpoint, the agent would write the fetch call inline.

The code worked. The tests passed. But a week later, when someone changed the backend response shape, only one of the two call sites got updated.

That kind of duplication doesn't show up in a code review unless the reviewer happens to remember that a hook exists.

Another example I saw constantly: the agent comparing a status field against the literal string "completed" instead of using the Status.Completed enum that the rest of the services used. The code ran. The tests ran. The next refactor of the enum quietly skipped the file. After a few days, someone would spend half a day debugging a state machine that was working fine until the agent's literal silently fell out of sync.

These were two-minute fixes once spotted, but spotting them took me a reasonable time per PR. The friction wasn't the difficulty. It was the repetition.

The pattern repeated across larger problems, too.

I once asked an agent to build an event creation wizard. The wizard needed several dropdowns and one new component.

We have a design system folder where shared UI components live, and the rule on the team is simple: check there first, and if you build something new, register it there.

The agent had no way to know that. It only loaded the wizard's own files, so it never opened the design system folder. It generated brand new dropdowns inline, with APIs that were almost identical to the ones we already had. The new component went straight into the wizard rather than into the design system. CI passed. The wizard worked. We caught the duplication in human review, but it was the kind of catch that depended entirely on a reviewer who happened to know the design system existed.

The same pattern hit in one of the repos I was looking at for backend architecture. Backend follows a strict four-layer pattern: route, controller, app, repo. Controllers must never call repository functions directly. That rule keeps authorisation centralised, business logic testable, and database concerns isolated.

One PR I reviewed had the agent calling repo functions straight from a controller, skipping the app layer entirely. The code worked. The tests passed because the agent had also written tests against the new shape. But it broke a discipline the team had spent years building. If that PR had landed, the next AI-assisted PR could have used it as a template, and the layering would have eroded one diff at a time.

The common thread is that all of these mistakes had something written down somewhere, in code, in a Slack thread, in a senior engineer's head, that would have prevented them. The information existed. The agent just couldn't see it.

Why I Did Not Just Buy a Tool

The obvious next move was to install one of the AI PR reviewers that flooded the market in 2026.

I evaluated several. Anthropic launched Claude Code Review in March 2026, billed on token usage and averaging \(15 to \)25 per review. CodeRabbit Pro charges \(24 per developer per month on annual billing, or \)30 per developer per month on monthly billing, with seats counted against developers who actually open PRs. Greptile in March 2026 moved to a base-plus-usage model at $30 per seat per month, including 50 reviews, after which each additional review costs a dollar. GitHub announced that all Copilot plans will transition to usage-based billing on June 1, 2026, with code reviews consuming both AI Credits and GitHub Actions minutes from that date.

For a small team with low PR volume, none of these is a dealbreaker. For a larger team running heavy AI-assisted development, the costs compound fast. A 10-person team running five PRs each per day blows through Greptile's included reviews in a single week. CodeRabbit Pro at \(24 per seat scales linearly with developers. The premium Claude Code Review at \)15 to $25 per PR is the most expensive option per review by an order of magnitude.

I looked at the cost numbers, but cost wasn't actually the deciding factor. The deciding factor was that none of these tools would have caught the problems I just listed.

A generic reviewer wouldn't have caught the v1/v2 middleware. It had no way to know v2 was the canonical path. A generic reviewer wouldn't have caught the duplicate dropdowns. It had no way to know our design system existed. A generic reviewer wouldn't have caught the bypassed architecture. It had no way to know that controllers must not call repositories.

The information that lets a reviewer flag any of these is exactly the information that lives in the team's head, not in any tool's default prompt.

The better-rated tools support custom rules, and that's where I started to see the real shape of the problem. Once you are configuring custom rules, you've already accepted that the value is in the rules. The tool is just whatever runs them.

This raised a different question: if the rules are the product, why pay per seat or per review for someone else's wrapper around them?

This is what made me change direction.

The Realisation: Move the Rules Into the Codebase

Once I started thinking of the rules as the product, the path forward got clearer.

I asked myself a simple question: what was I actually doing in code review that the AI was not? The answer turned out to be the same thing, over and over. I was typing review comments that captured a piece of the team's memory.

"Use the Status enum, not a string literal." "There is already a hook for this in /hooks/useDeleteItem." "Controllers must not import from the repo layer; route this through the app layer." "Check the design system folder before creating new components."

Each of those comments was knowledge that lived in my head and arrived in the codebase one PR comment at a time. None of it was available to the agent the next time it generated a similar PR.

So the fix was not to buy a smarter reviewer. The fix was to write the rules down in a place every agent on the team would read before any review happened.

If I had typed "use the enum, not a literal" three times in three different PRs, that was a rule the agent should know about from now on. If I had pointed at the design system folder for the fourth time, that was a rule. If I had explained the four-layer architecture twice in PR comments, that was a rule.

I needed somewhere to put these rules. That turned out to be a less obvious decision than I expected.

Two Files That Changed Everything: AGENTS.md and CLAUDE.md

If you start looking into how to give an AI agent a persistent project context, you run into two competing conventions almost immediately.

The first is AGENTS.md, an open standard that has gathered real momentum. According to InfoQ, by mid-2025, the format had already been adopted by more than 20,000 GitHub repositories and was being positioned as a complement to traditional documentation: machine-readable context that lives alongside human-facing files like README.md.

The standard's own site reports it is now used by more than 60,000 open-source projects and has moved to stewardship under the Agentic AI Foundation, which sits inside the Linux Foundation. The format is supported by OpenAI Codex, GitHub Copilot, Google Gemini, Cursor, and Windsurf, among others.

The second is CLAUDE.md, which is Anthropic's convention for Claude Code. The Claude Code documentation describes two complementary memory systems: CLAUDE.md, where you write the persistent context yourself, and an auto-memory mechanism that lets Claude save its own notes from corrections and observed patterns. By default, Claude Code reads CLAUDE.md, not AGENTS.md.

This split mattered for us because half the team uses Claude Code and the other half uses Cursor. We had two practical options: maintain both files with the same content (and accept the duplication), or symlink one filename to the other so both ecosystems read the same source of truth. We went with the symlink. It's one less thing to drift.

The next question was what to actually put in the file. After a few iterations, here's the shape that worked. Think of it as a briefing document for a new engineer who has read no code and seen no Slack threads. The minimum content was:

• The tech stack (languages, frameworks, package manager)

• The project structure, especially important for our monorepo

• Where shared utilities, components, and helpers live, and the rule that new code should reuse them before creating new versions

• Architectural patterns the project follows, with file path examples

• Anti-patterns and what to do instead

• Test conventions and where good examples live

• Pointers to deeper documentation when more detail is needed

Two practical rules emerged from the first month of using these files.

Keep them lean: There is a counterintuitive failure mode with long instruction lists: the agent doesn't just skip the new ones at the bottom. The average compliance across all of them drops. A bloated memory file becomes a memory file that the agent skims. If a section runs more than a paragraph or two, move it to a separate document and link to it.

Phrase rules as imperatives, not aspirations: "Controllers must not call repositories. Route through the app layer." beats "Try to keep controllers thin." The first is testable. The second is decorative.

That was the entry point. But a single root-level file was not enough for a monorepo with multiple services and frontends, which led to the next decision.

Where Per-Service Memory Files Earn Their Keep

A single AGENTS.md at the root of a monorepo collapses under its own weight pretty quickly. Each service in our codebase has its own architecture, conventions, and business rules. Trying to fit all of that into one file produced a long document that the agent treated as background noise, and we were back to the bloat problem from the previous section.

The pattern that worked: every service or app gets its own AGENTS.md at its root, and the project-level AGENTS.md becomes an index that points to them.

A per-service AGENTS.md covers things like:

• The architecture for this service (the four-layer pattern, the directory layout)

• Naming conventions specific to this service

• Test patterns and where good examples live

• Business rules that this service is responsible for

• Inter-service contracts and what other services consume from this one

• Pointers to deeper docs in docs/

• A "Lessons learned" section, which I'll come back to in the section on the compounding loop

The same lean rule applies. Keep it short, point at examples, and phrase guidance as imperatives.

The reason this works mechanically is that the agent loads the right files for the work at hand. When an engineer asks the agent to change something in backend/, the agent reads the project-level AGENTS.md, sees that work in backend/ should be guided by backend/AGENTS.md, and loads that file. It doesn't load the frontend's AGENTS.md, because that work is somewhere else. The context window stays focused on what's relevant.

Without this split, you have two bad options. Either you put everything in the root file, where the agent ignores most of it, or you put nothing in the root file, where the agent has no team context at all. The per-service split gives you both depth and signal.

But these files only work if the deeper docs they point to actually exist, which is where the next piece of the system came in.

What This Looks Like on Disk

Before going further, it helps to see the whole structure laid out. Here's the shape we settled on for our monorepo. The exact folder names follow Claude Code's conventions. If you use Cursor, it would be .cursor/, and if you use Cline, it would be .clinerules – but the shape transfers directly.

project-root/
├── AGENTS.md                       # symlink to CLAUDE.md
├── CLAUDE.md                       # root memory file
├── README.md                       # human-facing project readme
│
├── .claude/                        # tool-specific config folder
│   ├── README.md                   # explains the .claude/ layout
│   ├── settings.json               # permissions and guardrails
│   ├── agents/                     # specialised subagents (optional)
│   ├── commands/                   # slash commands engineers run
│   │   ├── review-pr.md            # the PR review command
│   │   └── plan-feature.md         # implementation plan command
│   ├── hooks/                      # lifecycle hooks (optional)
│   ├── pr-rules/                   # rule files for PR review
│   │   ├── common.md               # rules that apply to every PR
│   │   ├── frontend.md             # rules for frontend changes
│   │   ├── backend.md              # rules for backend changes
│   │   ├── service-a.md            # rules for service-a
│   │   └── service-b.md            # rules for service-b
│   └── skills/                     # reusable workflows
│
├── frontend/
│   ├── AGENTS.md                   # frontend conventions
│   ├── docs/
│   │   ├── overview.md
│   │   ├── architecture.md         # routing, state, data layer
│   │   ├── design-system.md        # design system reference
│   │   └── testing.md              # test conventions
│   └── src/
│
├── backend/
│   ├── AGENTS.md                   # the four-layer pattern
│   ├── docs/
│   │   ├── overview.md
│   │   ├── architecture.md         # route -> controller -> app -> repo
│   │   ├── auth.md                 # v1 vs v2 middleware
│   │   ├── business-rules.md
│   │   └── integrations.md
│   └── src/
│
├── service-a/
│   ├── AGENTS.md
│   ├── docs/
│   │   ├── overview.md
│   │   ├── business-rules.md
│   │   └── integrations.md
│   └── src/
│
└── service-b/
    ├── AGENTS.md
    ├── docs/
    │   ├── overview.md
    │   ├── business-rules.md
    │   └── integrations.md
    └── src/

A few things worth pointing out:

The .claude/ folder uses standard subfolder names: commands, agents, hooks, skills. These follow Claude Code's plugin model, but most modern AI coding tools have similar slots. Following the conventions makes the structure recognisable to anyone on the team and lowers the cost of switching tools later.

The pr-rules/ folder isn't a standard convention. It's a folder we created to hold per-area review rules that the PR review command loads selectively. You don't have to call it pr-rules – the name matters less than having one place where review rules live.

Each service has its own AGENTS.md plus a docs/ folder. The root AGENTS.md is short and acts as an index. It tells the agent things like "if you touch files in backend/, also read backend/AGENTS.md first." The per-service file then points at the deeper docs as needed.

Generated Documentation as a Side Effect

Setting up per-service AGENTS.md files surfaced a problem I had been quietly avoiding. Most of our services didn't have decent documentation. Not API reference material, which lives in code, but the higher-level "what does this service do, what business rules does it enforce, what does it consume and produce" information that lives in nobody's head except the original author's.

The honest reason was that writing this kind of documentation by hand had never paid back the time it took. By the time the doc was finished, half of it was already stale.

So I tried something I wouldn't have considered earlier. I used the AI itself to generate a first draft for each service. I pointed the agent at each service's code and asked it to produce a docs/ folder with a specific structure: an overview, a list of business rules, an integrations document, a domain model, and any quirks worth knowing. The agent read the code, traced the call paths, and wrote a draft.

I then reviewed the output by hand, corrected the things it got wrong, and committed the result. The first drafts were 70-80% correct. The remaining 20-30% was where the agent had made plausible but wrong inferences, and those were exactly the cases where human review mattered.

The generated docs ended up serving two audiences. The agent uses them when reasoning about changes, which means it has real context for the service it's touching rather than guessing from local files. And new engineers use them on their first day, which has cut our onboarding time noticeably.

We used to write onboarding documents that drifted out of date within months. These docs stay closer to current because the agent reads them on every PR, and any drift gets surfaced when the agent gives wrong advice based on stale information.

The pattern that works is to keep the per-service AGENTS.md short and pointing at the docs, rather than duplicating their content. AGENTS.md is the always-loaded index. docs/ holds the details. The agent loads the relevant doc on demand when the task calls for it.

With the rules in place and the docs in place, I had everything I needed to build the actual reviewer.

Building the PR Review Command

This is the piece that most directly unblocked my queue.

This command didn't appear out of nowhere. It started as the checklist I was running through in my head every time I opened a PR. I was reviewing every change manually, leaving the same comments, flagging the same patterns. So I wrote that checklist down, expanded it with references to the per-service docs for the harder rules, and turned it into a command anyone on the team could run.

Then I handed it to the engineers and changed the rule: run this on your own branch before marking the PR ready for review. That single shift moved the work from after the PR was opened to before. Engineers now catch 90-95% of the blockers, improvements, and nice-to-haves on their own machine, fix them locally, and only then push the change.

The PR description includes the AI's summary, so when anyone opens the PR, they can see the reviewer's green signal at the top before even reading the diff.

GitHub stays clean. The conversation on the PR becomes about the things that actually need a human, not the recurring stuff the team already knows how to fix.

The command lives in .claude/commands/review-pr.md. Here's a generalised version. Your tool's command structure may differ, but the shape is what matters.

# Review PR

Review the current branch's PR. Be direct. Cite `file:line`. Surface real issues,
no padding.

## 1. Scope the diff

Run, in order:

    gh pr view --json number,title,body,headRefName 2>/dev/null || true
    git fetch origin main
    git log --no-merges origin/main..HEAD --oneline
    git diff origin/main...HEAD --stat
    git diff origin/main...HEAD

Read the PR body. Note the stated intent. Every change should trace to it. Flag
anything that does not.

Use `...` (three dots) for the diff. It compares against the merge base and
excludes commits brought in by merging main.

## 2. Load rules

Always read `.claude/pr-rules/common.md`.

Then read the per-area file for each workspace touched in the diff:

| Workspace path | Rules file                      |
| -------------- | ------------------------------- |
| `frontend/**`  | `.claude/pr-rules/frontend.md`  |
| `backend/**`   | `.claude/pr-rules/backend.md`   |
| `service-a/**` | `.claude/pr-rules/service-a.md` |
| `service-b/**` | `.claude/pr-rules/service-b.md` |

For non-trivial changes, follow doc pointers inside the rules files (for
example, `backend/AGENTS.md`, `backend/docs/architecture.md`).

Apply every entry under each file's "Lessons learned" section as a check.

## 3. Output

Use exactly this format.

    ## Summary
    <one paragraph: what the PR does, whether it matches the stated intent>

    ## Blocking
    - [file:line] issue, why it blocks

    ## Should fix
    - [file:line] issue

    ## Nice to have
    - issue

    ## Verified
    - what was checked and looks good

If nothing blocks, say so. Do not manufacture concerns.

If you find an issue worth remembering for future PRs, suggest the bullet to
add to the relevant rules file's "Lessons learned" section. Do not edit the
rules file yourself, leave that to the human.

A few of the design choices in this command turned out to matter more than I expected.

The structured output format (Summary, Blocking, Should fix, Nice to have, Verified) keeps the review easy to scan and easy to paste into a PR description. The "Verified" section is the most underrated of the five: it tells the human reviewer what the AI already checked, so they can spend their attention elsewhere. Without it, the human reviewer ends up doing the same checks twice.

The instruction to be direct and stop padding does real work. Without it, AI reviewers tend to manufacture concerns to look thorough, which trains engineers to skim past the bot. Telling it explicitly to say "nothing blocks" when nothing blocks made the signal-to-noise ratio of the output much better.

The "suggest a bullet for the rules file" instruction at the end is the heart of the whole system, and I'll explain why in the section on the compounding loop. The key constraint here is that the agent suggests the bullet but doesn't commit to it. A human evaluates whether it's general enough to be a rule, and only then adds it to the file. That manual step is what keeps the rules sharp instead of bloated.

With each PR, if humans fix something or the AI suggests something, you keep adding those to your MD files and keep improving your agents for the future. The result compounds quickly.

One more thing here: the diff-scoping commands are all read-only. The command shouldn't be able to push, edit PRs, or close anything. Which is the next piece of the system.

Guardrails: Read-Only by Default

Giving an AI agent broad permissions on your codebase is a security incident waiting to happen. Even if you trust the model to behave, an LLM occasionally does unexpected things, and a fast-moving agent on an unrestricted shell can cause damage in seconds.

The fix is a settings.json (in Claude Code – other tools have their own equivalents) at the root of .claude/ that explicitly declares what the agent can and can't do. The deny list matters more than the allow list, and a good one is organised around four categories of risk.

The first is secrets and configuration. Any read against anything that appears to be a credential is blocked. That covers .env files of every variant (.env, .env.local, .env.production, .env.test, and so on), .npmrc, .netrc, .pgpass, id_rsa, id_ed25519, *.pem, *.key, *.p12, **/credentials.json, **/secrets.json, **/.aws/**, **/.ssh/**, **/.gcloud/**, and **/.kube/**. Environment dumps are blocked too: env, printenv, set, export. The agent has no legitimate reason to read or echo any of these, ever.

The second is destructive Git operations. The agent can read Git history but can't rewrite or push it. Blocked: git push, git commit, git revert, git cherry-pick, git merge, git rebase, git reset --hard, git tag. Allowed: git fetch, git status, git log, git diff, git show, git branch, git rev-parse, git merge-base, git config --get.

The third is write operations on PRs and issues. The agent can read your GitHub state but can't act on it. Blocked: gh pr create, gh pr edit, gh pr merge, gh pr close, gh pr comment, gh pr review, gh issue create, gh issue edit, gh issue close, gh issue comment, gh release create, gh repo create, gh repo edit, gh repo delete. Allowed: gh pr view, gh pr list, gh pr diff, gh pr checks, gh issue view, gh issue list, gh release view.

The fourth is workflow and automation control. These are the surfaces where a compromised or misled agent could do the most damage. Blocked: gh workflow run, gh run rerun, gh run cancel, gh secret, gh variable, gh auth, gh ssh-key, gh gpg-key, and the unrestricted gh api.

For shell commands the agent legitimately needs to run, like build and test commands, allowlist specific patterns: pnpm test, pnpm lint, pnpm format:check, pnpm build, pnpm vitest. Anything outside the allowed list requires human confirmation. These are your own settings – I've just mentioned what I prefer.

The pattern is simple: read-only by default, write-allowed only for the specific commands you have explicitly approved. The agent can investigate, plan, and recommend. It can't ship.

With the structure in place and the guardrails set, the system started doing its job. What I didn't expect was how much better it would get over the months that followed.

The Compounding Loop That Made the Real Difference

When we started, the AI reviewer was useful but not transformative. It caught some obvious issues, missed plenty of subtle ones, and produced a fair amount of noise.

The first month, my review burden dropped by 35%. The time I was spending on PR checking was reduced to 1/3, almost. Decent, not life-changing.

What changed over time wasn't the tool. It was the rules.

Every time a PR creator and reviewer caught something the AI had missed, we were adding bullets to the relevant rules file. Every time the AI flagged something useful that turned out to be a recurring pattern, the agent's own suggestion at the end of the review went into the file.

After a few days, the rules files had grown into something that captured a meaningful fraction of the team's collective review knowledge, written down in a place every agent on the team would read.

The catch rate went up. The noise went down because the rules also said what was acceptable and what we already considered solved. New engineers stopped getting the same comments on their first three PRs because the AI caught the comments first. Engineers joining the team didn't have to absorb the conventions through six months of review feedback. They installed the project, opened it in their editor, and the agent already knew.

This is the part most teams miss when they evaluate AI PR review tools. They look at the catch rate today and decide whether the tool is worth the price. The catch rate today isn't the right number. The right number is what the catch rate looks like in six months, after the rules file has absorbed every recurring mistake your team has made.

A single rule written down today saves a small amount of review time. Over a hundred PRs, it saves more. After a year, the rules file is a written-down version of a tech lead's accumulated taste. We've switched between Claude Code, the GitHub Copilot CLI, and Cursor for various tasks during this period. The AI tool changes, but the rules file in the repo stays the same.

The discipline that makes this work is treating the rules file as living documentation. Every recurring review comment is a candidate for promotion into the file. If you catch yourself typing the same feedback in two different PRs, that's a rule that belongs in pr-rules/. The "suggest a bullet" instruction in the review command is what makes this practical: the AI does the typing, the human does the deciding.

This is also what made me realise the system was worth the time it took to set up. The PR review command, on its own, is useful but unremarkable. The compounding loop is what turns it into infrastructure.

Starting From Zero on an Existing Project

If you've read this far and feel like the gap between your project and what I just described is a sprint of work, that's the most common reaction. It's also not correct.

The blank AGENTS.md is intimidating, especially on an existing codebase. You know your team has a thousand conventions, and writing a thousand rules sounds like a project that takes weeks before it produces any value.

The honest answer is that you can't write all the rules up front, and you shouldn't try. The first version of any of these files should take an afternoon, not a sprint.

Here's how I would actually start.

Run /init (or your tool's equivalent). In Claude Code, /init scans the project, infers the obvious shape (language, framework, entry points, build commands), and writes an initial CLAUDE.md. The output is a starting point, not a finished file. Read it, delete most of what it generates, and keep the bones.

Then add three things, each one bullet long.

First, an architecture rule. Pick the single most important convention your team enforces. For us, that was the four-layer pattern. The bullet was: "Controllers must not call repository functions directly. They must go through the app layer."

Second, a discoverability rule. Pick the single most important shared resource the team has, the one new code is most likely to duplicate. For us, that was the design system. The bullet was: "Before creating a new UI component, check /src/design-system/ first."

Third, a "do not touch" rule. Pick the single most dangerous file or area in the codebase. Auth, billing, or migrations whichever has the most production risk. The bullet was: "Do not modify files in /auth/ without human approval."

That's enough to start. Three rules, ten minutes of writing, and most of your team's recurring AI mistakes start to drop.

If even three rules feels like too much, start with one. Pick a single line that matters in your codebase and write it down.

"No any types in TypeScript." "Always use the enum, never compare against the string literal." "Run the linter before opening a PR." It doesn't have to be sophisticated. It doesn't have to cover edge cases. It just has to capture one piece of judgement that lives in your head today and would otherwise stay there.

Tomorrow, add another. The first week, you might catch 5% of the recurring mistakes. By 20 or 30 PRs in, you might catch 20-30%. The rules file doesn't need to be impressive on day one. It needs to exist and keep growing.

This is the compounding effect I'll come back to soon, and it's the reason this approach works on real projects rather than just in theory.

From there, the file grows the same way it would grow for any team. Every review catch becomes a candidate rule. After a few weeks, you have ten or fifteen rules. After a few months, you have a real review system.

The mistake is trying to write the perfect file on day one. The right file is the one you start with and keep editing.

What Still Needs Human Review

This system doesn't replace human review, and it shouldn't be allowed to.

The AI reviewer catches what the rules describe, plus a fair number of obvious things it would have spotted anyway. It doesn't catch problems that depend on context the rules don't capture. It doesn't catch product judgement. It doesn't catch the question of whether the change should have been built at all.

It also has an important blind spot when reviewing AI-authored code. The reviewer shares the same training data and reasoning patterns as the agent that wrote the code. If the original agent missed the v1/v2 distinction because it had no way to see the migration timeline, an AI reviewer reading the same diff has the same problem. Two AIs in a review loop are not two independent reviewers. They share blind spots.

That is why the AI reviewer in this setup never approves a PR. It produces a structured review that goes into the PR description. A human still reads the change and approves it. The AI is the first pass, not the gate.

Accountability also has to live with a human. When something the AI approved breaks production, someone has to own the post-mortem and decide what changes are needed for next time. The AI can't be that person. What it can do, well, is reduce the stack of small mistakes a human reviewer has to find before they get to the harder questions.

A Two-Week Setup Plan

If you want to set this up for your own team, here's a concrete plan that fits in a couple of weeks. None of this needs to happen in a single push.

Day 1: Bootstrap the memory file.

Run /init (or your tool's equivalent) at the root of the project. Read the generated CLAUDE.md (or AGENTS.md). Delete most of it. Keep the tech stack and project structure sections.

Add the three rules from the previous section: one architecture rule, one discoverability rule, and one "do not touch" rule. Decide whether you want both files or a symlink.

Day 2: Add per-service files for your highest-risk areas

Pick the two or three areas of the codebase that change most often or carry the most risk. Add an AGENTS.md to each, following the same lean pattern. Include the architectural pattern for that area, the naming conventions, where to find good test examples, and pointers to any existing docs. Skip anything that doesn't need to be there yet.

Day 3: Set up the directory structure and guardrails

Create a .claude/ folder (or your tool's equivalent) at the root, with commands/ and pr-rules/ subfolders. Add a settings.json with the deny list categories from the guardrails section. Test that the agent can't read a .env file, run git push, or create a PR. If any of those work, fix the settings before doing anything else.

Day 4: Write the PR review command

Adapt the command in this article to your structure. Include the diff scoping, the rule loading, the output format, and the "suggest a new rule" instruction at the end. Run it on a branch you've already merged, and tune the output until it's useful.

Day 5: Run it on real PRs

Have one or two engineers run the command on their next PRs before opening them. Read the output. Note what it caught, what it missed, and what was noise. Add the missing catches to the rules files. The first week is mostly tuning.

Week 2: Roll out and document

Once the command produces useful output reliably, ask the whole team to run it before opening PRs and paste the output into the PR description. Add a short section to your contributing guide explaining the workflow. Set a recurring item in your team's rituals to review the rules files monthly and trim anything that has gone stale.

That gets you to a working system. From there, the maintenance is incremental. Every recurring review comment becomes a candidate rule. Every architectural decision becomes a candidate update to the relevant AGENTS.md. The system improves as a side effect of the work the team is already doing.

What Is Working, What I Am Still Improving

Here's my honest assessment after a few months of running this:

What's Working

My review burden is meaningfully smaller. Engineers fix most of the easy mistakes before I see the PR. The "Verified" section of the AI's output tells me what to skip past. New engineers ramp faster because the conventions live in a place their tooling reads. The rules files have grown into something I would actually use to onboard someone new.

What Isn't Finished

The AI still misses problems that depend on context, and the rules don't capture them. The rules files grow, but they also need pruning, and we haven't been disciplined about that.

We're still figuring out how to handle rules that apply only conditionally. Docs are helping in that case, but we need to keep those up to date. And no system survives a determined engineer who skips the workflow or docs when they're in a rush.

There's no shortcut here. The work is real, ongoing, and mostly about discipline. The discipline is treating your codebase as something the AI needs to learn, and treating every recurring review comment as something that should be written down once instead of typed thirty times. If you're willing to do that, the tools take care of the rest.

If you take three things from this article, take these.

1. First, don't pay for a generic reviewer to do a job your codebase needs to inform. Generic reviewers catch generic problems. Most of your real review work is specific to your team.

2. Second, put the rules in a file the AI reads, not in your head. AGENTS.md, CLAUDE.md, per-service files, per-area rules files. Pick a structure and stick to it.

3. Third, treat every human review catch as a chance to update the rules. The compounding effect over months is the entire point. A review system that improves itself is worth more than any single tool.

That's the system. It took a couple of weeks to build the foundation and a few months for the rules to mature. It costs very little to run, and it has done more for our PR throughput than any tool I evaluated.

Sources

• CircleCI's 2026 State of Software Delivery report, analysing more than 28 million CI workflows from over 22,000 organisations: https://circleci.com/resources/2026-state-of-software-delivery/

• CircleCI's blog post detailing the year-over-year throughput numbers, including the 59% feature branch growth and the main branch decline: https://circleci.com/blog/five-takeaways-2026-software-delivery-report/

• GitHub announcement of Copilot's transition to usage-based billing on June 1, 2026: https://github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing/

• GitHub changelog confirming Copilot code review will start consuming GitHub Actions minutes on June 1, 2026: https://github.blog/changelog/2026-04-27-github-copilot-code-review-will-start-consuming-github-actions-minutes-on-june-1-2026/

• AGENTS.md, the open standard's official site, including its stewardship under the Agentic AI Foundation and the Linux Foundation: https://agents.md/

• Anthropic's Claude Code documentation on the memory system, including CLAUDE.md, auto memory, and the /init command: https://code.claude.com/docs/en/memory

• Anthropic's Claude Code GitHub Actions documentation, including notes on token-based billing and recommended cost controls: https://code.claude.com/docs/en/github-actions

• CodeRabbit's pricing documentation, confirming the per-developer-per-month seat model: https://docs.coderabbit.ai/management/plans

• Greptile's March 2026 pricing announcement, introducing the base-plus-usage model at $30 per seat per month with 50 included reviews: https://www.greptile.com/blog/greptile-v4

• HumanLayer's write-up on writing a good CLAUDE.md, including data on instruction-following degradation: https://www.humanlayer.dev/blog/writing-a-good-claude-md

And honestly? It was a fair point. You'd write the contract — the interface, the abstract class, the protocol — and then write the implementation. Two files where one would do. That's more surface area, more indirection, and more to maintain.

The Ruby and Rails communities built an entire philosophy around this: convention over configuration, less ceremony, fewer keystrokes. If the framework could infer your intent, why spell it out?

Then AI happened.

I was recently chatting with a CEO about what current-generation software engineers get wrong, and he put it cleanly:

"Abstract interfaces were challenging a few months ago just because it required twice as much code. But with AI, lines of code are free. The reason we still need such constructs is because at some point a human still needs to look at the code. Interfaces reduce the cognitive load."

That framing stuck with me. The cost of writing code has collapsed. The cost of reading it hasn't moved. And that asymmetry changes everything about how you should think about abstraction.

Here's what I mean.

Table of Contents

• Your Brain Is the Bottleneck

• The Greats Already Knew This

• The Economics Have Flipped

• The Data Backs It Up

• The Contrarian Case (And Why It Actually Agrees)

• What This Means for You

• References

Your Brain Is the Bottleneck

This isn't a vibes argument. There's actual neuroscience behind why interfaces help.

In 1988, educational psychologist John Sweller introduced Cognitive Load Theory. A 2022 ACM review covers how it's been applied to computing education since.

The short version: your brain juggles three types of load when processing information. Intrinsic load is the inherent difficulty of the problem itself. Extraneous load is the noise — poorly organized information, unnecessary details, bad naming. Germane load is the good stuff — the mental effort you spend building useful mental models.

Here's the kicker: your working memory can only hold a handful of chunks of information at a time — cognitive scientists typically estimate somewhere between 2 and 6. Not 2 to 6 files, or 2 to 6 classes — 2 to 6 things.

Felienne Hermans explores this in The Programmer's Brain (2021), arguing that design patterns act as chunking aids. When you recognize a Strategy pattern, your brain collapses an entire class hierarchy into a single cognitive unit. The word "Strategy" replaces five classes and their relationships. That's not hand-waving about clean code — that's how human memory actually works.

And we can literally see it on brain scans. In 2021, a team led by Norman Peitek and Janet Siegmund published an fMRI study on program comprehension that won the ACM SIGSOFT Distinguished Paper Award at ICSE.

They put developers in brain scanners and watched what happened when they read code. The finding: semantic-level comprehension — understanding what code does — required measurably less neural activation than bottom-up syntactic parsing — tracing how it does it.

An interface lets you comprehend at the semantic level. UserRepository.findById(id) tells you everything you need to know without opening the implementation. Your brain doesn't need to hold the SQL query, the connection pool logic, the error handling, and the result mapping in working memory simultaneously. The interface compresses all of that into one chunk.

That's not elegance. That's neuroscience.

The Greats Already Knew This

The case for abstraction isn't new. The people who built the foundations of computer science were making this argument before most of us were born.

Dijkstra said it with precision:

"The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise."

Abstraction isn't about hiding things from people who can't handle complexity. It's about creating a level of discourse where you can reason clearly.

David Parnas formalized information hiding in his 1972 ACM paper: "Every module is characterized by its knowledge of a design decision which it hides from all others." He proved that decomposing systems by design decisions (rather than processing steps) produced modules that were both more flexible and easier to understand. Comprehensibility wasn't a bonus — it was the design criterion.

Tony Hoare argued that abstraction is the most powerful tool available to the human intellect — a way to manage complexity by focusing on what matters and ignoring what doesn't. Martin Fowler brought it down to earth:

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

And then there's John Ousterhout, whose book A Philosophy of Software Design (2018) makes the connection to cognitive load explicit. His central argument: more lines of code can actually be simpler if they reduce cognitive load.

His concept of deep modules — simple interfaces hiding complex implementations — is essentially the argument that interfaces are worth their weight in code. The Unix file system API (open, close, read, write, lseek) is five functions hiding an enormous amount of complexity. That's a deep module. That's the goal.

The Gang of Four put it first in their book for a reason. Page one: "Program to an interface, not an implementation."

None of this is controversial. But it's easy to forget when your AI tool just generated 200 lines of perfectly functional inline code in three seconds.

The Economics Have Flipped

Here's where the CEO's insight becomes an economic argument.

The historical case against interfaces was always about writing cost. Interfaces meant more code to write, more files to create, more boilerplate to maintain. The entire dynamic typing movement — Python, Ruby, JavaScript — was partly a reaction to the ceremony that languages like Java imposed. Convention over configuration. Don't Repeat Yourself. Less is more.

But ask yourself: what exactly is the cost of writing boilerplate now?

GitHub's 2022 controlled study found that developers using Copilot completed tasks 55% faster. The boilerplate that used to justify skipping interfaces — the extra file, the type definitions, the method signatures — takes seconds to generate. The writing cost of an interface has effectively collapsed to zero.

But again, the reading cost hasn't budged.

Robert C. Martin argued in Clean Code (2008) that developers spend far more time reading code than writing it — an observation he framed as a ratio of 10 to 1.

You can quibble with the exact number (it's anecdotal), but the direction is consistent across studies. A large-scale field study tracking 78 professional developers across 3,148 working hours found they spend roughly 58% of their time on program comprehension alone. New developer onboarding averages six weeks — most of which is spent understanding existing systems, not producing new ones.

Addy Osmani named this asymmetry perfectly. In a March 2026 piece, he described comprehension debt:

"When a developer on your team writes code, the human review process has always been a bottleneck — but a productive and educational one. Reading their PR forces comprehension. AI-generated code breaks that feedback loop. The volume is too high."

The output looks clean, passes linting, follows conventions — precisely the signals that historically triggered merge confidence. But comprehension debt is distinct from technical debt because it accumulates invisibly — your velocity metrics, your DORA scores, your PR counts all look fine while your team's actual understanding of the codebase quietly erodes.

So here's the math: AI reduced the cost of writing abstractions to near zero. The cost of not having them — in human reading time, onboarding friction, and comprehension debt — hasn't changed at all. The break-even point for "is this interface worth it?" just shifted massively in favor of "yes."

The Data Backs It Up

This isn't theoretical. We have data on what happens when AI generates code without good abstractions.

GitClear analyzed 211 million changed lines of code between 2020 and 2024. Their findings: code churn — lines reverted or updated within two weeks — doubled compared to the pre-AI baseline. Copy-pasted code blocks rose from 8.3% to 12.3%. And refactoring-associated changes dropped from 25% to under 10%.

AI-generated code, as they put it, "resembles an itinerant contributor, prone to violate the DRY-ness of the repos visited."

The METR study (2025) found something even more striking. Experienced open-source developers predicted AI would make them 24% faster. They perceived being 20% faster while using it. They were actually 19% slower. The perception gap is the story — you feel productive while generating code that creates more work downstream.

And then there's a study from Anthropic (yes, the company that makes Claude — full disclosure). They observed 52 software engineers learning a new library. The AI-assisted group completed tasks at the same speed, but scored 17% lower on comprehension quizzes afterward — 50% versus 67%. The biggest declines were in debugging ability. You can ship code you don't understand. You can't debug code you don't understand.

Kent Beck put it bluntly: "The value of 90% of my skills just dropped to $0. The leverage for the remaining 10% went up 1000x." What that remaining 10% is, he leaves deliberately open — but it's hard to read that and not think about system design.

The Contrarian Case (And Why It Actually Agrees)

I'd be dishonest if I didn't address the people who argue against abstraction. And some of them are very smart.

Casey Muratori's "Clean Code, Horrible Performance" demonstrated that polymorphism and virtual dispatch can make code 10 to 15 times slower than straightforward procedural alternatives.

His benchmark is real. If you're writing a game engine or a high-frequency trading system, abstract interfaces on your hot path will cost you.

Dan Abramov wrote "Goodbye, Clean Code" after watching a premature abstraction make his codebase harder to modify:

"My code traded the ability to change requirements for reduced duplication, and it was not a good trade."

Sandi Metz put it more sharply: "Duplication is far cheaper than the wrong abstraction."

And Rich Hickey, in his talk "Simple Made Easy", draws the critical distinction: simple (not intertwined) is not the same as easy (familiar). Wrong abstractions complect — they braid concerns together rather than separating them.

Here's the thing: none of these are arguments against abstraction. They're arguments against bad abstraction.

Muratori's performance argument applies to hot paths in performance-critical systems — not to your REST API's service layer. Abramov and Metz argue against premature abstraction — pulling patterns out before you understand the domain. And Hickey's entire talk is a case for the right abstractions, the ones that genuinely decompose rather than complect.

The irony is that in an AI-assisted world, these arguments are easier to address. You can generate the explicit, unabstracted version first. Let it stabilize. Watch the patterns emerge. Then extract the abstraction — with AI handling the mechanical refactoring. The cost of the "duplicate first, abstract later" approach just dropped to near zero.

What This Means for You

If you're writing code with AI tools — and at this point, most of us are — the temptation is to let the AI produce whatever it produces and move on. It works. It passes the tests. Ship it.

But "it works" is table stakes. The harder question is: can the next person who opens this code understand it in under five minutes? Can you understand it in six months?

Interfaces aren't about making code prettier or satisfying some abstract (pun intended) design principle. They're compression algorithms for human cognition. They let your brain operate at the semantic level instead of the syntactic level. And now that AI has eliminated the only real cost of creating them — the boilerplate — there's no economic argument left for skipping them.

The rules haven't changed. The excuse has just expired.

References

Academic Papers

• Duran, R., Zavgorodniaia, A., & Sorva, J. (2022). "Cognitive Load Theory in Computing Education Research: A Review." ACM Transactions on Computing Education, 22(4), Article 40.

• Parnas, D.L. (1972). "On the Criteria To Be Used in Decomposing Systems into Modules." Communications of the ACM, 15(12), 1053–1058.

• Peitek, N., Apel, S., Parnin, C., Brechmann, A., & Siegmund, J. (2021). "Program Comprehension and Code Complexity Metrics: An fMRI Study." ICSE 2021. ACM SIGSOFT Distinguished Paper Award.

• Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590.

• Shen, J.H. & Tamkin, A. (2026). "How AI Impacts Skill Formation." arXiv:2601.20245.

• Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A.E., & Li, S. (2018). "Measuring Program Comprehension: A Large-Scale Field Study with Professionals." IEEE Transactions on Software Engineering, 44(10), 951–976.

• METR. (2025). "Measuring the Impact of Early 2025 AI on Experienced Open Source Developer Productivity." metr.org.

Talks and Blog Posts

• Hickey, R. (2011). "Simple Made Easy." Strange Loop Conference.

• Beck, K. (2023). "90% of My Skills Are Now Worth $0." Tidy First? Substack.

• Osmani, A. (2026). "Comprehension Debt: The Hidden Cost of AI-Generated Code." addyosmani.com.

• Muratori, C. (2023). "Clean Code, Horrible Performance." Computer Enhance.

• Abramov, D. (2020). "Goodbye, Clean Code." overreacted.io.

• Metz, S. (2016). "The Wrong Abstraction." sandimetz.com.

• GitClear. (2025). "AI Assistant Code Quality in 2025." gitclear.com.

An API fails in production. You restart the service, errors go away, and it feels resolved. Until it happens again. And again. What's happening here is simple: you're treating symptoms, not the underlying cause.

The 5 Whys technique is a straightforward way to deal with this. It comes from the Toyota Production System and was designed to help teams dig deeper into problems instead of settling for quick fixes.

The idea is simple. Ask "why" repeatedly until you reach the real cause.

But in practice, this is where things go wrong.

Teams often:

• Stop too early

• Assume answers without checking data

• Focus on people instead of systems

• Treat "five" as a rule instead of a guideline

So even though the process looks structured, the outcome is still shallow.

In this article, we'll focus on how to actually use the 5 Whys in real situations. Not just the theory, but what it looks like when you apply it to an engineering problem.

Here's What We'll Cover:

• What is the 5 Whys Technique?

• Origins of the 5 Whys Method

• How to Conduct an Effective 5 Whys Analysis

• Real-World Example: Applying 5 Whys in an Engineering Scenario

• When to Use (and When Not to Use) 5 Whys

• Benefits of the 5 Whys Technique

• Common Pitfalls and Limitations

• Tips for Using 5 Whys Effectively

• Summary

What is the 5 Whys Technique?

The 5 Whys technique is a way to break down a problem by repeatedly asking why it happened, with the goal of reaching a cause that actually explains the issue and can be addressed.

At its core, it's not about the number five. The name can be misleading. What matters is the process of following a chain of cause and effect until the explanation stops being superficial and starts becoming useful.

Each answer you uncover should move you one level deeper. You start with what went wrong, then explore what led to it, and continue until you reach something that is both believable and actionable. In most real situations, that final answer is not a single event but a gap in a system, a missing check, or an assumption that was never validated.

The technique became widely known through the Toyota Production System, where it was used to improve processes by focusing on causes rather than quick fixes.

That context is important because it highlights the original intent. The goal was not just to explain problems, but to prevent them from happening again.

A simple example makes this clearer. Imagine a mobile app suddenly starts crashing after a release. Asking "Why?" might look like this:

1. Why is the app crashing? → Because a null value is being accessed in the code.

2. Why is there a null value? → Because the API response is missing a required field

3. Why is the field missing? → Because a recent backend change made the field optional.

4. Why was this change not handled in the app? → Because the app assumes the field is always present.

5. Why was this assumption not caught earlier? → Because there are no contract tests validating API responses.

At this point, the issue is no longer just "fix the null check". The deeper problem is the lack of validation between systems, which allows breaking changes to slip through.

A useful way to think about the 5 Whys is that it forces you to stay with the problem a little longer than you normally would. Most of the time, the first explanation feels sufficient, so it's easy to stop there. This method pushes you to go one step further, and then another, until the explanation holds up under scrutiny.

At the same time, it's not a rigid formula. You might reach a solid root cause in three steps, or it might take more than five. The quality of the reasoning matters more than the count.

Origins of the 5 Whys Method

The 5 Whys method comes from the Toyota Production System, a manufacturing approach focused on continuous improvement and problem solving at the source.

It's often associated with Sakichi Toyoda, whose philosophy was simple: don’t just fix a problem. Understand why it happened so it doesn't happen again.

Inside Toyota, this wasn't treated as a formal tool or checklist. It was part of the day-to-day way of working. When something went wrong on the production line, the goal wasn't to get things running quickly and move on. The goal was to stop, investigate, and make sure the same issue wouldn't repeat.

That mindset is important to understand. The 5 Whys was never meant to be a rigid exercise where you ask five questions and stop. It was a way to encourage deeper thinking and accountability in processes.

Another key idea in the Toyota system is that problems are usually caused by processes, not people. Instead of asking "who made the mistake", the focus is on "what allowed this mistake to happen". The 5 Whys fits naturally into this approach because it pushes you toward system level causes rather than individual blame.

Over time, the method spread beyond manufacturing and is now used in software engineering, product teams, operations, and many other fields. The context has changed, but the core idea remains the same: if you don't understand the cause, you're likely to see the same problem again.

This origin story is useful not just as background, but as a reminder of intent. The value of the 5 Whys doesn't come from the questions themselves. It comes from the discipline of not settling for the first answer.

How to Conduct an Effective 5 Whys Analysis

A 5 Whys analysis works best when it is treated as a structured way of thinking, not a checklist to rush through. The quality of the outcome depends less on how many times you ask "why" and more on how carefully you reason through each step.

It helps to approach it in stages, each with a clear purpose.

Step 1: Define the Problem Clearly

Start with a problem statement that is specific and observable. Avoid vague descriptions like "the system is slow" or "things are failing". Instead, describe what actually happened in a way that can be verified.

For example, "API response time exceeded 5 seconds for 30 percent of requests between 2 PM and 3 PM" is much more useful than "API is slow".

A clear problem statement keeps the analysis grounded. If the starting point is fuzzy, the entire chain of reasoning will drift.

Step 2: Ask "Why" Iteratively

Once the problem is defined, begin asking why it happened. Each answer should directly address the question before it and naturally lead to the next one.

The key here is continuity. Every step should feel like a logical extension of the previous one. If you find yourself jumping topics or introducing unrelated explanations, it's a sign that the chain is breaking.

Keep going until the answers stop being immediate symptoms and start pointing toward underlying conditions or decisions.

Also, don't force the process to stop at five. Some problems may need fewer steps, while others may need more. What matters is reaching a point where the explanation is meaningful and actionable.

Step 3: Validate Each Answer with Evidence

This is where many analyses go wrong. It's easy to come up with plausible answers, but plausibility is not enough.

Each "why" should be backed by some form of evidence. This could be logs, metrics, recent changes, or direct observation. If an answer can't be verified, treat it as a hypothesis and confirm it before moving forward.

Without validation, the entire analysis becomes a chain of assumptions. Even if the final answer sounds reasonable, it may not reflect reality.

Step 4: Identify the Root Cause

A good root cause is one that explains the sequence of events and can be acted upon to prevent the issue in the future.

In many cases, this turns out to be a gap in a process rather than a single technical failure. It could be a missing validation step, an incomplete test, or an assumption that was never challenged.

If the final answer still feels like a symptom, you probably need to go one level deeper. On the other hand, if the answer points to something you can change in your system or workflow, you are likely in the right place.

Step 5: Define Corrective Actions

The analysis is only useful if it leads to meaningful action.

Once you've identified the root cause, the next step is to define changes that prevent the problem from happening again. These should go beyond quick fixes and address the underlying issue.

For example, instead of just fixing a bug, you might introduce better testing, add monitoring, or improve review processes.

Good corrective actions share a few traits: they're specific, practical to implement, and they directly address the root cause identified in the analysis.

Real-World Example: Applying 5 Whys in an Engineering Scenario

To see how this works in practice, let’s walk through a realistic backend issue. The goal here is not just to reach an answer, but to show how each step builds on evidence and leads to something actionable.

The Problem:

Users report intermittent failures while fetching order details:

GET /api/orders/{id}
→ HTTP 500 Internal Server Error

Application logs show:

// Java 21 example (Spring Boot style logging)
logger.error("Database connection timeout while fetching order", ex);

At this point, it's tempting to conclude that the database is the problem. But that's only what we can see on the surface.

Applying the 5 Whys

1. Why did the API return a 500 error?

Because the database query timed out.

This is directly supported by the error logs, so we can treat it as a confirmed fact.

2. Why did the query time out?

Because the database connection pool was exhausted.

Metrics show that all available connections were in use during peak traffic.

3. Why was the connection pool exhausted?

Because some requests were holding database connections for too long.

Slow query logs confirm that a subset of queries had unusually high execution times.

4. Why were some queries slow?

Because a recently introduced feature added a query on a non-indexed column.

Looking at recent deployments reveals a change that introduced filtering without proper indexing.

5. Why was an unoptimized query deployed to production?

Because there is no performance validation step in the development or release process.

There are no checks in code review or CI/CD to catch inefficient database queries before deployment.

Root Cause

The issue is not the timeout itself.

It's this:

The system allows inefficient database queries to reach production without any safeguards.

What a Shallow Fix Would Look Like

If we stopped early, we might:

• Increase the database timeout

• Increase the connection pool size

These might reduce the frequency of failures, but they don't solve the underlying problem.

What a Strong Fix Looks Like

A proper 5 Whys analysis leads to changes that improve the system:

• Add appropriate indexing for frequently queried fields

• Introduce query performance checks in CI/CD pipelines

• Add monitoring and alerts for slow queries

• Include database considerations in code reviews

Why This Example Matters

The difference between a shallow fix and a real solution is depth.

The first explanation often feels sufficient, especially under pressure. But stopping there means the issue is likely to return in a different form.

The value of the 5 Whys comes from following the chain all the way to something you can change in your system.

When to Use (and When Not to Use) 5 Whys

Like any problem-solving method, the 5 Whys is useful in the right context and less effective in others. Knowing when to apply it is just as important as knowing how to use it.

If used appropriately, it can uncover meaningful insights. If used in the wrong situation, it can lead to oversimplified or misleading conclusions

When to Use 5 Whys

The 5 Whys is most useful when your goal is to understand why something happened, not just to fix it and move on.

It works well in situations where problems are recurring or not fully explained by the first answer. For example, production incidents, repeated bugs, or issues that reappear after a quick fix are strong signals that you need deeper analysis. In these cases, the technique helps uncover what is happening beneath the surface.

It's also effective during retrospectives and postmortems. When a release doesn't go as expected or a sprint runs into issues, the 5 Whys helps teams move beyond observations like "this failed" and get to "why did this fail in the first place".

In general, use it when:

• The problem is not obvious

• The issue has occurred more than once

• You want to prevent recurrence, not just resolve the current instance

When Not to Use 5 Whys

The 5 Whys has its limits, and using it in the wrong context can lead to oversimplified conclusions.

If a problem involves multiple interacting factors, a single chain of "why" questions may not capture the full picture. Complex systems often have several contributing causes, and forcing them into one linear explanation can hide important details. In such cases, the 5 Whys should be combined with other approaches.

It's also less effective when there's not enough data. If each answer is based on assumptions rather than evidence, the analysis quickly becomes unreliable. The method depends on validation at every step.

Another limitation is in time-critical situations. During an active incident, the priority is to restore the system. The deeper analysis should happen later, once things are stable.

Finally, if your goal is quantitative analysis or optimization, the 5 Whys alone isn't enough. You'll need more data-driven methods to support decision making.

A simple rule of thumb is this. If you are trying to learn from a problem, use the 5 Whys. If you are trying to fix something immediately or analyze complex data, use it carefully or alongside other techniques.

Benefits of the 5 Whys Technique

The 5 Whys technique is simple, but it offers several powerful benefits that can help you solve problems more effectively and make lasting improvements. Here are the key advantages:

Simple and Easy to Apply

One of the biggest strengths of the 5 Whys is how easy it is to start using. You don't need special tools, training, or complex frameworks. It can be applied in a quick discussion, during debugging, or as part of a formal postmortem.

This low barrier makes it accessible across teams, regardless of experience level.

Encourages Deeper Thinking

The method naturally pushes you to go beyond the first explanation. Instead of reacting to what's visible, it encourages you to question why the problem occurred in the first place.

This shift from surface-level fixes to deeper understanding often leads to better decisions.

Promotes System-Level Improvements

When used correctly, the focus moves away from individual people and toward systems. Instead of asking who made a mistake, the analysis asks what allowed the mistake to happen.

This leads to improvements in processes, safeguards, and overall system design rather than one-off fixes.

Works Well in Team Settings

Because the approach is simple, it's easy for multiple people to contribute. Different perspectives help uncover gaps that might otherwise be missed.

It also creates a shared understanding of the problem, which is valuable during retrospectives and incident reviews.

Helps Prevent Recurring Issues

Quick fixes often solve the immediate problem but don't stop it from happening again. The 5 Whys helps identify underlying causes, which makes it easier to prevent similar issues in the future.

Over time, this leads to more stable systems and fewer repeated incidents.

Common Pitfalls and Limitations

While the 5 Whys technique is useful, it’s not always perfect. There are some limitations to keep in mind, so you can use it effectively and know when it might not be enough.

Stopping Too Early

One of the most common mistakes is ending the analysis after the first or second answer. These early answers usually describe symptoms, not causes.

Stopping too soon leads to fixes that address the surface but leave the underlying issue unresolved.

Treating Assumptions as Facts

It's easy to come up with explanations that sound reasonable. But without evidence, they're just assumptions.

If each step isn't validated with logs, metrics, or observations, the entire analysis can drift away from reality.

Focusing on Individuals Instead of Systems

Answers like "someone made a mistake" don't add much value. While they may be true, they don't explain why the system allowed that mistake to have an impact.

Focusing on processes and safeguards leads to more meaningful improvements.

Oversimplifying Complex Problems

The 5 Whys follows a linear chain of reasoning, but real-world systems often have multiple contributing factors.

Relying on a single chain can hide important interactions. In such cases, the method should be combined with other approaches.

Treating It as a Rigid Formula

The name suggests asking "why" five times, but this shouldn't be taken literally. Some problems require fewer steps, while others need more.

Forcing the structure can lead to artificial or weak conclusions.

Not a Replacement for Deeper Analysis

The 5 Whys isn't designed for every type of problem. For complex system failures, performance optimization, or data-heavy investigations, additional tools and methods are often required.

It works best as a starting point or a complement to other techniques, not a complete solution on its own.

Tips for Using 5 Whys Effectively

To get the most out of the 5 Whys technique, there are a few tips that can help you use it effectively. These will guide you to ask the right questions and reach useful, actionable insights.

Start with a Clear, Specific Problem

A vague problem leads to vague answers. Spend a little extra time making sure the problem statement is precise and based on observable facts. This keeps the analysis grounded and avoids unnecessary detours.

Base Every Step on Evidence

Treat each answer as something that needs to be verified. Use logs, metrics, recent changes, or direct observations to support your reasoning. If something can't be validated, call it out as a hypothesis and confirm it before moving forward.

Keep the Chain Logical and Connected

Each "why" should naturally follow from the previous answer. If the reasoning starts to jump between unrelated ideas, pause and re-evaluate. A clean, logical chain is a strong indicator that you're on the right track.

Focus on Systems, Not Individuals

Avoid stopping at explanations that point to human error. Instead, ask what allowed that error to have an impact. This shift in thinking leads to improvements that actually reduce the chances of similar issues in the future.

Do Not Force Exactly Five Steps

The number five is a guideline, not a rule. Some problems become clear in three steps, while others need more exploration. Stop when you reach a cause that's both convincing and actionable.

Involve the Right People

If possible, do the analysis as a group. People from different parts of the system bring different perspectives, which helps uncover details that might otherwise be missed. It also creates shared ownership of both the problem and the solution.

Turn Insights into Actions

The analysis only matters if it leads to change. Make sure the final outcome includes clear, practical steps that address the root cause. Without this, even a well-done analysis has limited impact.

Summary

The 5 Whys is a simple technique, but using it well takes some discipline.

At its core, it's about resisting the urge to stop at the first explanation. By following the chain of cause and effect, you move from symptoms to something you can actually fix. In many cases, that turns out to be a gap in a process rather than a one-off failure.

When applied thoughtfully, it helps teams learn from problems instead of just reacting to them. Over time, this leads to better systems, fewer recurring issues, and more confidence in how problems are handled.

The key is to treat it as a way of thinking, not just a set of steps.

I have known for a while now that most learners today don't share my love of big thick books, and to be honest, I rarely read those books in their entirety. Those big books often had a lot more exposition about the code than was probably needed. As a book buyer I wanted to make sure that I was getting my money's worth so the thicker they were, the better. It is much more common these days for learners to consume blog based tutorials and videos.

If you're learning to code right now, you've probably experienced the frustration of these formats too. I want to share something I've been working on that might help.

Blogs and Videos

Blog-style tutorials mix code and the explanation of it in a top-to-bottom fashion. Scrolling through these web-based explanations feels familiar and one can copy and paste with ease. However, linking the explanation of the code and the code itself has always been less than ideal. Often I find myself jumping around the blog post wishing I could see the entire code example while working through the explanation. Instead, I am only able to see small parts of the code and it is challenging to see how those parts relate to other parts.

Video tutorials are very popular these days. They solve some of the problems associated with blog-style tutorials. Videos are great because you get two streams of information: the author's audible narrative and the code being written. A viewer can focus on the two streams simultaneously. However, videos have some problems too.

Viewing Videos

From the perspective of the viewer, videos are hard to search through and are not useful as a copy and paste source or a code reference. More importantly, though, they discourage the viewer from taking their time and reflecting on the material. Often, when I am viewing a video tutorial I don't pause and let concepts sink in before the video moves on. Yes, I could be more disciplined and pause and rewind more often but usually I don't.

Making videos

From the perspective of the video creator, it is clear that not all code being developed is interesting to watch. Some of it is not really worth showing the viewer. Not all video creators can keep the narrative interesting the whole time.

I know I struggle with the 'performance' aspect of making videos (you won't find me coding on Twitch anytime soon). Many times after I am done making a video, as I review it, I wish I had mentioned something that I forgot. It is hard to go back and edit the video without scrapping it and starting over.

Storyteller

I have created a new medium to guide viewers through code examples. It combines the best of books, blog posts, and videos. This new medium allows a developer to write code using a top-notch editor (Visual Studio Code) and then replay the development of that code in the browser.

The author can add comments at important points in the evolution of the code. The comments can include text, hand drawn pictures, screenshots, and audio and video recordings. This allows the author to add visualizations that we have in our heads but don't make it into the code itself. The tool is called Storyteller.

Here are a few examples of a 'playback':

• Enlarging a Picture (Python)

• Dynamic Variables and Pointers (C++)

These work best on a big screen. If you are viewing a playback on a small screen you can view it in 'blog' mode (there is button in the top right to switch from 'code' mode to 'blog' mode).

I have created groups of these guided code walk-throughs to help me teach different topics to my students. These are all free and hosted on a website I created called Playback Press. Here are some of the 'books' I have created so far:

• An Animated Introduction to Programming in C++

• An Animated Introduction to Programming with Python

• An Introduction to Web Development from Back to Front

• An Animated Introduction to Clojure

• An Animated Introduction to Elixir

• Database Design and SQL for Beginners

• Mobile App Development with Dart and Flutter

• OO Design Patterns with Java

I usually assign these as readings in my classes instead of using expensive textbooks. It is a lot easier for me to write several programs than it is to find a perfect textbook.

I also use them for in-class demos instead of writing code live. This makes code demos flow much faster and smoother. If I make an interesting mistake while preparing the code I can still highlight it with a comment. If I make an uninteresting or embarrassing mistake I can just ignore it and the students won't focus on it.

The Advantages of Code Playbacks:

• The primary focus is on the code. It is always visible and easy to search and navigate.

• Since the code is so accessible, the explanation of it tends to be short and concise.

• The narrative can include whiteboard style drawings, screenshots, or videos of running code in addition to a text explanation.

• As an author, I can review the code several times and add/edit comments each time I go through it. I don't have to give a perfect performance like I do with a video.

• Comment points highlight when the author wants the viewer to take a moment to really think about the code and reflect on it. The playback only moves forward when the viewer is ready.

• The code mentioned in a comment can be highlighted so the viewer knows exactly where they should be looking.

• The code can be downloaded at any point in the playback. Then a viewer can run it, change it, and add to it.

• The tool is a language independent editor plug-in and can be used to describe programs in any language.

• Viewers only need a web browser to go through a playback.

Recently, I've been exploring how to make playbacks even more useful for learners.

AI as an Infinitely Patient Tutor

I have extended code playbacks to include an AI tutor. One thing I've learned in my years of teaching is that students often hesitate to ask questions. They worry about looking foolish, or they don't want to slow down the class, or they simply can't articulate what's confusing them.

What if every student had access to a patient tutor who never got frustrated with repeated questions and could explain concepts in multiple ways until something clicked?

I've integrated AI directly into the playback experience. As students work through a playback, they can ask questions about anything they're seeing. This might be a specific line of code, a concept I mentioned in a comment, or how something connects to material from earlier in the playback. The AI has full context. It can see the code, it understands where the student is in the playback, and it can provide explanations tailored to that exact moment. The AI is right there with the student, looking at the same code, understanding the same context.

The AI can also generate self-grading multiple choice questions based on the code and comments in a playback. These low-stakes quizzes make the learning experience more engaging and help learners check their understanding as they go.

Let me be clear: the AI doesn't replace me as an instructor. I still create the playbacks. I still decide what concepts to cover, what order to present them, and what examples best illustrate the ideas. The AI is an extension of my teaching, not a replacement for it.

Note: The AI features are available to registered users on Playback Press. Registration is free but logging in is required to access the AI tutor. If you want to see what this feels like, try one of the playbacks linked above and ask the AI a question about what you're seeing.

Conclusion

My goal has always been to help people learn to code. Books gave us depth but demanded commitment. Blogs gave us accessibility but fragmented the code. Videos gave us narrative but took away control. Playbacks keep the code front and center while letting learners move at their own pace and reflect when they need to. Adding AI doesn't change that philosophy, it just means there's always someone available to answer questions. Together, they get closer to the experience of having an expert sit beside you and walk you through a program. That's what I've been trying to build, and I think we're getting there.

The Singleton Design Pattern is a creational design pattern that solves this problem by ensuring that a class has exactly one instance and provides a global point of access to it.

This pattern is widely used in mobile apps, backend systems, and Flutter applications for managing shared resources such as:

• Database connections

• API clients

• Logging services

• Application configuration

• Security checks during app bootstrap

In this article, we'll explore what the Singleton pattern is, how to implement it in Flutter/Dart, its variations (eager, lazy, and factory), and physical examples. By the end, you'll understand the proper way to use this pattern effectively and avoid common pitfalls.

Table of Contents

1. Prerequisites

2. What is the Singleton Pattern?

   • When to Use the Singleton Pattern

3. How to Create a Singleton Class

   • Eager Singleton

   • Lazy Singleton

   • Choosing Between Eager and Lazy

4. Factory Constructors in the Singleton Pattern

   • What Are Factory Constructors?

   • Implementing Singleton with Factory Constructor

5. When Not to Use a Singleton

   • Why Singletons Can Be Problematic

   • Scenarios Where You Should Avoid Singletons

   • General Guidelines

6. Conclusion

Prerequisites

Before diving into this tutorial, you should have:

1. Basic understanding of the Dart programming language

2. Familiarity with Object-Oriented Programming (OOP) concepts, particularly classes and constructors

3. Basic knowledge of Flutter development (helpful but not required)

4. Understanding of static variables and methods in Dart

5. Familiarity with the concept of class instantiation

What is the Singleton Pattern?

The Singleton pattern is a creational design pattern that ensures a class has only one instance and that there is a global point of access to the instance.

Again, this is especially powerful when managing shared resources across an application.

When to Use the Singleton Pattern

You should use a Singleton when you are designing parts of your system that must exist once, such as:

1. Global app state (user session, auth token, app config)

2. Shared services (logger, API client, database connection)

3. Resource heavy logic (encryption handlers, ML models, cache manager)

4. Application boot security (run platform-specific root/jailbreak checks)

For example, in a Flutter app, Android may check developer mode or root status, while iOS checks jailbroken device state. A Singleton security class is a perfect way to enforce that these checks run once globally during app startup.

How to Create a Singleton Class

We have two major ways of creating a singleton class:

1. Eager Instantiation

2. Lazy Instantiation

Eager Singleton

This is where the Singleton is created at load time, whether it's used or not.

In this case, the instance of the singleton class as well as any initialization logic runs at load time, regardless of when this class is actually needed or used. Here's how it works:

class EagerSingleton {
  EagerSingleton._internal();
  static final EagerSingleton _instance = EagerSingleton._internal();

  static EagerSingleton get instance => _instance;

  void sayHello() => print("Hello from Eager Singleton");
}

//usage
void main() {
  // Accessing the singleton globally
  EagerSingleton.instance.sayHello();
}

How the Eager Singleton Works

Let's break down what's happening in this implementation:

First, EagerSingleton._internal() is a private named constructor (notice the underscore prefix). This prevents external code from creating new instances using EagerSingleton(). The only way to get an instance is through the controlled mechanism we're about to define.

Next, static final EagerSingleton _instance = EagerSingleton._internal(); is the key line. This creates the single instance immediately when the class is first loaded into memory. Because it's static final, it belongs to the class itself (not any particular instance) and can only be assigned once. The instance is created right here, at declaration time.

The static EagerSingleton get instance => _instance; getter provides global access to that single instance. Whenever you call EagerSingleton.instance anywhere in your code, you're getting the exact same object that was created when the class loaded.

Finally, sayHello() is just a regular method to demonstrate that the singleton works. You could replace this with any business logic your singleton needs to perform.

When you run the code in main(), the class loads, the instance is created immediately, and EagerSingleton.instance.sayHello() accesses that pre-created instance to call the method.

Pros:

1. This is simple and thread safe, meaning it's not affected by concurrency, especially when your app runs on multithreads.

2. It's ideal if the instance is lightweight and may be accessed frequently.

Cons:

1. If this instance is never used through the runtime, it results in wasted memory and could impact application performance.

Lazy Singleton

In this case, the singleton instance is only created when the class is called or needed in runtime. Here, a trigger needs to happen before the instance is created. Let's see an example:

class LazySingleton {
  LazySingleton._internal(); 
  static LazySingleton? _instance;

  static LazySingleton get instance {
    _instance ??= LazySingleton._internal();
    return _instance!;
  }

  void sayHello() => print("Hello from LazySingleton");
}

//usage 
void main() {
  // Accessing the singleton globally
  LazySingleton.instance.sayHello();
}

How the Lazy Singleton Works

The lazy implementation differs from eager in one crucial way: timing.

Again, LazySingleton._internal() is a private constructor that prevents external instantiation.

But notice that static LazySingleton? _instance; is declared as nullable and not initialized. Unlike the eager version, no instance is created at load time. The variable simply exists as null until it's needed.

The magic happens in the getter: _instance ??= LazySingleton._internal(); uses Dart's null-aware assignment operator. This line says "if _instance is null, create a new instance and assign it. Otherwise, keep the existing one." This is the lazy initialization: the instance is only created the first time someone accesses it.

The first time you call LazySingleton.instance, _instance is null, so a new instance is created. Every subsequent call finds that _instance already exists, so it just returns that same instance.

The return _instance!; uses the null assertion operator because we know _instance will never be null at this point (we just ensured it's not null in the previous line).

This approach saves memory because if you never call LazySingleton.instance in your app, the instance never gets created.

Pros:

1. Saves application memory, as it only creates what is needed in runtime.

2. Avoids memory leaks.

3. Is ideal for resource heavy objects while considering application performance.

Cons:

1. Could be difficult to manage in multithreaded environments, as you have to ensure thread safety while following this pattern.

Choosing Between Eager and Lazy

Now that we've broken down these two major types of singleton instantiation, it's worthy of note that you'll need to be intentional while deciding whether to create a singleton the eager or lazy way. Your use case/context should help you determine what singleton pattern you need to apply during object creation.

As an engineer, you need to ask yourself these questions when using a singleton for object creation:

1. Do I need this class instantiated when the app loads?

2. Based on the user journey, will this class always be needed during every session?

3. Can a user journey be completed without needing to call any logic in this class?

These three questions will determine what pattern (eager or lazy) you should use to fulfill best practices while maintaining scalability and high performance in your application.

Factory Constructors in the Singleton Pattern

Applying factory constructors in the Singleton pattern can be powerful if you use them properly. But first, let's understand what factory constructors are.

What Are Factory Constructors?

A factory constructor in Dart is a special type of constructor that doesn't always create a new instance of its class. Unlike regular constructors that must return a new instance, factory constructors can:

1. Return an existing instance (perfect for singletons)

2. Return a subclass instance

3. Apply logic before deciding what to return

4. Perform validation or initialization before returning an object

The factory keyword tells Dart that this constructor has the flexibility to return any instance of the class (or its subtypes), not necessarily a fresh one.

Implementing Singleton with Factory Constructor

This allows you to apply initialization logic while your class instance is being created before returning the instance.

class FactoryLazySingleton {
  FactoryLazySingleton._internal();
  static final FactoryLazySingleton _instance = FactoryLazySingleton._internal();

  static FactoryLazySingleton get instance => _instance;

  factory FactoryLazySingleton() {
    // Your logic runs here
    print("Factory constructor called");
    return _instance;
  }
}

How the Factory Constructor Singleton Works

This implementation combines aspects of both eager and lazy patterns with additional control.

The FactoryLazySingleton._internal() private constructor and static final _instance create an eager singleton. The instance is created immediately when the class loads.

The static get instance provides the traditional singleton access pattern we've seen before.

But the interesting part is the factory FactoryLazySingleton() constructor. This is a public constructor that looks like a normal constructor call, but behaves differently. When you call FactoryLazySingleton(), instead of creating a new instance, it runs whatever logic you've placed inside (in this case, a print statement), then returns the existing _instance.

This pattern is powerful because:

1. You can log when someone tries to create an instance

2. You can validate conditions before returning the instance

3. You can apply configuration based on parameters passed to the factory

4. You can choose to return different singleton instances based on conditions

For example, you might have different configuration singletons for development vs production:

factory FactoryLazySingleton({bool isProduction = false}) {
  if (isProduction) {
    // Apply production configuration
    _instance.configure(productionSettings);
  } else {
    // Apply development configuration
    _instance.configure(devSettings);
  }
  return _instance;
}

Pros

1. You can add logic before returning an instance

2. You can cache or reuse the same object

3. You can dynamically return a subtype if needed

4. You avoid unnecessary instantiation

5. You can inject configuration or environment logic

Cons

1. Adds slight complexity compared to simple getter access

2. The factory constructor syntax might confuse developers unfamiliar with the pattern

3. If overused with complex logic, it can make debugging harder

4. Can create misleading code where FactoryLazySingleton() looks like it creates a new instance but doesn't

When Not to Use a Singleton

While singletons are powerful, they're not always the right solution. Understanding when to avoid them is just as important as knowing when to use them.

Why Singletons Can Be Problematic

Singletons create global state, which can make your application harder to reason about and test. They introduce tight coupling between components that shouldn't necessarily know about each other, and they can make it difficult to isolate components for unit testing.

Scenarios Where You Should Avoid Singletons

Avoid using the Singleton pattern if:

You need multiple independent instances

If different parts of your app need their own separate configurations or states, singletons force you into a one-size-fits-all approach.

For example, if you're building a multi-tenant application where each tenant needs isolated data, a singleton would cause data to bleed between tenants.

Alternative: Use dependency injection to pass different instances to different parts of your app. Each component receives the specific instance it needs through its constructor or a service locator.

// Instead of singleton
class UserRepository {
  final DatabaseConnection db;
  UserRepository(this.db); 
}

// Usage
final dbForTenantA = DatabaseConnection(tenantId: 'A');
final dbForTenantB = DatabaseConnection(tenantId: 'B');
final repoA = UserRepository(dbForTenantA);
final repoB = UserRepository(dbForTenantB);

Your architecture avoids shared global state

Modern architectural patterns like BLoC, Provider, or Riverpod in Flutter specifically aim to avoid global mutable state. Singletons work against these patterns by reintroducing global state.

Alternative: Use state management solutions designed for Flutter. Provider, Riverpod, BLoC, or GetX offer better ways to share data across your app while maintaining testability and avoiding tight coupling.

// Using Provider instead of singleton
class AppConfig {
  final String apiUrl;
  AppConfig(this.apiUrl);
}

// Provide it at the top level
void main() {
  runApp(
    Provider<AppConfig>(
      create: (_) => AppConfig('https://api.example.com'),
      child: MyApp(),
    ),
  );
}

// Access it anywhere in the widget tree
class MyWidget extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final config = Provider.of<AppConfig>(context);

  }
}

It forces tight coupling between unrelated classes

When multiple unrelated classes depend on the same singleton, they become indirectly coupled. Changes to the singleton affect all these classes, making the codebase fragile and hard to refactor.

Alternative: Use interfaces and dependency injection. Define what behavior you need through an interface, then inject implementations. This way, classes depend on abstractions, not concrete singletons.

// Define an interface
abstract class Logger {
  void log(String message);
}

// Implementation
class ConsoleLogger implements Logger {
  @override
  void log(String message) => print(message);
}

// Classes depend on the interface, not a singleton
class PaymentService {
  final Logger logger;
  PaymentService(this.logger);

  void processPayment() {
    logger.log('Processing payment');
  }
}

// Easy to test with mock
class MockLogger implements Logger {
  List<String> logs = [];
  @override
  void log(String message) => logs.add(message);
}

You need clean, isolated testing

Singletons maintain state between tests, causing test pollution where one test affects another. This makes tests unreliable and order-dependent.

Alternative: Use dependency injection and create fresh instances for each test. Most testing frameworks support this pattern, allowing you to inject mocks or fakes easily.

// Testable code
class OrderService {
  final PaymentProcessor processor;
  OrderService(this.processor);
}

// In tests
void main() {
  test('processes order successfully', () {
    final mockProcessor = MockPaymentProcessor();
    final service = OrderService(mockProcessor); 

  });
}

General Guidelines

Use singletons sparingly and only when you truly need exactly one instance of something for the entire application lifecycle. Good candidates include logging systems, application-level configuration, and hardware interface managers.

For most other cases, prefer dependency injection, state management solutions, or simply passing instances where needed. These approaches make your code more flexible, testable, and maintainable.

Conclusion

The Singleton pattern is a powerful creational tool, but like every tool, you should use it strategically.

Overusing singletons can make apps tightly coupled, hard to test, and less maintainable.

But when used correctly, the Singleton pattern helps you save memory, enforce consistency, and control object lifecycle beautifully.

The key is understanding your specific use case and choosing the right implementation approach – whether eager, lazy, or factory-based – that best serves your application's needs while maintaining clean, testable code.

Bragging on social media about a clever script is one thing, but pushing a vibe coded app to prod comes with many security risks.

With so many AI dev tools out there now, code reviews become more critical than ever.

This article will explore what vibe coding means and how code reviews should adapt in the era of AI.

Table of Contents:

1. What is Vibe Coding?

2. How to Implement Vibe Coding in Practice

3. Why isn’t Vibe Coded Output Production Ready?

   • Context gaps are the first crack.

   • Those gaps lead directly to integration blind spots.

   • The most serious risk is security by omission.

   • Testing and correctness evidence are thin.

   • Operability lags behind.

4. Guidelines for AI Code Reviews

   • Code Review Process in Vibe Coding

5. Checklist for Reviewing AI Generated Code

6. How to Work Effectively with AI Tools

7. Conclusion

What is Vibe Coding?

In early 2025, AI researcher Andrej Karpathy popularized the term vibe coding to describe a new way of development in which you “fully give in to the vibes” and let AI write code while you focus on high level intent.

A developer expresses their desired functionality in plain language, and an AI system (like an LLM) generates the source code to implement it.

This code-by-prompt approach allows even beginners to produce working code without deep knowledge of programming languages. Karpathy joked that with advanced IDE agents (like Cursor’s Composer mode), “I barely even touch the keyboard... I ‘Accept All’ always, I don’t read the diffs anymore... and it mostly works”.

So, vibe coding is coding by vibe and trusting AI to handle the heavy lifting.

How to Implement Vibe Coding in Practice

In practice, vibe coding usually involves using AI assistants and adapting your workflow to a more interactive, prompt-driven style.

Here’s an overview of how you can “vibe code” a project:

Step 1: Choose an AI assistant

Select a development env that supports AI code generation. Popular choices include Cursor and GitHub Copilot.

Step 2: Define your requirements

Instead of writing boilerplate code, describe what you want to build. Provide AI with a specific prompt detailing functionality. The more context and detail you give, the better AI can fulfill your intent.

For example, when I ran an SEO inspection for my website, DevTools Academy, I used this prompt in Cursor:

“Now, act as a senior product engineer and UX strategist. Evaluate and improve https://www.devtoolsacademy.com with a practical, no-fluff lens.

Scope:

• UX

• SEO and technical SEO

• Positioning and messaging

• Copywriting and information architecture

• What to add to stand out in the developer tools space.”

This prompt works well because it gives the AI a clear role, a defined scope, and a specific intent. AI knows it’s not just fixing SEO but also reviewing how the site communicates value to devs. That combination of clarity and context produces actionable insights instead of surface-level suggestions.

Below is a screenshot of that audit in progress and showing how I reviewed code, metadata, and UX recommendations side by side.

You can checkout the full code on my open source blog here and check out closed PRs. This will help you learn how I use all these coding agents on a production ready app.

Step 3: Review the code

AI will produce initial code based on your prompt. Think of this as a prototype – it’s not perfect. Run the code and see how it behaves.

Let’s look at an example: here, CodeRabbit is reviewing one of my pull requests on GitHub. I had pushed a small fix to sort blog posts correctly and make sure the RSS feed reflects the latest publish date. Within seconds, CodeRabbit analyzed the diff, understood the intent behind my change, and explained exactly what the new code does.

It pointed out that the fix now sorts posts before mapping them, uses the sorted data for both items and the lastBuildDate, and ensures proper chronological order throughout the feed.

It’s like having a senior reviewer who not only checks syntax but also validates logic and confirms that your reasoning holds up.

This is just a reminder to expect imperfections. Vibe coding embraces a “code first, refine later” mindset. This means you get a working version quickly, then iteratively improve it. You might go through a few cycles of prompt -> code -> test -> tweak.

Step 4: Validate, debug, polish

Once AI generated code meets your expectations, do a final review.

Throughout the process, the core idea is that you collaborate with the AI. The AI agent serves as a coding assistant, making real-time suggestions, automating tedious boilerplate, and even generating entire modules on your behalf.

Why Isn’t Vibe Coded Output Production Ready?

Vibe coding moves fast: you describe intent, the AI produces something that runs, and you’re off to the next prompt. What’s missing is the slow, unglamorous work that usually turns a draft into shippable software, like shared context, architectural alignment, verification, and documentation.

AI generates plausible code based on patterns it has seen. But it doesn’t understand your team’s history, your system’s constraints, or the implicit rules that keep everything coherent over time.

That mismatch shows up the moment a “works on my machine” demo meets a real codebase.

Let’s explore the common pitfalls of vibe-coded code, so you’ll know what to watch for. Then, in the checklist section below, I’ll outline practical strategies to address or prevent each issue.

Context gaps are the first crack.

AI only sees what you show it, so it’s easy for it to make the right local choice and the wrong global one: duplicating logic that already exists, choosing defaults that conflict with prior decisions, or introducing functions that don’t respect domain boundaries.

The result is code that looks reasonable in isolation but collides with existing assumptions and conventions once integrated.

Those gaps lead directly to integration blind spots.

Drafts often ignore the lived details of your environment – shared utilities, cross-cutting concerns, configuration, deployment hooks, and operational policies. Interfaces may line up at a glance and still fail at runtime because the draft doesn’t fit how your system composes modules, handles errors, or manages state across services.

The most serious risk is security by omission.

AI rarely includes robust input validation, clear authentication and authorization paths, or rate limiting unless you spell it out. Secrets handling and logging tend to be superficial or missing. That leaves common exposure points like request handlers, job processors, and webhook endpoints without the checks that prevent injection, SSRF, mass assignment, or data exfiltration.

Even when the surface looks tidy, the absence of explicit security controls means you’re trusting defaults you didn’t choose.

Testing and correctness evidence are thin.

Quality suffers in quieter ways, too. Beyond “it runs,” there’s little to demonstrate behavior across edge cases or to guard against regressions.

Performance and scalability remain unknowns: extra network calls, N+1 patterns, and quadratic loops sneak in because nobody measured them. Dependencies and environments drift as versions aren’t pinned, infrastructure isn’t declared, and configuration lives only in the author’s head, making behavior differ across machines and CI.

Operability lags behind.

A lack of metrics, missing health/readiness probes, and no runbook make failures harder to detect and slower to recover from. Add in data quality and compliance concerns (PII handling, encoding assumptions, transitive license obligations), and you have code that demos well but isn’t ready for production’s reliability, security, and audit demands.

In short, vibe-coded output accelerates drafting but skips the shared understanding and evidence that make software safe to ship.

Until those gaps are closed, it’s a prototype, not a release.

Guidelines for AI Code Reviews

Your team should keep pre-AI engineering standards as the bar, including security, tests, readability, maintainability, performance, and docs. AI should change how fast you gather the evidence for those standards, not how much evidence you require. In other words, use AI to accelerate the path to your existing bar, never to lower it.

Using AI, you can generate code at speed. But if reviews take the same amount of time (or more time), you lose some of the benefit. The goal isn’t to relax standards, it’s to shorten the time to prove you met them. That means layering in automation (tests, static analysis, secret scans, SCA) and AI-assisted review to catch obvious issues quickly so human reviewers can focus on intent, architecture, and risk.

Well-used assistants can help here. For example, tools like CodeRabbit, GitHub Copilot PR Reviewer, Claude Code, Cursor’s Bugbot, Graphite’s AI Review, and Greptile can highlight potential bugs, security gaps, style deviations, and mismatched intent, and summarize diffs for faster context. Treat these as accelerators for your existing process, not as replacements for judgment.

Code Review Process in Vibe Coding

The fundamentals of good code reviews haven’t changed – and in fact, they’re more critical now.

Below are some key principles to maintain speed without sacrificing quality.

1. Trust, but verify.

A reviewer usually assumes the author understands the system. With vibe-coded output, the “author” may be an AI with limited context. If something looks odd or unnecessary, question it. Run the code, add/execute tests, or ask the developer/AI for clarification on intent and constraints.

2. Don’t let reviews become a bottleneck.

Vibe coding generates code quickly. If human review takes as long as hand-writing the change, you’ve erased the gain.

Combat this by front-loading automation: run unit/integration tests, static analysis (lint/SAST), secret scans, SCA, and basic perf checks in CI to clear the noise. Then reviewers spend their time on design trade-offs, boundary cases, and risk. The balance is: high standards, faster evidence.

3. Use AI code reviews wisely

AI can help review code just as it helps generate it. Modern “pair reviewer” tools scan a PR and surface likely bugs, security issues, missing tests, or style violations in minutes plus give natural-language summaries of the change.

Tools you can consider include CodeRabbit, GitHub Copilot PR Reviewer, Claude Code, Cursor Bugbot, Graphite, and Greptile. Many integrate with the CLI/IDE and GitHub/GitLab to leave actionable comments.

Think of them as fast first-pass reviewers that increase coverage and consistency across PRs.

4. Human judgment is still irreplaceable.

Even the best AI reviewer is an assistant. Keep humans accountable for correctness, security posture, architectural fit, and user impact. A healthy pattern is AI first-pass > human second-pass that inspects invariants, failure modes, and long-term maintainability.

5. Maintain a high bar for quality.

It’s tempting to accept “it runs” when an AI wrote it. Don’t. Stakeholders still expect software to be robust, secure, and maintainable. Keep DRY, readability, and testability standards. Insist on input validation, authZ checks where relevant, and sensible logging/metrics. If you can’t provide evidence that you met the bar, you haven’t met it.

6. Educate and document

When reviewers find bugs or security flaws in AI-generated code, capture the lesson.

Update internal guides with patterns like “When generating handlers, validate and bound inputs, add rate limits, log request IDs, avoid N+1 queries, and sanitize user-visible output.” Over time, bake these into prompts, templates, repo scaffolds, and CI checks so the next AI draft starts closer to done.

Checklist for Reviewing AI Generated Code

Before approving any vibe-coded change, make the standards explicit and verifiable. Use this checklist to confirm behavior, security, performance, integration, and documentation so the draft you got from AI becomes code you can safely ship.

Here’s a checklist a human reviewer should go through before approving vibe-coded output:

1. Define the code’s purpose (scope & non-goals).

Be explicit about what this change does and does not do. Tie it to a user story/ticket and call out non-goals so “helpful” AI changes don’t creep in.

2. Verify X and Y (behavior and edge cases).

Be clear about what you’re verifying. For example, verify input parsing and pagination boundaries, verify that error paths return the correct status and body, and verify that database writes are idempotent. Run existing tests, add missing unit/integration tests, and reproduce edge inputs (empty, null, huge, unicode).

3. Perform code-quality checks (readability, DRY, refactor needs).

AI often produces verbose or duplicated logic. Ensure names are meaningful, side effects are clearly stated, and duplication is removed or minimized. Run linters/formatters, collapse repetition, and extract helpers where they aid clarity.

4. Analyze organization and structure (make sure it fits the architecture).

AI writes code in isolation. Confirm the change uses existing utilities, layers, and boundaries (domain/services/controllers/jobs). Check imports and module placement, avoid reinventing existing helpers, and align with repository conventions.

5. Validate inputs and assumptions (make the implicit explicit).

List the assumptions the AI made (default locale/timezone, allowed ranges, required fields). Add schema validation (DTO/class validators/JSON Schema). Empty, null, over-max, non-ASCII, unexpected enum, malicious strings. And finally, enforce limits/timeouts.

6. Perform security audits (minimum pass).

• AuthN/AuthZ: Confirm endpoint checks identity and authorization paths; deny-by-default.

• Inputs: Sanitize/validate inputs, prevent injection (SQL/NoSQL/command), and escape user-visible output.

• Secrets: No secrets in code/diff/logs, use env/secret manager, and rotate any test keys.

• Abuse controls: Add rate limits, size limits, and timeouts on network and disk operations. Run SAST/secret scan/SCA, and fix or justify findings.

7. Do a performance evaluation (right now, at a small scale).

Look for N+1s, needless network calls, unbounded loops, quadratic sorts. Add a micro-benchmark or run a quick load test for hot paths. Set sensible cache/timeout/retry with jitter where applicable.

8. Manage dependencies (pin, justify, minimize).

Review any new libraries. Are they necessary? Maintained? License compatible? Pin versions, add lockfiles, or remove unused transitive adds.

9. Review documentation (what to add and where).

Ensure the docs are in line with the code. AI often changes some parts or adds code blocks at different places while resolving various issues. These changes might not make it into the docs.

10. Observability (see problems early).

Use structured logs with request/trace IDs, key counters/timers (success/error/latency), health/readiness probes, and a basic dashboard or alert stub.

11. Compliance and data handling (when applicable).

Identify any personally identifiable information (PII), document collection/retention, ensure masking/redaction in logs, verify dependency licenses and data-residency constraints.

How to Work Effectively with AI Tools

At this point, you can probably see why it’s very important to understand the actual skills involved in AI-assisted development.

There’s a pretty big difference between an experienced developer who uses AI tools to help them get more done, and a newbie who thinks AI can build the next Facebook or Google just with a simple prompt.

An inexperienced dev will ask AI something like "Hey, Build me Twitter and make no mistakes"

But an experienced developer who has a solid fundamentals might say say something like:

• "AI, we're building a Twitter replica. Use $SQL_Database, Use $Language, Avoid $Common_Pitfalls, Follow $Standard_Practices."

• "The generated code is prone to X problem, implement this fix."

• "Implementation of $X is flawed because of $Y, do $Z instead."

So as you can see, you still need to know the how's and the why's and what depends on what. Often you’ll just need to make the changes manually, because it will be faster. And you don’t want to outsource the critical thinking part, which is the part that AI can't actually do.

LLMs are good at information retrieval. If you know nothing about what you’re looking for, then asking an AI isn’t going to be that helpful (or that reliable). But if you have an idea, some background knowledge/context, and the skills to verify AI’s responses, then it can be really helpful.

Last month, I shared in my newsletter how my current coding loop looks in practice.

I draft with Claude Code (or Copilot/Cursor), open a PR, and let an AI reviewer like CodeRabbit (or Copilot PR Reviewer / Cursor Bugbot or Greptile) do the first pass. CI runs tests and scans.

I repeat until everything’s green and the PR is ready to merge. It’s fast, but it’s still disciplined.

If you want to understand why this kind of workflow is becoming essential, read this article: Era of AI Slop Cleanup Has Begun. I talk about what’s happening in AI-assisted engineering, where generating code is easy, but keeping it clean and production ready takes experience – and you must have good programming skills.

Conclusion

AI-generated code can boost productivity – but production value still comes from software that is robust, secure, and maintainable.

Mindless code generation creates technical debt. But when you integrate AI thoughtfully, with guardrails, verification, tests, security checks, and documentation, you can go faster without lowering your standards.

That's it for this article. I hope you learned something new today.

If you have any questions about code reviews, engineering, startups, or business in general, please find me on Twitter: @TheAnkurTyagi. I’d be more than happy to discuss them.

Want to read more interesting articles like this?

You can read more about the latest dev tools like this one on my website.

These terms are frequently used interchangeably, even among experienced developers. But while they may sound similar and occasionally overlap in practice, they address distinctly different problems and serve separate architectural goals. Understanding this distinction is not just an academic exercise. It directly impacts how you build scalable, efficient systems.

Whether you’re developing a high-traffic web server, training complex machine learning models, or optimising application performance, a solid grasp of these concepts can mean the difference between a solution that merely functions and one that scales elegantly under real-world conditions.

This article provides a comprehensive breakdown of both concepts through visual analogies, practical examples, and technical implementations. By the end, you will be equipped to confidently apply these principles in your software projects.

Here’s what we’ll cover:

1. Understanding the Fundamental Concepts

   • The Kitchen Analogy

2. What Concurrency Looks Like in Practice

   • Python Example: Implementing Concurrency with asyncio

3. What Parallelism Looks Like in Practice

   • Python Example: Implementing Parallelism with multiprocessing

4. Concurrency vs. Parallelism: A Detailed Comparison

   • When to Use Each

5. Real-World Applications and Use Cases

   • Concurrency in Production Systems

   • Parallelism in Production Systems

   • Hybrid Approaches

6. Choosing the Right Approach for Your Problem

   • Common Pitfall to Avoid

7. Why This Distinction Matters in Practice

8. Common Misconceptions and Clarifications

9. Practical Implementation Strategies

   • When Implementing Concurrency

   • When Implementing Parallelism

10. Tools and Technologies by Language

11. Further Learning Resources

12. Conclusion

Understanding the Fundamental Concepts

Before diving into implementations, let’s establish some clear definitions:

Concurrency refers to the ability of a system to manage multiple tasks within overlapping time periods. It does not necessarily mean these tasks execute at the exact same instant. Rather, concurrency is about structuring a program to handle multiple operations by interleaving their execution, often on a single processor core.

Parallelism, by contrast, involves the simultaneous execution of multiple tasks. This typically requires multiple CPU cores or processors working in tandem, with each handling a separate portion of the workload at the same time.

The Kitchen Analogy

Consider the process of cooking as a helpful mental model:

A concurrent kitchen employs a single chef who rapidly switches between preparing multiple dishes. The chef might chop vegetables for one dish, then stir a sauce for another, then return to the first dish to continue preparation. From an observer's perspective, it appears that multiple dishes are being prepared "at once," but in reality, the chef is performing one action at a time in rapid succession.

A parallel kitchen has multiple chefs, each working on different dishes simultaneously. One chef prepares the appetiser while another works on the main course, and a third handles dessert. True simultaneous work is happening across multiple workers.

Same kitchen, different strategies, different outcomes.

What Concurrency Looks Like in Practice

Concurrency is fundamentally about task scheduling, coordination, and resource management. It enables a program to handle multiple operations by strategically interleaving their execution, whether on a single core or across multiple threads.

A practical example: when you stream a video on YouTube while your device downloads a file in the background and your messaging app checks for new messages, your CPU is rapidly context-switching between these tasks. Each task gets a slice of processing time, creating the illusion of simultaneous execution even on a single-core processor.

Python Example: Implementing Concurrency with asyncio

To examine concurrency in more detail, we’ll create a simple application which gets data on various APIs asynchronously. This is an example of how Python’s library, asyncio, lets us spawn multiple network operations without blocking so we can effectively use the waiting time.

In this implementation, we’ll be simulating API calls to a weather service, a news service, and a user profile database. Pay attention to the fact that all three requests begin nearly at the same time, yet the program doesn’t wait until one of them is completed before it begins the next one.

import asyncio

async def fetch_data_from_api(api_name, delay):
    print(f"Starting request to {api_name}...")
    await asyncio.sleep(delay)  # Simulates network I/O wait
    print(f"Received response from {api_name}")
    return f"Data from {api_name}"

async def fetch_user_profile(user_id):
    print(f"Fetching profile for user {user_id}...")
    await asyncio.sleep(1.5)
    print(f"Profile loaded for user {user_id}")
    return {"user_id": user_id, "name": "John Doe"}

async def main():
    # All tasks start and are managed concurrently
    results = await asyncio.gather(
        fetch_data_from_api("Weather API", 2),
        fetch_data_from_api("News API", 1),
        fetch_user_profile(12345)
    )
    print("\nAll operations completed!")
    print("Results:", results)

asyncio.run(main())

What happens during execution:

1. All three async functions are initiated at approximately the same time.

2. The event loop manages their execution, switching between tasks when one is waiting (during await statements).

3. While one task waits for simulated I/O, the event loop allows other tasks to make progress.

4. The task with the shortest delay completes first, even though all were started together.

5. No task blocks the others, resulting in efficient use of the single thread.

Key insight: Concurrency optimises responsiveness and resource utilisation. It doesn’t inherently make individual tasks complete faster. Instead, it allows multiple tasks to make progress during the same time period, particularly when those tasks involve waiting for external resources.

What Parallelism Looks Like in Practice

Parallelism concerns itself with genuine simultaneous execution. This approach leverages multiple CPU cores or processors to divide work and execute portions concurrently in real time.

Parallelism shines when dealing with CPU-intensive operations such as mathematical computations, image processing, video rendering, or training deep learning models.

Python Example: Implementing Parallelism with multiprocessing

To better understand parallel execution, we’re going to make a program that carries out intensive calculations in a set of cores of CPUs. The given example relies on Python and the multiprocessing module to create different processes that are executed on different processor cores.

To work with a sufficiently complex example, we’ll compute the sum of the squares of millions of numbers. In contrast to the concurrent code sample, where we were waiting to receive I/O, we are actually doing some CPU-intensive work. You’ll notice the reduction in the time taken to execute the work when it’s shared by a number of cores.

from multiprocessing import Process, current_process
import time

def compute_heavy_task(task_name, iterations):
    """Simulates a CPU-intensive operation"""
    process_name = current_process().name
    print(f"{task_name} started on {process_name}")

    # Simulate CPU-bound work
    result = 0
    for i in range(iterations):
        result += i ** 2

    time.sleep(1)  # Additional simulated work
    print(f"{task_name} completed on {process_name}. Result: {result}")
    return result

if __name__ == "__main__":
    start_time = time.time()

    # Create separate processes for each task
    p1 = Process(target=compute_heavy_task, args=("Task 1", 10000000))
    p2 = Process(target=compute_heavy_task, args=("Task 2", 10000000))
    p3 = Process(target=compute_heavy_task, args=("Task 3", 10000000))

    # Start all processes (they run on separate CPU cores)
    p1.start()
    p2.start()
    p3.start()

    # Wait for all processes to complete
    p1.join()
    p2.join()
    p3.join()

    end_time = time.time()
    print(f"\nAll tasks completed in {end_time - start_time:.2f} seconds")

What happens during execution:

1. Three separate processes are spawned, each allocated to available CPU cores.

2. Each process runs independently with its own memory space and Python interpreter.

3. All three CPU-intensive calculations execute truly simultaneously across multiple cores.

4. The total runtime is determined by the longest-running task, not the cumulative sum of all tasks.

5. On a multi-core system, this completes approximately three times faster than sequential execution.

Key insight: Parallelism achieves actual speedup by distributing computational workload across multiple processors. This directly reduces total execution time for CPU-bound operations.

Concurrency vs. Parallelism: A Detailed Comparison

When to Use Each:

Use concurrency when you want to handle more tasks effectively within the same time period, particularly when those tasks spend time waiting for external resources.

Use parallelism when you want to complete tasks faster by leveraging multiple processors to divide the computational workload.

Real-World Applications and Use Cases

Concurrency in Production Systems

1. Web Servers and APIs

Modern web frameworks like Node.js, Django with async views, and FastAPI handle thousands of simultaneous client connections. Each request may involve database queries, external API calls, or file operations. Concurrency allows the server to handle new requests while waiting for I/O operations from previous requests to complete.

2. Real-Time Communication

Chat applications, collaborative editing tools, and live streaming platforms manage multiple simultaneous connections. Messages must be received, processed, and broadcast to multiple clients concurrently without blocking any single connection.

3. Mobile Applications

Mobile apps perform background synchronization, push notification handling, and data caching while maintaining a responsive user interface. The UI thread remains free while background operations proceed concurrently.

4. Microservices Orchestration

Service meshes coordinate multiple API calls to different microservices, aggregating results efficiently without waiting for each call to complete sequentially.

Parallelism in Production Systems

1. Machine Learning and AI

Training neural networks involves massive matrix computations that can be distributed across multiple GPU cores or even multiple machines. Frameworks like TensorFlow and PyTorch automatically parallelise operations across available hardware.

2. Big Data Processing

Distributed computing frameworks such as Apache Spark, Hadoop, and Dask divide large datasets across cluster nodes. Each node processes its portion of the data in parallel, enabling analysis of petabyte-scale datasets.

3. Media Processing

Video transcoding, image batch processing, and audio rendering leverage multiple CPU cores or GPUs. Each frame or segment can be processed independently in parallel.

4. Scientific Computing

Computational physics simulations, genome sequencing, and climate modelling require enormous computational resources. Parallelism across supercomputer clusters enables these calculations to complete in reasonable time frames.

5. Financial Modelling

Risk analysis and portfolio optimisation involve running thousands of scenarios. Parallel processing allows these computations to execute simultaneously, providing results quickly enough for real-time decision making.

Hybrid Approaches

In practice, sophisticated systems frequently combine both paradigms. Consider a modern web application:

1. The web server handles client requests concurrently (handling multiple users simultaneously).

2. Each request may trigger parallel data processing tasks (such as image resizing across multiple cores).

3. The database connection pool manages concurrent query execution.

4. Background job workers process tasks in parallel (such as sending emails or generating reports).

This layered approach leverages the strengths of both concurrency and parallelism to create systems that are both responsive and computationally efficient.

Choosing the Right Approach for Your Problem

Understanding which paradigm to apply requires analysing the nature of your workload:

Common Pitfall to Avoid

A frequent mistake is attempting to use threading for CPU-bound tasks in languages with a Global Interpreter Lock (like Python's CPython) and expecting parallel speedups. In such cases, threads provide concurrency but not true parallelism.

The GIL ensures only one thread executes Python bytecode at a time, leading to context-switching overhead without genuine parallel execution. For CPU-bound work in Python, multiprocessing or C extensions are necessary for true parallelism.

Why This Distinction Matters in Practice

Grasping the difference between concurrency and parallelism extends beyond writing faster code. It fundamentally influences how you architect systems and make technological decisions:

First of all, choosing the appropriate execution model for each component of your system leads to cleaner, more maintainable code. You avoid over-engineering solutions or applying the wrong tool to a problem.

Understanding these concepts also prevents wasteful patterns such as spawning unnecessary processes for I/O-bound work or using single-threaded approaches for parallelizable computations. This directly translates to reduced infrastructure costs.

Systems designed with proper concurrency models also scale horizontally more effectively. Those leveraging parallelism appropriately utilise hardware resources fully as you scale vertically.

In addition, you’ll get some key performance optimisations by choosing the right approach. When profiling reveals bottlenecks, knowing whether to optimise for concurrency or parallelism guides your refactoring efforts in the right direction.

Beyond this, in cloud environments where you pay for compute resources, efficient use of concurrency and parallelism directly affects operational costs. An efficiently concurrent system might handle 10x the load on the same hardware compared to a poorly designed synchronous alternative.

And these concepts are fundamental to backend engineering, distributed systems, DevOps, machine learning engineering, and systems programming. They appear frequently in technical interviews and are essential for senior engineering roles.

Common Misconceptions and Clarifications

"Using threads automatically gives me parallelism."

In reality, threads enable concurrency but do not guarantee parallel execution. In systems with a Global Interpreter Lock (like CPython) or on single-core machines, threads run concurrently but not in parallel. True parallelism requires multiple CPU cores and mechanisms that avoid locking constraints.

"Parallelism is always faster than sequential execution."

In fact, parallelism introduces overhead, including process creation, inter-process communication, and data synchronisation costs. For small tasks or I/O-bound operations, this overhead can outweigh benefits. Parallelism shows gains when the computational work justifies the overhead.

"Concurrency and parallelism are mutually exclusive."

As you’ve learned, modern high-performance systems routinely combine both. A web server can handle requests concurrently, with each request triggering parallel processing. Understanding how to layer these approaches is key to building sophisticated systems.

"More threads or processes always mean better performance."

Beyond a certain point, adding more threads or processes leads to diminishing returns and even performance degradation due to increased context switching and resource contention. The optimal number depends on workload characteristics and available hardware.

“Async/await makes my code run faster."

Async/await improves efficiency for I/O-bound operations by reducing idle time, but it does not speed up CPU-bound computations. It changes how waiting is handled, not how quickly individual operations execute.

Practical Implementation Strategies

How to Implement Concurrency

To introduce concurrency into your programs, first you’ll need to find where time is wasted. Blocking operations that are held waiting on external resources are the best candidates to be put under concurrent execution.

Say you’re building a web scraper to fetch a bunch of data on a variety of sites. Every single HTTP request is most likely waiting until the server gets a response back. Other requests might be underway in your program instead of waiting around during this waiting period. These wait points are identifiable by profiling your application and searching the operations with network calls, file I/O, or database queries.

After you’ve discovered these wait points, the next big step will be to select the concurrency primitive. In Python, I/O-bound operations perform very well using the patterns of async/await with the support of the asyncio framework. It also comes with a minimal cost.

Take a situation when you have to retrieve user data in a REST API and query a database at the same time. With asyncio, you can write code that initiates both tasks almost simultaneously, and then have the event loop alternate between them during periods of waiting.

Here's a practical example:

import asyncio
import aiohttp

async def fetch_user_api(user_id):
    async with aiohttp.ClientSession() as session:
        async with session.get(f'https://api.example.com/users/{user_id}') as response:
            return await response.json()

async def query_database(user_id):
    # Simulating database query
    await asyncio.sleep(0.5)
    return {'preferences': 'theme:dark', 'notifications': True}

async def get_complete_user_data(user_id):
    api_data, db_data = await asyncio.gather(
        fetch_user_api(user_id),
        query_database(user_id)
    )
    return {**api_data, **db_data}

This gives a thorough look at how concurrency works in practice.

When Implementing Parallelism

Before committing parallelism into your system, you’ll need to profile the system and make sure that what causes your bottleneck is CPU-bound computation. Many developers think that their code requires parallelism when it should actually employ concurrency.

You can use profiling tools such as Python cProfile or line profilers to determine where time is being used or wasted in your program. When the time spent in computational loops is as large as compared to waiting in I/O, then parallelism can be beneficial.

To take an example, when processing images, the execution time in pixel manipulation algorithms consumes 90% of the execution time. This is a good sign that parallelism would be useful.

Deciding how to partition the work between multiple processors is sometimes a complex issue that you should consider carefully (in terms of dividing tasks into independent points). These chunks should be able to be processed individually without needing to communicate with each other on a regular basis.

Imagine that you have to examine the log files of several servers. Processing each file may happen on a different core, and the results will get added at the final stage.

Here's how you might structure this:

from multiprocessing import Pool
import re

def analyze_log_file(filepath):
    error_count = 0
    with open(filepath, 'r') as f:
        for line in f:
            if re.search(r'ERROR|CRITICAL', line):
                error_count += 1
    return filepath, error_count

if __name__ == '__main__':
    log_files = ['server1.log', 'server2.log', 'server3.log', 'server4.log']

    with Pool(processes=4) as pool:
        results = pool.map(analyze_log_file, log_files)

    for filepath, count in results:
        print(f'{filepath}: {count} errors found')

In this example, each log file is processed entirely on one core without needing to communicate with other processes until the final result aggregation.

Tools and Technologies by Language

Various programming languages offer different methods of achieving concurrency and parallelism with their own advantages and disadvantages. And when you understand the available tools in your language of choice, you’ll be able to make a wise architectural choice.

Python

Python has a concurrent and parallel environment. For concurrent programming, the asyncio library offers a more modern syntax of async/await that’s ideal in I/O-bound tasks such as web scraping or API communication.

The threading module allows shared memory execution, but is restricted on CPU-bound tasks by the Global Interpreter Lock. The concurrent futures module is a high-level interface to concurrent task execution, which can be useful when you want to parallelize I/O operations without having to write the low-level code of asynchronous operations.

Sometimes you’ll need actual parallelism because your job requires a lot of CPU time. Multiprocessing starts individual Python processes, which don’t use the GIL at all.

In the case of data science and machine learning processes, distributed parallelism is offered in libraries such as joblib, ray, and dask and can run on your laptop up to a cluster of computers.

JavaScript and Node.js

The event loop architecture had concurrency as its foundation in JavaScript and Node.js. Asynchronous programming is now intuitive with native syntax and Promises being used as the standard model of dealing with I/O operations (like HTTP requests or file system access).

JavaScript is single-threaded, and Node.js is designed to execute single-thread programs that make good use of I/O bound concurrent tasks, such as web servers, which support thousands of parallel connections.

In cases of actual parallelism (for example, image processing or cryptographic tasks), worker threads enable you to execute JavaScript on multiple cores. The child processes module can launch individual instances of Node.js, and the cluster module allows you to launch a pool of workers to accept incoming connections and make the most of all CPU cores in a web server.

Java

Java has mature and battle-tested concurrency and parallelism support. CompletableFuture offers a fluent interface to asynchronous operations, so it’s easier to sequence dependent asynchronous tasks together without any callback hell.

The ExecutorService model also provides detailed management of thread pools and task scheduling, which is necessary in developing high performance server programs. Parallelism Java thread pools are effective at handling worker threads to execute CPU-bound tasks, whereas ForkJoinPool uses work-stealing algorithms that are useful in divide-and-conquer problems.

Java 8 offers parallel streams, which allow you to process collections in parallel with a minimal amount of code rewrites – but you have to pay close attention to when they actually will or will not improve performance.

Go

Go introduced concurrency as a first-class language: goroutines and channels. Goroutines are lightweight threads controlled by the Go runtime, which means that you can run thousands or even millions of operations concurrently with minimal overhead.

The philosophy of communication in Channels offers a secure means of communication between goroutines, and it includes the expression "do not communicate by sharing memory; share memory by communicating." Such a design makes concurrent programming more user-friendly and error-free.

In parallelism, Go automatically allocates goroutines to multiple CPU cores according to the GOMAXPROCS environment variable, and parallel execution is achieved automatically. This renders Go especially effective in the construction of parallel systems such as web servers, network tools, and distributed systems.

Rust

Rust provides concurrent and parallel programming with memory safety without performance degradation. The ownership system of the language eliminates all forms of data races at compile-time, which means that the entire category of concurrency bugs found in other languages doesn’t exist.

In the case of async operations, you can apply the syntax of Rust to operations of an asynchronous type with runtime libraries such as tokio or async-std and achieves similar performance to C++ without sacrificing safety.

The Rayon library makes parallelism of data exceedingly easy. At times, you can parallelise a calculation by substituting .iter() with .par_iter(). Rust thread pools and channels give you low-level control where necessary, and the type system keeps the threads safe, making sure that problems don’t arise in your code.

Conclusion

Concurrency and parallelism represent fundamental pillars of modern computing architecture. They are not interchangeable buzzwords, but are rather distinct paradigms that address different challenges:

Concurrency focuses on program structure and efficient task coordination. It allows systems to handle multiple operations within overlapping time periods, maximizing resource utilization and responsiveness.

Parallelism focuses on computational throughput and execution speed. It divides work across multiple processors to complete tasks faster through simultaneous execution.

The most powerful systems strategically combine both approaches, applying each where it provides the greatest benefit.

The next time you face a performance challenge, ask yourself these critical questions:

1. Is my bottleneck caused by waiting (I/O-bound) or by computation (CPU-bound)?

2. Am I trying to handle more tasks simultaneously or complete tasks faster?

3. Do I need better resource utilisation or raw computational throughput?

Your answers will guide you toward the right solution. Understanding when to apply concurrency, when to leverage parallelism, and when to combine them is what separates adequate solutions from exceptional ones. This knowledge empowers you to build systems that are not only fast but also efficient, scalable, and economically viable.

Master these concepts, and you will find yourself equipped to tackle increasingly complex engineering challenges with confidence and precision.

Further Learning Resources

• "Go Concurrency Patterns" by Rob Pike (Google Tech Talk)

• "Concurrency is not Parallelism" by Rob Pike

• Python AsyncIO Official Documentation

• Real Python: Concurrency and Parallelism in Python

• Java Concurrency in Practice by Brian Goetz

• Apache Spark Documentation for Big Data Parallelism

This comprehensive guide covers everything from basic concepts to advanced optimization techniques, helping you avoid common pitfalls and build better Docker images.

Table of Contents

1. What is Docker Build?

2. Docker Build Architecture: How It All Works

3. Docker Build Features

4. Docker Build Context

5. Types of Docker Build Contexts

6. Common Docker Build Mistakes (And How to Fix Them)

7. How to Optimize and Monitor Build Performance

8. Best Practices for Docker Build Performance

9. Troubleshooting Docker Build Issues

10. Conclusion

What is Docker Build?

Docker build is the process of creating a Docker image from a Dockerfile and a set of files called the build context. When you run docker build, you're instructing Docker to:

1. Read your Dockerfile instructions

2. Gather the necessary files (build context)

3. Execute each instruction step-by-step

4. Create a final Docker image

Think of it like following a recipe: the Dockerfile is your recipe, and the build context contains all the ingredients you might need.

Docker Build Architecture: How It All Works

Docker Build uses a client-server architecture where two separate components (Buildx and BuildKit) work together to build your Docker images. This is different from how many people think Docker works, as it's not just one monolithic program doing everything.

What is Buildx (The Client)?

Buildx serves as the user interface that you interact with directly whenever you work with Docker builds. When you type docker build . in your terminal, you're actually communicating with Buildx, which acts as the intermediary between you and the actual build engine.

Buildx’s primary jobs:

• Interprets your build command and options

• Sends structured build requests to BuildKit

• Manages multiple BuildKit instances (builders)

• Handles authentication and secrets

• Displays build progress to you

What is BuildKit (The Server/Builder)

BuildKit functions as the actual build engine that performs all the heavy lifting during the Docker build process. This powerful backend component receives the structured build requests from Buildx and immediately begins reading and interpreting your Dockerfiles line by line.

BuildKit’s primary jobs:

• Receives build requests from Buildx

• Reads and interprets Dockerfiles

• Executes build instructions step by step

• Manages build cache and layers

• Requests only the files it needs from the client

• Creates the final Docker image

How They Communicate

Here's what happens when you run docker build .:

When you run docker build, the command initiates a multi-step process with BuildKit (as illustrated in the above image).

First, it sends a build request containing your Dockerfile, build arguments, export options, and cache options. BuildKit then intelligently requests only the files it needs when it needs them, starting with package.json to run npm install for dependency installation.

After that's complete, it requests the src/ directory containing your application code and copies those files into the image with the COPY command.

Once all build steps are finished, BuildKit sends back the completed image. Optionally, you can then push this image to a container registry for distribution or deployment.

This on-demand file transfer approach is one of BuildKit's key optimizations: rather than sending your entire build context upfront, it only requests specific files as each build step needs them, making the build process more efficient.

Key Communication Details

Build request contains:

{
  "dockerfile": "FROM node:18\nWORKDIR /app\n...",
  "buildArgs": {"NODE_ENV": "production"},
  "exportOptions": {"type": "image", "name": "my-app:latest"},
  "cacheOptions": {"type": "registry", "ref": "my-app:cache"}
}

Resource requests:

• BuildKit asks: "I need the file at ./package.json"

• Buildx responds: Sends the actual file content

• BuildKit asks: "I need the directory ./src/"

• Buildx responds: Sends all files in that directory

Why This Architecture Exists

1. Efficiency

The old Docker builder had a major flaw: it always copied your entire build context upfront, regardless of what was actually needed. Even if your Dockerfile only used a few files, Docker would transfer hundreds of megabytes before starting the build.

BuildKit fixes this through on-demand file transfers. It only requests specific files at each step.

# Old Docker Builder (legacy)
# Always copied ENTIRE context upfront
$ docker build .
Sending build context to Docker daemon  245.7MB  # Everything!

# New BuildKit Architecture  
# Only requests files when needed
$ docker build .
#1 [internal] load build definition from Dockerfile    0.1s
#2 [internal] load .dockerignore                       0.1s
#3 [1/4] FROM node:18                                  0.5s
#4 [internal] load build context                       0.1s
#4 transferring context: 234B  # Only package.json initially!
#5 [2/4] WORKDIR /app                                  0.2s  
#6 [3/4] COPY package*.json ./                         0.1s
#7 [4/4] RUN npm install                               5.2s
#8 [internal] load build context                       0.3s  
#8 transferring context: 2.1MB  # Now requests src/ files
#9 [5/4] COPY src/ ./src/                              0.2s

2. Scalability

The client-server architecture enables scalability features. Multiple Docker CLI clients can connect to the same BuildKit instance, and BuildKit can run on remote servers instead of your local machine. This means you could execute builds on a cloud server while controlling them from your laptop. Teams can also deploy multiple BuildKit instances for different teams or purposes, scaling from individual developers to large enterprises.

3. Security

Security is improved by only requesting sensitive files when explicitly needed. BuildKit never sees files your Dockerfile doesn't reference, reducing the attack surface. It also handles credentials through separate, secure channels rather than mixing them with your build context, preventing secrets from being embedded in image layers or exposed in build logs.

Real-World Example

Let's trace through a typical build step by step. You can find the full code available here: 02-python-cache.

FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY src/ ./src/
COPY main.py .
CMD ["python", "main.py"]

Let’s see what actually happens here:

1. You run docker build .

2. Buildx says to BuildKit:

   "Here's a build request with this Dockerfile"

3. BuildKit processes: FROM python:3.9-slim

   • No client files needed, pulls base image

4. BuildKit processes: COPY requirements.txt .

   • BuildKit to Buildx: "I need requirements.txt"

   • Buildx to BuildKit: Sends the file content

5. BuildKit processes: RUN pip install -r requirements.txt

   • No client files needed, runs inside container

6. BuildKit processes: COPY src/ ./src/

   • BuildKit to Buildx: "I need all files in src/ directory"

   • Buildx to BuildKit: Sends all files in src/

7. BuildKit processes: COPY main.py .

   • BuildKit to Buildx: "I need main.py"

   • Buildx to BuildKit: Sends the file

8. BuildKit to Buildx: "Build complete, here's your image"

From the illustration, you can see that BuildKit only requests what it needs, when it needs it. Not this entire context:

my-app/
├── src/                 # ← Only loaded when COPY src/ runs
├── tests/              # ← Never requested (not in Dockerfile)
├── docs/               # ← Never requested  
├── node_modules/       # ← Never requested (in .dockerignore)
├── requirements.txt    # ← Loaded early (first COPY)
└── main.py            # ← Loaded later (second COPY)

Docker Build Features

Named Contexts

👉 Demo project: 07-named-contexts

Named contexts allow you to include files from multiple sources during a build while keeping them logically separated. This is useful when you need documentation, configuration files, or shared libraries from different directories or repositories in your build.

# Build with additional named context
docker build --build-context docs=./documentation .

# Use named context in Dockerfile
FROM alpine
COPY . /app
# Mount files from named context
RUN --mount=from=docs,target=/docs \
    cp /docs/manual.pdf /app/

Build Secrets

👉 Demo project: 06-build-secrets

Build secrets let you pass sensitive information (like API keys or passwords) to your build without including them in the final image or build history. The secrets are mounted temporarily during specific RUN commands and are never stored in image layers.

# Pass secret to build
echo "api_key=secret123" | docker build --secret id=apikey,src=- .

# Use secret in Dockerfile
FROM alpine
RUN --mount=type=secret,id=apikey \
    export API_KEY=$(cat /run/secrets/apikey) && \
    curl -H "Authorization: $API_KEY" https://api.example.com/data

Docker Build Context

What is a Build Context?

The build context is the collection of files and directories that Docker can access during the build process. It's like gathering all your cooking ingredients on the counter before you start cooking.

docker build [OPTIONS] CONTEXT
                       ^^^^^^^
                       This is your build context

Why Build Contexts Matter

1. Security: Only files in the context can be accessed during build

2. Performance: Large contexts slow down builds

3. Functionality: Your Dockerfile can only COPY/ADD files from the context

4. Efficiency: Understanding contexts helps you build faster, leaner images

Types of Docker Build Contexts

1. Local Directory Context (Most Common)

👉 See code here: 01-node-local-context

This is what you'll use in 90% of cases – pointing to a folder on your machine:

# Use current directory
docker build .

# Use specific directory
docker build /path/to/my/project

# Use parent directory
docker build ..

Example Project Structure:

my-webapp/
├── src/
│   ├── index.js
│   └── utils.js
├── public/
│   ├── index.html
│   └── styles.css
├── package.json
├── package-lock.json
├── Dockerfile
├── .dockerignore
└── README.md

Corresponding Dockerfile:

FROM node:18-alpine
WORKDIR /app

# Copy package files first for better layer caching
COPY package*.json ./
RUN npm ci --only=production

# Copy application source
COPY src/ ./src/
COPY public/ ./public/

EXPOSE 3000
CMD ["node", "src/index.js"]

2. Remote Git Repository Context

You can build directly from Git repositories without cloning locally:

# Build from GitHub main branch
docker build https://github.com/<username>/project.git

# Build from specific branch
docker build https://github.com/<username>/project.git#develop

# Build from specific directory in repo
docker build https://github.com/<username>/project.git#main:docker

# Build with authentication
docker build --ssh default git@github.com:<username>/private-repo.git

This has various cases like CI/CD pipelines, building open-source projects, ensuring clean builds from source control, automated deployments, and so on.

3. Remote Tarball Context

You can also build from compressed archives hosted on web servers. A remote tarball is a .tar.gz or similar compressed archive file accessible via HTTP/HTTPS. This is useful when your source code is packaged and hosted on a web server, artifact repository, or CDN. Docker downloads and extracts the archive automatically, using its contents as the build context.

This approach works well for CI/CD pipelines where build artifacts are stored centrally, or when you want to build images from released versions of your code without cloning entire repositories.

# Build from remote tarball
docker build http://server.com/context.tar.gz

# BuildKit downloads and extracts automatically
docker build https://example.com/project-v1.2.3.tar.gz

4. Empty Context (Advanced)

When you don't need any files, you can pipe the Dockerfile directly:

# Create image without file context
docker build -t hello-world - <<EOF
FROM alpine:latest
RUN echo "Hello, World!" > /hello.txt
CMD cat /hello.txt
EOF

Common Docker Build Mistakes (And How to Fix Them)

Mistake 1: Wrong Context Directory

👉 Reproduced here: 04-wrong-context

This mistake occurs when you run docker build from the wrong directory, causing the build context to be different from what your Dockerfile expects.

In the example, running docker build frontend/ from the /projects/ directory means the context is /projects/frontend/, but the Dockerfile tries to access ../shared/utils.js, which is outside this context. Docker can only access files within the build context, so any attempt to reference files outside it will fail.

# Project structure
/projects/
├── frontend/
│   ├── Dockerfile
│   ├── src/
│   └── package.json
└── shared/
    └── utils.js

# WRONG - Running from projects directory
docker build frontend/
# This won't work if Dockerfile tries to COPY ../shared/utils.js

How to fix wrong context directory:

The key is aligning your build context with what your Dockerfile needs.

• Option 1 changes your working directory so the context matches your Dockerfile's expectations. You run the build from inside frontend/, making that directory the context root.

• Option 2 keeps you in the parent directory but explicitly sets it as the context (the . argument) while telling Docker where to find the Dockerfile with the -f flag. Now both frontend/ and shared/ are accessible since they're both within the /projects/ context.

# Option 1: Run from correct directory
cd frontend
docker build .

# Option 2: Use parent directory as context
docker build -f frontend/Dockerfile .

Mistake 2: Including Massive Files

👉 Optimized version with .dockerignore: 05-dockerignore-optimization

This mistake happens when your build context contains large, unnecessary files that slow down the build process.

Docker must transfer the entire context to the build daemon before starting, so including files like node_modules (which can be hundreds of MB), git history, build artifacts, logs, and database dumps makes builds painfully slow. These files are rarely needed in the final image and should be excluded.

# This context includes everything!
my-app/
├── node_modules/        # 200MB+ 
├── .git/               # Version history
├── dist/               # Built files
├── logs/               # Log files
├── temp/               # Temporary files
├── database.dump       # 1GB database backup
└── Dockerfile

How to fix Docker build massive files:

Use .dockerignore to exclude unnecessary files, dramatically reducing context size and build time. We’ll discuss this in more detail below.

Mistake 3: Inefficient Layer Caching

👉 See good practice code here: 02-python-cache

This mistake wastes Docker's layer caching system by copying frequently-changing files (like source code) before running expensive operations (like npm install). When you modify your source code, Docker invalidates the cache for that layer and all subsequent layers, forcing npm install to run again even though dependencies haven't changed. This can turn a 5-second build into a 5-minute build.

# BAD - Changes to source code rebuild npm install
FROM node:18
COPY . /app
WORKDIR /app
RUN npm install
CMD ["npm", "start"]

How to fix docker build inefficient layer caching:

Copy dependency files first, install dependencies, then copy source code. This way, npm install only runs when package.json actually changes:

# GOOD - npm install only rebuilds when package.json changes
FROM node:18
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["npm", "start"]

How to Optimize and Monitor Build Performance

Understanding build performance metrics helps you identify bottlenecks and measure improvements.

How to Optimize Docker Builds with .dockerignore

The .dockerignore file is your secret weapon for faster, more secure builds. It tells Docker which files to exclude from the build context.

Creating .dockerignore Patterns

Create a .dockerignore file in your project root. The syntax is similar to .gitignore, and you can use wildcards (*), match specific file extensions (*.log), exclude entire directories (node_modules/), or use negation patterns (!important.txt) to include files that would otherwise be excluded. Each line represents a pattern, and comments start with #.

Example of a .dockerignore file:

# Dependencies
node_modules/
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Build outputs
dist/
build/
*.tgz

# Version control
.git/
.gitignore
.svn/

# IDE and editor files
.vscode/
.idea/
*.swp
*.swo
*~

# OS generated files
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db

# Logs and databases
*.log
*.sqlite
*.db

# Environment and secrets
.env
.env.local
.env.*.local
secrets/
*.key
*.pem

# Documentation
README.md
docs/
*.md

# Test files
test/
tests/
*.test.js
coverage/

# Temporary files
tmp/
temp/
*.tmp

Measuring Build Performance

Analyzing Build Time

Understanding where your build spends time helps identify bottlenecks and optimization opportunities. The detailed progress output shows timing for each build step, cache hits/misses, and resource usage.

# Enable BuildKit progress output
DOCKER_BUILDKIT=1 docker build --progress=plain .

# Use buildx for detailed timing
docker buildx build --progress=plain .

Profiling Context Transfer

Monitor context transfer time to understand how build context size affects overall performance. Profile which directories contribute most to help target .dockerignore optimizations.

# Measure context transfer time
time docker build --no-cache .

# Profile context size by directory
du -sh */ | sort -hr

Measuring .dockerignore Impact

Before .dockerignore, you'll notice that the transfering context size is 245.7MB in 15.2s:

$ docker build .
#1 [internal] load build context
#1 transferring context: 245.7MB in 15.2s

After adding the .dockerignore file, the context reduced to 2.1MB in 0.3s:

$ docker build .
#1 [internal] load build context  
#1 transferring context: 2.1MB in 0.3s

Result: 99% reduction in context size and 50x faster context transfer!

Best Practices for Docker Build Performance

We've covered several optimization techniques throughout this guide. Here's a quick recap of the key practices, plus some additional strategies:

1. Layer Caching (covered in Mistake 3): Copy dependency files before source code to maximize cache reuse.

2. Using .dockerignore (covered in Mistake 2): Exclude unnecessary files to reduce context size and improve build speed.

3. Choosing the Right Context (covered earlier): Select appropriate context types (local, Git, tarball) based on your use case.

Now let’s talk about some more ways you can improve performance:

Use Multi-Stage Builds

👉 Demo project: 03-multistage-node

Multi-stage builds let you use one image for building/compiling your application and a different, smaller image for running it. This dramatically reduces your final image size by excluding build tools, source code, and other unnecessary files from the production image.

# Build stage
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage
FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

Use Specific Base Images

Generic base images like ubuntu:latest include many packages you don't need, making your images larger and slower to download. Specific images like node:18-alpine or distroless images contain only what's necessary for your application to run.

# Large base image
FROM ubuntu:latest

# Smaller, more specific base image  
FROM node:18-alpine

# Even smaller distroless image
FROM gcr.io/distroless/nodejs18-debian11

Combine RUN Commands

Each RUN command creates a new layer in your image. Multiple RUN commands create multiple layers, increasing image size. Combining commands into a single RUN instruction creates just one layer, and you can clean up temporary files in the same step.

# Creates multiple layers
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get clean

# Single layer
RUN apt-get update && \
    apt-get install -y curl && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

Troubleshooting Docker Build Issues

Issue: "COPY failed: no such file or directory"

Problem: File not in build context
What’s going wrong: Docker can only access files within the build context (the directory you specify in docker build). If your Dockerfile tries to COPY a file that doesn't exist in the context directory, the build fails. This often happens when running the build command from the wrong directory or when the file path is incorrect relative to the context root.

Solution:

# Check what's in your context
ls -la

# Verify file path relative to context
docker build -t debug . --progress=plain

Issue: "Docker Build is extremely slow"

Problem: Large build context
What’s going wrong: Docker must transfer your entire build context to the BuildKit daemon before building starts. If your context contains large files, directories like node_modules, or unnecessary files, this transfer can take minutes instead of seconds. The larger the context, the slower your builds become.

Solution:

# Check context size
du -sh .

# Add more patterns to .dockerignore
echo "large-directory/" >> .dockerignore
echo "*.zip" >> .dockerignore

Issue: "Cannot locate specified Dockerfile"

Problem: Dockerfile not in context root
What’s going wrong: By default, Docker looks for a file named Dockerfile in the root of your build context. If your Dockerfile is in a subdirectory or has a different name, Docker can't find it. This is common in monorepo setups where Dockerfiles are organized in separate folders.

Solution:

# Specify Dockerfile location
docker build -f path/to/Dockerfile .

# Or move Dockerfile to context root
mv path/to/Dockerfile .

Issue: "Cache misses on unchanged files"

Problem: File timestamps or permissions changed
What’s going wrong: Docker's layer caching relies on file checksums and metadata. Even if file content is unchanged, different timestamps or permissions can cause cache misses, forcing unnecessary rebuilds. This often happens after git operations, file system operations, or when files are copied between systems.

Solution:

# Check file modifications
git status

# Reset timestamps
git ls-files -z | xargs -0 touch -r .git/HEAD

Conclusion

Understanding Docker build contexts and architecture is essential for achieving faster builds. We’ve covered various techniques in this article, like optimized contexts and caching strategies, creating smaller images with efficient layering and multi-stage builds, maintaining better security with proper secret handling and minimal attack surface, and delivering an improved developer experience with faster iteration cycles.

👉 Full code examples are available on GitHub here: Docker build architecture examples

As always, I hope you enjoyed the article and learned something new. If you want, you can also follow me on LinkedIn or Twitter.

For more hands-on projects, follow and star this repository: Learn-DevOps-by-building

While I figured out early on that narrowing my focus by using a project-based learning approach (as opposed to the topic-by-topic laundry list approach) was important, finding the best tools for the job was a different matter.

If you happen to be searching for some of those things, then this article is for you. After over a decade of programming products, building client applications, answering thousands of questions from junior and intermediate developers, and wrestling with these questions myself, I will do my best to explain how to find the best things.

This article is intended for junior to intermediate level developers looking to get some practical answers to difficult problems. You will not need extensive programming experience to get through it and you may skip over any technical discussions specifics. Those are meant to be helpful pieces of information, but the core of this article is about how to make these decisions in general using what I call: The Law of Suitability.

The topics I will cover are:

• How to Find the Best Anything

  • How to Find the Best Water Bottle

• How to Find the Best Programming Language

  • Navigating Public and Expert Opinions

  • Low Level vs High Level

  • Tightly Structured (Static) vs Loosely Structured (Dynamic)

  • Popularity Is Only One Factor

  • Popularity Is Not A Guarantee Of Employment

  • Hardware & Working With What You Have

  • What if I Want the AI to Code For Me?

  • Avoid the Sunk-Cost Fallacy

• How to Find the Best Libraries and Frameworks

  • What Are Libraries and Frameworks?

  • How to Choose Libraries and Frameworks

• How to Find the Best Programming Principles and Practices

  • D.R.Y – Don’t Repeat Yourself

  • What About Other Programming Principles?

• A Note About Patterns and Architectures

  • How to Find the Best Software Architecture

  • The Design Pattern Trap

• Summary

How to Find the Best Anything

I invite you to follow along with some basic non-technical examples which lay the groundwork for the rest of this article. The examples may sound silly to some, but I have layered in some conceptual patterns which you can apply to programming languages, tools, and concepts – as well as just about anything where terms like good, bad, best, or worst can apply.

How to Find the Best Water Bottle

Suppose that you are looking to solve the problem of staying hydrated and wish to purchase a water bottle.

You consider that figuring out the best water bottle might involve looking into:

• Public opinions and reviews

• Expert opinions and reviews

• The manufacturer’s descriptions and reputation

• Buying and testing water bottles (though preferably not all of them, as that costs too much time and money, generally)

All of those things are fair for consideration. Something which is not well-tested in public presents uncertainty. Expert reviews can help inform your decision, but you have to consider the biases and motivations of such experts. You also should consider if the manufacturer has a history of quality, design, and customer support, or whether they’re simply maximizing profit.

After doing some research, you come up with these options:

• A plastic bottle of water from a vending machine

• A high-tech metal bottle which you can even boil stuff in!

• A simple but ethically sourced, refillable, BPA-free plastic water bottle

However, you noticed that nobody seemed to universally agree about which option was the best. There were usually some common opinions, but it was never the case that every expert had the same evaluation or recommendation.

After reflecting on this, it became obvious that you need to consider how, when, and where you will be using this water bottle. In other words, you need to consider the context or situation of its usage.

Suppose three different contexts within which you have to make this decision:

• You are standing in a rest stop in Death Valley California (often thought to be the hottest place on Earth in summer) and there is a vending machine full of micro-plastic filled, old, generic, and cheap plastic bottles of water in front of you. But you have no other options and are very thirsty

• You are in a camping store preparing for a camping trip to New Zealand with lots of hiking and not too much access to filtered water

• You are looking on your favorite shopping website for something you can bring to work each day to avoid those dehydration headaches from not drinking enough water

In summary, you shouldn’t ignore second-hand knowledge, expert opinions, popularity, ratings, reviews, testimonials, and even first-hand experience. But you will never find the best water bottle for every situation you find yourself in. The best “anything” depends on the problem you are trying to solve and the context (words like requirements or situation also apply here) of that problem.

In other words, none of these things have an absolute or fixed value – their value is always relative. I call that the Law of Suitability.

Let’s now discuss some examples which are directly related to software design and development.

How to Find the Best Programming Language

The Law of Suitability applies just as much to programming languages as it does to water bottles – even though the details and contexts are different. There is no such thing as the best programming language for every person, team, problem, or feature.

But I do have some specific details and contexts to offer which may help you answer that question for yourself. This section will cover concrete details and some general ideas on how to choose a programming language. These patterns also apply to frameworks, libraries, and most other aspects of programming.

If you are not interested in the topic of finding a programming language, feel free to skip to the next sections on topics like libraries, principles, and patterns.

Navigating Public and Expert Opinions

Firstly, you need to be skeptical of popularity and the opinions of experts and “influencers” here.

My general tip here is to be extremely cautious about anyone who makes one of these claims:

• “X” language is the best (though saying “X” language is my favourite is perfectly acceptable)

• “X” language is the worst, is terrible, is dead, is garbage, is useless, and so on.

There are three groups of people who generally make these sorts of statements:

• Actual experts who are voicing their personal preferences but presenting them as immutable facts (how I wish this was not so common in this industry)

• Non-experts parroting opinions of the above group or who have not yet understood the Law of Suitability

• Engagement farmers

It’s also worth noting that people can be experts in a subset of problems but that doesn’t guarantee their opinions about all problems are expert level.

This doesn’t mean you should reflexively dismiss expert opinions in general. Consider both the track records of the person and the degree to which they pay attention to the context of their statement.

Let us look at two examples:

• Expert A says: “Python has the best tooling, support, and ecosystem for ML development”

• Expert B says: “Python is the best programming language”

While Expert B is obviously displaying a lack of precision (either deliberately or not) if you have followed me so far, Expert A is a different case. Whether or not the statements made by Expert A are true (for the record, I have written a bit of Python but no ML code), you can tell that they are considering details and context. Look for people like Expert A!

Low Level vs High Level

In simple terms, low level programming languages are difficult for humans to read and write. Also, they tend to be faster and have a lower memory footprint than high level languages. Conversely, high level languages are closer to human language which generally makes them easier for humans to work with.

I must confess, though, that I have seen plenty of examples of people writing unintelligible code in high level languages – please don’t do that.

Someone working on an embedded system might want to do so in a language like C or C++ to optimize performance or work around the limitations of memory and processing power.

But in enterprise systems, which need to run on a variety of platforms and which closely intermingle with business requirements, rules, and real world objects (thing products, users, and so on), lower level languages are not so popular. After all, upper and middle management generally tends to care about low level optimizations only to the extent it affects the user experience.

I love optimization in general, but never forget that everything is fast with small datasets (that is, when n is small) or high processing power. In simpler terms, sometimes human concerns like legibility are significantly more important than insignificant optimizations on efficiency.

Tightly Structured (Static) vs Loosely Structured (Dynamic)

Ironically, the main downsides and benefits of a language like Java (which is very structured and verbose) versus a language like JavaScript (quite the opposite, depending on how you use it) are the same depending on the context.

Speaking of enterprise systems, using structures, types, interfaces, classes, threading, concurrency primitives, and similar programming constructs can have a variety of benefits. It can insulate you from safety while providing some flexibility via type hierarchies, interfaces, protocols, abstractions and so on.

Further, studying design patterns can teach you repeatable solutions to problems which have been encountered since the dawn of the general purpose computer – or shortly thereafter.

But the Law of Suitability still applies here. Maybe you just need to write a quick script to migrate some data from one SQL database to another. Maybe you know how to approach problems using a more functional approach that doesn’t require or discourages the use of objects, classes, or structs. Maybe you realize one day that trying to apply design patterns, architectures, hierarchies and similar constructs in every situation has actually created as many problems as they were solving. More on that later.

It’s also worth mentioning that most modern language designers and maintainers have understood that we developers like flexibility. Many of us want to avoid premature optimization and unnecessary structure but also don’t want our code to blow up because we accidentally told the program to add together 1.23356 + “Rhinoceros”.

The main point is that structure or a lack of structure is both a blessing and a curse, depending on where it is used.

Popularity Is Only One Factor

I’m not going to say that popularity is irrelevant and that you should start with the least popular hipster programming language you can find. No shade intended to hipster programming languages, but unpopularity is not generally a good thing in isolation, either.

The key point is that many people weighing in on programming languages (probably most) don’t have abundant experience working on a variety of platforms, languages, and settings. If someone has only ever written Python and enjoys doing so, they will naturally tend to regard it above others.

We humans have a tendency to find the first thing that works for us and then die on a hill defending it. But to take a more anecdotal approach here, I know a couple dozen intermediate to senior level developers who have extensive experience in Java and other programming languages. Despite Java still being ranked as one of the most popular languages globally, only one of those developers I know actually prefers to write in Java if they have the choice.

Don’t make the mistake of assuming that the first thing that works for many people will be the last thing you need to try out. I have from time to time experimented with languages such as Haskell, which taught me many valuable lessons about the benefits of making my code more functional (and functionally pure) in nature.

But I have zero intention of using Haskell as my go to solution for building GUI applications.

Popularity Is Not A Guarantee Of Employment

One of the most common things you ought to consider is whether or not the language you pick will help you get a job – assuming that’s a concern. Influencers absolutely love to tell people to choose one particular language because it has the most public commits on GitHub, or another because it has the largest number of programmers using it (which is in practice something that’s impossible to say for sure).

Let me flip that on its head: suppose that the most common programming language and platform combination is JavaScript and web. Let’s further suppose that we have pretty concrete data on the number of job postings on the web which confirms that the largest volume of jobs available is for that combination. Let’s finally suppose that for whatever reason, you strongly dislike JavaScript and enjoyed building a website using PHP.

You will find voices who will tell you that PHP is a dead language and a dead end for job searching.

But if you go looking, you may notice that there is a good supply of job postings out there looking for PHP developers who can expand and maintain existing codebases. You might also have a much better chance of getting an interview because the ratio of job postings to applications is significantly better for PHP developers than JavaScript developers. In fact, my team recently hired a PHP developer!

Hardware & Working With What You Have

This section is largely irrelevant to web developers, but may be extremely important for those looking to target specific hardware or operating systems. Simply put, if you don’t have a computer with Mac OS and XCode, you will have a very hard time developing an iOS app, for example. In my case, back in 2014, I chose Android development partly because I had studied a bit of Java – though a big consideration was that I had an Android phone.

There are some ways around this by paying to use a remote device (such as a remote Mac via an online service), but my experience with such services years ago is that they weren’t great.

Think about what resources you have and how that fits into what you want to build or whom you want to work for. If you have not much other than a cheap computer with a web browser, and you still want to build GUI applications, web development can be a great choice.

What if I Want the AI to Code For Me?

While this topic is worthy of a separate article, I don’t believe this is an unreasonable question to ask. A year ago, I would have told you that at best, the AI can write some basic code and help you learn some things which may or may not be wrong.

How things have changed! Though I would be bad at my job if I copy pasted code I didn’t understand or didn’t test, AI has absolutely become a force multiplier for me as a developer.

Back to the topic in question, how does this relate to choosing a programming language? Well, after telling you that popularity is not always a big deal, by the nature of how LLMs work, popularity is a factor. In terms of general use, languages like Python, JavaScript, and Java are likely to have the largest amount of training data. My experience has been that the languages I typically use, such as Kotlin, TypeScript, and Swift also do fine.

But there is a curious side effect of developing Android or iOS applications that I don’t experience so much in web development. The nature of these constantly changing platforms and SDKs, with tens of thousands of third party libraries, dozens of architectures, and endless opinions about best practices and anti-patterns, means that LLMs can have serious troubles with complexity or specificity.

I expect this issue to be fixed as LLM services improve correctness checking or other methods to reduce hallucinations.

Avoid the Sunk-Cost Fallacy

Perhaps the most important point I can make in choosing a programming language is to avoid the sunk-cost fallacy. For the first several years of my part-time studies, I didn’t imagine learning a second language based on how difficult it was for me to learn Java.

Roughly 12 years later, I have written non-trivial code in Java, Kotlin, Swift, C++, TypeScript, and SQL. Further, I have dabbled with code in C, Python, JavaScript, Racket, Haskell, Objective C, Visual Basic, and C#.

It was not the case that I sought out to learn all of these things artificially – I don’t tend to learn things outside of the problems in front of me. It’s that these learning opportunities naturally unfolded along with my personal and professional interests.

Learning the fundamentals or approaching mastery of any general purpose programming language will have carry over to others. It’s true that someone learning Python or JavaScript without CS fundamentals is not going to have much of a clue how things work at the OS level or lower.

It’s also true that I have met several people who could probably code circles around me in C/C++/Assembly but never made it past building toy programs in University or College.

Just keep learning and try to find a balance between personal interest and professional goals.

How to Find the Best Libraries and Frameworks

The next few topics revolve around a question which we’ll revisit a couple of times before the end of this article: “Does it solve more problems than it creates?”

What Are Libraries and Frameworks?

Before we proceed, here’s a useful but not definitive definition about the relationship between libraries and frameworks. You’ll find other definitions, but there’s remarkably little consensus on topics like this in this industry.

For me, a library is code which you can take from somewhere and use it to build things. It could be anything from a single line to a large and complex sub-system – usually something in between. I could give you a long and pedantic definition, but that’s not appropriate for this context (suitability!).

One example could be Java’s Math (java.lang.Math) library, which provides you with the following: “The class Math contains methods for performing basic numeric operations such as the elementary exponential, logarithm, square root, and trigonometric functions.”

Some people use the term framework interchangeably with library, and I don’t have any problems with that. When I think of a framework, I’m thinking about something which you build stuff around and is not necessarily to do with solving a specific problem domain (such as mathematics).

An example of this would be RxJava, which is a rather complex framework you can use to bind together and manage data flows across an entire application. I’ve used this framework in almost a dozen applications which did very different things in principle.

I do consider a framework to be a library, fundamentally – they just have a different set of goals and often a larger footprint.

How to Choose Libraries and Frameworks

When I think about choosing libraries and frameworks, I ask myself these questions:

• Does it solve more problems than it creates compared to writing my own solution?

• Is it well-maintained (regularly worked on, responsive authors, backed by tech companies)?

• Does it have good documentation (less of a problem now that we can leverage AI for this purpose)?

• What kind of footprint does it have?

Let’s take two examples. I won’t refer to the specific platform or name of these libraries to avoid offending anyone. But they were/are both used in mobile development (though they are solving common GUI problems on any platform).

Firstly, one of my favorite libraries had one job: It loads images into the UI.

Although devices are more powerful than they used to be, it can still be a problem to load large images on smart phones for display. Mobile operating systems can be aggressive about killing programs (that is, processes) which use up too much of the system’s resources.

This library handles all aspects of loading images that I’m concerned about:

• Loading the image into a particular widget

• Displaying an appropriate loading indicator

• Displaying an optional error or fallback state that tells the user something went wrong

• Handling the complexities of asynchronously loading in (via URL/URI), processing, and compressing potentially large streams of bits (that is, image data)

• Doesn’t inflate the packaged application’s size unnecessarily

• Doesn’t change its public API frequently (think changing function names which cause people’s implementations to break when updating versions)

• It solves problems that I’m not interested in solving

Secondly, one of my least favorite libraries also had one job: Pagination. Pagination, or paging, in this case refers to loading data in chunks into an application. This is an extremely common pattern in shopping cart or social media applications.

The library I am thinking of approaches that problem like so:

• Tightly couples every layer of your client application (from front end to back end) to its dependencies

• This tight coupling makes testing difficult without jumping through some hoops

• Handles the core problem of pagination well unless you need customization or specialized cases

• Frequently changed its public facing API

• Solved (in general) a problem which I am quite happy to write my own solution for

• Did not play well with other frameworks due to a restrictive set of types and lack of flexibility

• Was constantly updated for a couple of years then ditched and marked deprecated

• Did not inflate the packaged applications size too much but certainly more than my own solution would

As you can see, even something which solves a core problem reasonably well, can still fail this simple test of: Does it solve more problems than it creates? Having written pagination code by myself on a couple of occasions now, I would have to be pretty strongly convinced not to.

How to Find the Best Programming Principles and Practices

There are more best practices and principles than I care to describe in detail. What I will do is explain why I treat programming principles as being distinct from immutable/unbreakable laws. Similarly to my goal of finding the best programming language, I wanted to find the best principles in order to write the best code.

The problem is that any programming principle I’ve come across has also been subject to the Law of Suitability. I’ll discuss one example from personal experience and point out that the question we asked above, “does it solve more problems than creates,” also applies here.

D.R.Y – Don’t Repeat Yourself

This principle can be summarized with the idea that if you find duplicated code, you should pull it into a separate module (file, function, class, library, and so on). Without getting into the weeds, the act of pulling the duplicated code into a separate module can be thought of as a process of abstraction.

To be fair to the creators and proponents of this idea, it’s more nuanced than that. But many developers never bother to dig that deep into nuances – nor should they have to. I ran into the nuances simply by applying this idea more than I should have.

There a couple of cases where code duplication is sometimes preferable:

• You have a set of similar modules (say similar widgets or business rules) but they get used in different places for different reasons

• You have a set of similar modules which might change for different reasons (for example, rapidly changing demands from product teams and clients with different priorities)

• You’re deliberately grouping certain modules that work together in distinct packages, files, or directories to insulate modules/groupings from affecting each other

• You find that you need to add details about one particular implementation into your abstraction, but those details don’t apply to other implementations (that is, it’s a bad abstraction)

All of the things I listed above are summaries of things I have run into in the past. The key take away is that I broadly agree with avoiding code duplication. I also know of some cases where I prefer it. Suitability!

What About Other Programming Principles?

In general, you can think of all programming principles like YAGNI, DRY, SRP (and other aspects of SOLID), and even software development methodologies like AGILE and Waterfall in the same way. Contextually, you can use them as guidelines to help avoid some common problems. But a person of sufficient creativity and experience can come up with a situation where following any of these principles creates more problems than it solves.

In many cases, you need to apply these things too much in order to understand what too much means in practical terms. Just be careful not to swing too far in the other direction when one of these principles really breaks down in front of you. I’ve also made that mistake and had to re-adjust.

To date, I haven’t come across a programming principle which is universally true. There are some which come close to that, but I can always imagine a situation where they’re not the best approach. Take a good one like: Always write the simplest code you can write. In other words, don’t add extra complexity without a reason.

Well, suppose you have a not great value system or incentive structure which encourages inflating your work artificially. Need I say more?

A Note About Patterns and Architectures

I’ll now discuss the topic of patterns and architectures in software systems with respect to the Law of Suitability. Software architecture is the only thing I consider myself an expert in, and I have read multiple books on design patterns. I always try to provide some useful info on these topics when I get the chance.

How to Find the Best Software Architecture

To summarize entire articles, courses, and public talks I have given on this topic: The best software architecture depends on project and personal requirements.

One way to grasp the main idea is to ask yourself whether the best architecture for a hospital is also a good fit for a 2-bedroom apartment. The obvious answer is that we might expect a few commonalities (doors, windows, bathrooms of some kind, and so on) among these different sets of requirements. But the ideal, or even just a good architecture for a 2 bedroom apartment cannot possibly be the same for a hospital.

In short, you will never find an architecture which works well for all projects and requirements.

Here is a list of architectures I have some familiarity with:

• Model-View-Controller

• Model-View-Presenter

• Model-View-ViewModel

• VIPER

• Clean Architecture (Robert C. Martin style)

• Model-View-Intent

To make matters more confusing, there are multiple different ways to implement these architectures – almost as many ways as developers implementing them! M-V-VM is one of the more common architectures in mobile development, and I can think of at least five different variations on how to achieve what some people think of as a single architecture.

Here are my general suggestions for working with these architectures:

• Be wary of adding unnecessary complexity with the more complex architectures (particularly Clean Architecture, as many people get this horribly wrong)

• Don’t try to make the project requirements fit the architecture – work the other way around (the best indicator for this is noticing that something you’re trying to implement is made unnecessarily difficult because of the architecture you’re using)

• Don’t be afraid of applying different approaches in different features of the same application instead of blindly applying the same pattern just for consistency’s sake

The Design Pattern Trap

One of the most common engagement farming tactics I see on social media is to post lists of design patterns “that you must know” in order to get a job or to scare junior programmers into buying your low-quality, copy-pasted content of each pattern.

Don’t get me wrong, I loved studying design patterns and I use a couple key patterns in most GUI applications I build. The Observer (a.k.a. Publisher-Subscriber or Pub-Sub) Pattern really shines when you need to glue together a bunch of asynchronous data sources. I love seeing library developers give me a nice Builder Pattern to work with their APIs. I think understanding the basics of patterns like the Bridge or Facade can teach you how to hide details behind abstractions, which is actually simpler than the big scary words describing these things make it sound.

But I spend very little time in my day to day work thinking about or in design patterns. Instead, I’m always thinking about the kinds of attitudes and principles that give rise to these patterns:

• Promoting loosely coupled code (separating the creation and usage of dependencies and parameters, reasonable usage of abstraction)

• Writing classes, interfaces, protocols, and functions which do one thing (though this “one thing” might be a macroscopic goal instead of a microscopic operation)

• Avoiding complexity wherever possible (a common source of this complexity is over-use of abstractions)

• Not pretending that every complex problem has a simple solution (that is, as simple as it can be but no simpler)

• Avoiding pre-mature optimization

Again, I will only apply these principles and attitudes to the extent that I find they solve more problems than they create. Design patterns, when applied too rigorously, can break many of those principles – particularly when it comes to avoiding complexity and pre-mature optimization.

Don’t try to make your project requirements fit your patterns. Instead, think about which patterns might suit your project requirements and deviate as necessary.

Summary

My goal with this piece was to provide three things:

• A practical overview of choosing a programming language and avoiding the traps the people can fall into when navigating these sorts of topics

• A philosophical but pragmatic framework which you can use to evaluate the suitability of anything – with emphasis on learning and developing software

• A breakdown of how I approach other topics like tools, architectures, and patterns

While it can be important to consider things like job opportunities and your current hardware, don’t discount personal interest as a driving factor. From what little I remember about studying cognition (learning how to learn), interest is tightly coupled to motivation and memory. We can’t always exclusively do what interests us, but I suggest you look for intersections between personal and practical concerns as often as you can.

In closing, I encourage you to think about other areas where you might be able to explore the principles of suitability and the problems of tribalistic thinking. Change is constant and value is relative.

Zero-Trust Authentication fixes this. Instead of trusting people once they log in, it checks every person, every device, and every request, every single time. The rule is simple: "Trust no one, verify everything."

This isn't just theory – it works. Companies using zero-trust security have smaller breaches, meet compliance rules easier, and control who sees what data. This matters because 95% of data breaches happen due to human mistakes, and the average breach now costs $4.88 million.

In this article, you will learn how to build a complete Zero-Trust Authentication system into your web app step by step. From multi-factor authentication (MFA) to behavioral anomaly detection, we will discuss the architecture decisions, code examples, and some real-world approaches you are likely able to implement right away.

Table of Contents

• Prerequisites

• What Is Zero-Trust Authentication?

• Architecture Overview

• Multi-factor Authentication (MFA)

• JWT Token Management

• Session Security

• Role-Based Access Control (RBAC)

  • Using Middleware to Enforce RBAC

  • Testing Access Control Logic

• Continuous Verification

  • Behavioral Analysis

  • Step-Up Authentication

• Security Monitoring

  • Automating Threat Response

• Conclusion

Prerequisites

Before implementing zero-trust, make sure your stack aligns with frequent calls for token checks, volumes of logging, and the additional auth step, all without impairing system performance on the users' end.

You should at least have knowledge of:

• JWT and secure session handling

• MFA, specifically understanding TOTP

• Basic understanding of middleware design

Audit your system: examine login flows, token handling, protected routes, session termination, and identify weak spots like long sessions or unprotected routes.

What Is Zero-Trust Authentication?

Zero-Trust Authentication (ZTA) redefines how access is granted in contemporary applications. It doesn't take network location or a single login event into account – it demands the continuous validation of an identity, context, and intent.

Whereas perimeter-based models consider anyone inside a network "safe," zero-trust presumes every request can be compromised. This means that access decisions are made in real time over verified identity, device posture, and behavioral signals. In short, it’s a "security-first" approach designed for a cloud-native, threat-aware world.

Architecture Overview

Building a ZTA system means checking everyone and everything, all the time. The architecture you can see below demonstrates this "never trust, always verify" approach in action:

Image source: civilsdaily

Here's how it works:

• Every request gets checked: When anyone tries to access your network (from office, home, or mobile), they hit the authentication layer first. No exceptions.

• Identity + context verification: The system doesn't just check passwords. It looks at who you are, what device you're using, where you're connecting from, and what you're trying to access.

• Continuous protection: Once inside, the system keeps watching. It protects your data, devices, networks, people, and workloads through constant monitoring and access controls.

• The big change: Traditional security created a "trusted inside" and "untrusted outside." Zero-trust eliminates this boundary. Whether you're connecting to cloud services (AWS, Office 365) or internal systems, every request goes through the same verification process.

Multi-factor Authentication (MFA)

MFA is the foundation of zero-trust security. It requires users to prove who they are with multiple pieces of evidence before getting access. In ZTA, even the strongest password isn't enough on its own.

To begin, start with a strong password, then add a second factor. For example, Time-based One-Time Password (TOTP) is the most secure. TOTP is the best second factor because it works offline and doesn't rely on SMS or email (which can be intercepted). Apps like Google Authenticator generate a new code every 30 seconds.

Here’s an example of what that would look like:

const speakeasy = require('speakeasy');
const QRCode = require('qrcode');

// Generate TOTP secret for new user
function generateTOTPSecret(userEmail) {
  const secret = speakeasy.generateSecret({
    name: userEmail,
    issuer: 'YourApp',
    length: 32
  });

  return {
    secret: secret.base32,
    qrCodeUrl: secret.otpauth_url
  };
}

When a new user signs up, this function creates a unique secret key just for them. The name is their email, issuer is your app name, and length: 32 makes it extra secure. It returns two things: the secret key (in base32 format) and a special URL that creates a QR code for easy setup.

To verify the code from their app, you check it against the stored secret:

// Verify TOTP token
function verifyTOTP(token, secret) {
  return speakeasy.totp.verify({
    secret: secret,
    token: token,
    window: 2,
    encoding: 'base32'
  });
}

When the user enters their 6-digit code, this function checks if it's correct. The window: 2 is smart – it allows for timing differences (like if their phone clock is slightly off). It returns true if the code is valid, false if not.

SMS verification can serve as a backup option. It’s less secure than TOTP but can work as a backup. Always limit how many SMS codes someone can request to prevent abuse:

// SMS verification with rate limiting
async function sendSMSVerification(phoneNumber, userId) {
  const attempts = await getRecentSMSAttempts(userId);
  if (attempts >= 3) {
    throw new Error('Too many SMS attempts. Please try again later.');
  }

  const code = generateRandomCode(6);
  await storeSMSCode(userId, code, 300); // 5-minute expiry

  await smsProvider.send(phoneNumber, `Your verification code: ${code}`);
}

Before sending an SMS, it checks how many times this user has already requested codes. If they've tried 3 times, it blocks them (prevents spam/abuse). If they're under the limit, it creates a random 6-digit code, saves it for 5 minutes (300 seconds), then sends it via SMS.

But what happens if a user loses their phone or authenticator app? Backup codes provide emergency access:

// Generate backup codes
function generateBackupCodes(userId) {
  const codes = [];
  for (let i = 0; i < 10; i++) {
    codes.push(generateRandomCode(8));
  }

  const hashedCodes = codes.map(code => hashCode(code));
  storeBackupCodes(userId, hashedCodes);

  return codes; // Only show to user once
}

This creates 10 emergency backup codes (each 8 characters long). The for loop runs 10 times, creating a new random code each time. Before storing them in the database, it "hashes" them (scrambles them for security). Then it returns the original codes to show the user once, but stores the scrambled versions so even if someone hacks your database, they can't see the real codes.

JWT Token Management

JSON Web Tokens (JWTs) are stateless authentication in a zero-trust system. Using them safely is critical because you need to carefully think through payload design, implement short expiration policies, and implement token rotation and blocklisting that could prevent token theft, token reuse, or privilege escalation.

Let's walk through how to securely implement and manage JWTs in your web application.

First, define a minimal and secure structure for your access tokens. Only add information that’s necessary for making authorization decisions, and never put anything sensitive even if it is encrypted.

// JWT payload structure
const tokenPayload = {
  sub: userId,           // Subject (user ID)
  email: userEmail,      // User identifier
  roles: userRoles,      // User roles array
  permissions: userPermissions, // Specific permissions
  iat: Math.floor(Date.now() / 1000), // Issued at
  exp: Math.floor(Date.now() / 1000) + 900, // Expires in 15 minutes
  jti: generateUniqueId(), // JWT ID for blocklisting
  aud: 'your-app',       // Audience
  iss: 'your-auth-service' // Issuer
};

In the code above, the payload consists of the user identity, roles, permissions, and metadata such as the issued time (iat), expiration (exp), and unique token ID (jti). While aud and iss describe the token's origin and audience for validation, jti is used for revocation. Thus, it keeps the payload as lean as possible to minimize exposure and overhead.

For security and usability, it’s better to use access tokens with a short lifespan and refresh tokens with a considerably longer duration, which minimizes the window for potential utilization of compromised tokens while providing a smooth user session.

Let's take this example:

// Token generation service
class TokenService {
  generateTokenPair(user) {
    const accessToken = jwt.sign(
      this.createAccessTokenPayload(user),
      process.env.JWT_SECRET,
      { expiresIn: '15m', algorithm: 'HS256' }
    );

    const refreshToken = jwt.sign(
      { sub: user.id, type: 'refresh' },
      process.env.REFRESH_SECRET,
      { expiresIn: '7d', algorithm: 'HS256' }
    );

    return { accessToken, refreshToken };
  }

  async refreshAccessToken(refreshToken) {
    try {
      const decoded = jwt.verify(refreshToken, process.env.REFRESH_SECRET);

      // Check if refresh token is blocklisted
      if (await this.isTokenBlocklisted(decoded.jti)) {
        throw new Error('Token has been revoked');
      }

      const user = await getUserById(decoded.sub);
      return this.generateTokenPair(user);
    } catch (error) {
      throw new Error('Invalid refresh token');
    }
  }
}

generateTokenPair will generate two signed JWTs – that is, an access token with a 15-minute expiration and a refresh token with a validity of 7 days. The refresh tokens are verified to grant new ones and are checked against a blocklist. This ensures that revoked tokens can’t be reused, even if they’re still technically valid.

If you choose, a sliding session can be implemented to reduce friction by renewing tokens for an active user without violating your expiration strategy.

Now, let's implement a sliding session that automatically refreshes JWTs when they're close to expiring and the user is still active.

// Sliding session implementation
async function extendSessionIfActive(token) {
  const decoded = jwt.decode(token);
  const timeUntilExpiry = decoded.exp - Math.floor(Date.now() / 1000);

  // If token expires within 5 minutes and user is active, refresh
  if (timeUntilExpiry < 300 && await isUserActive(decoded.sub)) {
    const user = await getUserById(decoded.sub);
    return this.generateTokenPair(user);
  }

  return null;
}

The above function checks for token expiration. If the token expires within 5 minutes and the user continues to interact, a new access token pair is issued. This way, the session is kept alive during real activity but still forces expiration for idle users.

// Token blocklist service
class TokenBlocklistService {
  async blocklistToken(token) {
    const decoded = jwt.decode(token);
    const expiresAt = new Date(decoded.exp * 1000);

    // Store in Redis with automatic expiry
    await redis.setex(
      `blocklist:${decoded.jti}`,
      Math.max(0, Math.floor((expiresAt - Date.now()) / 1000)),
      'revoked'
    );
  }

  async isTokenBlocklisted(jti) {
    const result = await redis.get(`blocklist:${jti}`);
    return result !== null;
  }
}

In the above code, when users log out or tokens are compromised, the jti is stored in Redis with an expiration time of the remaining life of the token. You can block future uses of a token by checking if its ID exists on the blocklist. This allows for instant invalidation, even though JWTs are stateless.

Session Security

In zero-trust environments, session management goes far beyond keeping users logged in. A session must be treated as a constantly evaluated contract between the user, their device, and the system – and should be revoked the moment trust breaks down.

Here, we’ll build a session system that incorporates adaptive trust scoring, dynamic timeouts, real-time visibility, and revocation mechanisms – all aligned with zero-trust principles.

For example, when a user successfully authenticates, you don’t just store a session ID. Instead, you collect contextual metadata to evaluate ongoing risk. The function below demonstrates how to initialize a session that’s both secure and context-aware.

// Comprehensive session creation
async function createSecureSession(userId, deviceInfo, clientInfo) {
  const sessionId = generateSecureSessionId();

  const session = {
    id: sessionId,
    userId: userId,
    deviceFingerprint: generateDeviceFingerprint(deviceInfo),
    ipAddress: clientInfo.ipAddress,
    userAgent: clientInfo.userAgent,
    location: await resolveLocation(clientInfo.ipAddress),
    createdAt: new Date(),
    lastActivity: new Date(),
    trustScore: calculateInitialTrustScore(deviceInfo, clientInfo),
    securityLevel: determineSecurityLevel(userId, deviceInfo)
  };

  await storeSession(session);
  return session;
}

Many other tools are tracking concerning details during session creation. The device fingerprint, IP address, geolocation, and browser agent data are collected. These metadata are used to compute a trust score, and finally, a security level is assigned to the session to be used for dynamically adjusting policies later.

With this contextual information captured during session creation, the system can spot suspicious behavior during the sessions and, in turn, adapt policies like re-authentication of users or termination of the session.

Not all sessions should be treated equally. If a user logs in via an unfamiliar device or risky location, they should have less time for their session lifespan compared to a trusted setup's time. The following implementation changes timeout periods on the basis of trust and risk factors:

// Adaptive session timeout
class SessionTimeoutManager {
  calculateTimeoutPeriod(session) {
    const baseTimeout = 30 * 60 * 1000; // 30 minutes
    const trustMultiplier = session.trustScore / 100;
    const securityMultiplier = this.getSecurityMultiplier(session.securityLevel);

    return Math.max(
      5 * 60 * 1000, // Minimum 5 minutes
      baseTimeout * trustMultiplier * securityMultiplier
    );
  }

  async checkSessionValidity(sessionId) {
    const session = await getSession(sessionId);
    if (!session) return false;

    const now = Date.now();
    const timeout = this.calculateTimeoutPeriod(session);

    // Check both idle timeout and absolute timeout
    const idleExpired = (now - session.lastActivity) > timeout;
    const absoluteExpired = (now - session.createdAt) > 8 * 60 * 60 * 1000; // 8 hours max

    return !idleExpired && !absoluteExpired;
  }
}

The above code keeps session duration adaptable to the risk context at hand. The timeout is calculated by adjusting the base value according to trust and security level, while imposing minimum and maximum bounds.

The system then periodically intervenes to see if the session has become invalid due to inactivity (idle timeout) or simply outlives its initial duration (absolute timeout). This provides a more flexible yet enforceable way of mitigating the risk behind stale or hijacked sessions.

Zero-trust should also mean visibility across all access points. The user should be able to view all active sessions associated with their account, and security systems should also allow them to control these sessions in fine-grained detail. The following code lets you manage those active sessions across devices.

// Cross-device session management
class SessionManager {
  async getUserSessions(userId) {
    const sessions = await getActiveSessionsForUser(userId);

    return sessions.map(session => ({
      id: session.id,
      deviceType: this.identifyDeviceType(session.userAgent),
      location: session.location,
      lastActivity: session.lastActivity,
      current: session.id === currentSessionId
    }));
  }

  async revokeSession(sessionId, requestingSessionId) {
    const session = await getSession(sessionId);
    if (!session) throw new Error('Session not found');

    // Verify requesting session has permission
    const requestingSession = await getSession(requestingSessionId);
    if (requestingSession.userId !== session.userId) {
      throw new Error('Unauthorized');
    }

    await this.terminateSession(sessionId);
    await this.logSecurityEvent('session_revoked', session);
  }
}

Here, users fetch a list of their active sessions along with identifying information such as device type and location. Any session can be securely revoked by the user who owns it, preventing unauthorized access if the session ID is compromised.

This also allows the user to detect suspicious activities in time. All revocations are logged for auditing purposes to enable post-incident investigations as well as compliance reports.

When a trust breaks due to credential theft, suspicious activity, or user-level actions such as password reset, all sessions have to be immediately revoked. This example guarantees a full revocation, promptly applied to all devices:

// Real-time session revocation
class SessionRevocationService {
  async revokeAllUserSessions(userId, reason) {
    const sessions = await getActiveSessionsForUser(userId);

    // Blocklist all tokens for this user
    await Promise.all(sessions.map(session => 
      this.blocklistSessionTokens(session.id)
    ));

    // Notify all active clients
    await Promise.all(sessions.map(session => 
      this.notifySessionTermination(session.id, reason)
    ));

    // Clear session data
    await clearUserSessions(userId);

    // Log security event
    await this.logSecurityEvent('all_sessions_revoked', {
      userId,
      reason,
      sessionCount: sessions.length
    });
  }
}

The above code permits full-scale revocation. It blocklists all session tokens, sends out termination notices to active clients (for example, through WebSockets), clears the session records on the server-side, and logs the event for auditing. It is an instantaneous and complete response to compromised accounts or states where user risk is very high. It is the foremost component of real-time zero-trust enforcement in any serious authentication system.

Role-Based Access Control (RBAC)

Identity verification determines what users can access once they’re logged in. As the basis for any system that is aware of permissions and follows least privilege, RBAC doesn’t grant access on an individual basis – it groups users into roles that define the operations they are permitted to perform.

Before assigning roles to users, you need a structured system to define what each role can do. A set of granular permissions is first identified and then aggregated under these roles, optionally allowing inheritance and hierarchy. The code below shows how to build a basic permission system:

// RBAC permission system
class PermissionSystem {
  constructor() {
    this.permissions = new Map();
    this.roles = new Map();
    this.roleHierarchy = new Map();
  }

  // Define granular permissions
  definePermission(name, description, resource, action) {
    this.permissions.set(name, {
      name,
      description,
      resource,
      action,
      createdAt: new Date()
    });
  }

  // Create role with inherited permissions
  createRole(name, description, parentRole = null) {
    const role = {
      name,
      description,
      permissions: new Set(),
      createdAt: new Date()
    };

    // Inherit permissions from parent role
    if (parentRole && this.roles.has(parentRole)) {
      const parent = this.roles.get(parentRole);
      role.permissions = new Set(parent.permissions);
      this.roleHierarchy.set(name, parentRole);
    }

    this.roles.set(name, role);
    return role;
  }

  // Add permission to role
  addPermissionToRole(roleName, permissionName) {
    const role = this.roles.get(roleName);
    if (!role) throw new Error('Role not found');

    if (!this.permissions.has(permissionName)) {
      throw new Error('Permission not found');
    }

    role.permissions.add(permissionName);
  }
}

The code above lets you specify fine-grained permissions like documents.read.own and organizes them into roles such as employee or manager that you can independently reuse. You can define roles to inherit from other roles, which avoids redundancy and promotes a consistent, scalable access control logic.

As a general rule to avoid privilege creep, permissions should always be as fine-grained as possible. This lets the application refine access decisions to specific actions or scopes: for example, allowing users to read only their documents versus reading all documents for their team.

// Fine-grained permission definitions
const permissions = {
  // User management
  'users.read': { resource: 'users', action: 'read' },
  'users.create': { resource: 'users', action: 'create' },
  'users.update': { resource: 'users', action: 'update' },
  'users.delete': { resource: 'users', action: 'delete' },

  // Document management
  'documents.read.own': { resource: 'documents', action: 'read', scope: 'own' },
  'documents.read.team': { resource: 'documents', action: 'read', scope: 'team' },
  'documents.read.all': { resource: 'documents', action: 'read', scope: 'all' },
  'documents.create': { resource: 'documents', action: 'create' },
  'documents.update.own': { resource: 'documents', action: 'update', scope: 'own' },
  'documents.delete.own': { resource: 'documents', action: 'delete', scope: 'own' },

  // System administration
  'system.logs.read': { resource: 'system', action: 'read', subresource: 'logs' },
  'system.config.update': { resource: 'system', action: 'update', subresource: 'config' }
};

With an array of permissions at its disposal, the app can undertake very precise access control decisions. Instead of merely addressing the binary "is admin" question, this capability enables the system to answer questions such as "can this user delete their own document but not others?"

Static roles are often insufficient. You may want to give people temporary or conditional access, for example, when the team lead takes over for a manager or when a user approves a higher access level for the sake of incident response.

To support these cases, the RBAC system must allow dynamic role assignment – that is, the ability to assign roles on the basis of time, context, or an external trigger such as a security workflow.

The code below assigns a temporary role to a user, notes the exact time at which the role was assigned to the user, and periodically revokes the right after some fixed amount of time. Also, it has a method to calculate a user's complete set of active rights, depending on their permanent rights, temporary rights, and role-based contextual rights.

// Dynamic role assignment system
class DynamicRoleAssignment {
  async assignTemporaryRole(userId, roleName, duration, reason) {
    const assignment = {
      userId,
      roleName,
      assignedAt: new Date(),
      expiresAt: new Date(Date.now() + duration * 1000),
      reason,
      active: true
    };

    await this.storeRoleAssignment(assignment);
    await this.logRoleAssignment(assignment);

    // Schedule automatic revocation
    setTimeout(() => {
      this.revokeExpiredAssignment(assignment.id);
    }, duration * 1000);

    return assignment;
  }

  async getUserEffectivePermissions(userId, context = {}) {
    const user = await getUserById(userId);
    const permanentRoles = user.roles || [];
    const temporaryRoles = await this.getActiveTemporaryRoles(userId);
    const contextualRoles = await this.getContextualRoles(userId, context);

    const allRoles = [...permanentRoles, ...temporaryRoles, ...contextualRoles];
    const permissions = new Set();

    for (const roleName of allRoles) {
      const rolePermissions = await this.getRolePermissions(roleName);
      rolePermissions.forEach(permission => permissions.add(permission));
    }

    return Array.from(permissions);
  }
}

This allows for more flexible security configurations. Temporary roles that are granted have an automatic expiration. The context roles may be added dynamically depending on contextual factors such as location or type of device. Permanent roles are combined with temporary and context roles to compute the aggregate permission set for the user on a per-request basis, which maintains flexibility without compromising control.

Using Middleware to Enforce RBAC

The RBAC policies have to be enforced before any request reaches a protected route or protected data. Middleware is a good place to run such checks in the scope of a web application. We’ll now look into how the reusable middleware function for authorization works.

// Authorization middleware
function createAuthorizationMiddleware(requiredPermission) {
  return async (req, res, next) => {
    try {
      // Extract user from validated JWT
      const user = req.user;
      if (!user) {
        return res.status(401).json({ error: 'Authentication required' });
      }

      // Get user's effective permissions
      const context = {
        ipAddress: req.ip,
        userAgent: req.get('User-Agent'),
        resourceId: req.params.id,
        timestamp: new Date()
      };

      const permissions = await roleSystem.getUserEffectivePermissions(
        user.id,
        context
      );

      // Check if user has required permission
      if (!permissions.includes(requiredPermission)) {
        await logUnauthorizedAccess(user.id, requiredPermission, context);
        return res.status(403).json({ error: 'Insufficient permissions' });
      }

      // Add permissions to request for downstream use
      req.userPermissions = permissions;
      next();
    } catch (error) {
      res.status(500).json({ error: 'Authorization check failed' });
    }
  };
}

// Usage in routes
app.get('/api/users', 
  authenticateToken,
  createAuthorizationMiddleware('users.read'),
  getUsersController
);

In the code above, the middleware will validate user identities in real-time, check if adequate permissions are granted, and allow or deny access accordingly. It’s a central mechanism for enforcing access rules in a uniform way across your routes, and it even records unauthorized attempts for auditing.

Testing Access Control Logic

Once you’ve implemented the RBAC system, testing becomes a must. You want to guarantee that permissions are inherited properly, that access is actually denied when a user isn’t authorized, and that your roles behave as designed in the real world as well as in edge-case scenarios.

The following example uses a testing framework to demonstrate the verification of two fundamental behaviors: inheritance of permissions from parent roles and rejection of unauthorized access.

// RBAC testing suite
describe('RBAC System', () => {
  test('should inherit permissions from parent roles', async () => {
    const manager = await roleSystem.createRole('manager', 'Team Manager', 'employee');
    await roleSystem.addPermissionToRole('manager', 'team.manage');

    const permissions = await roleSystem.getRolePermissions('manager');
    expect(permissions).toContain('documents.read.own'); // From employee
    expect(permissions).toContain('team.manage'); // Manager-specific
  });

  test('should deny access without proper permissions', async () => {
    const user = { id: 1, roles: ['employee'] };
    const req = { user, params: { id: 'doc123' } };
    const res = { status: jest.fn().mockReturnThis(), json: jest.fn() };

    const middleware = createAuthorizationMiddleware('documents.delete.all');
    await middleware(req, res, () => {}); // Middleware call simulating request

    expect(res.status).toHaveBeenCalledWith(403);
  });
});

The tests represent the positive and negative validations of the access rules. The first test determines whether inherited permissions flow freely from the parent to child roles. The second test blocks any user without the required permission, returning a status code appropriately.

Over time, you can enrich test coverage to include temporary role assignments, contextual conditions, and session-aware behavior to alert you to any regressions before they start affecting production access.

Continuous Verification

Modern access security is not a one-shot check but an ongoing process. A strong system must continuously verify user identity and context throughout the ongoing session while adapting to newly emerging risk signals.

In continuous verification, it’s an assurance that access stays appropriate while the user behavior, device posture, or environment changes mid-session.

To uniquely identify a device, you can combine subtle traits like browser settings, hardware specs, and plugin data. This forms a device “fingerprint,” which helps flag new or suspicious devices attempting access.

// Advanced device fingerprinting
class DeviceFingerprintService {
  generateFingerprint(deviceInfo) {
    const components = [
      deviceInfo.userAgent,
      deviceInfo.screenResolution,
      deviceInfo.timezone,
      deviceInfo.language,
      deviceInfo.platform,
      deviceInfo.hardwareConcurrency,
      deviceInfo.memorySize,
      deviceInfo.availableFonts?.join(','),
      deviceInfo.plugins?.map(p => p.name).join(','),
      deviceInfo.webglRenderer,
      deviceInfo.audioContext
    ];

    return this.hashComponents(components);
  }

  calculateTrustScore(currentFingerprint, knownFingerprints) {
    if (knownFingerprints.length === 0) return 50; // Neutral for new device
    const similarities = knownFingerprints.map(known =>
      this.calculateSimilarity(currentFingerprint, known)
    );
    return Math.min(100, Math.max(...similarities) * 100);
  }

  async updateDeviceTrust(userId, deviceFingerprint, securityEvents) {
    const device = await this.getOrCreateDevice(userId, deviceFingerprint);
    let trustAdjustment = 0;

    securityEvents.forEach(event => {
      switch (event.type) {
        case 'successful_login': trustAdjustment += 5; break;
        case 'failed_login': trustAdjustment -= 10; break;
        case 'suspicious_activity': trustAdjustment -= 25; break;
      }
    });

    device.trustScore = Math.max(0, Math.min(100, device.trustScore + trustAdjustment));
    await this.updateDevice(device);
    return device.trustScore;
  }
}

Generating a fingerprint hash from device traits, this service uses historical events to dynamically adjust the device's trust score. Step-up authentication may be prompted by low scores, or access may be denied altogether.

Behavioral Analysis

People tend to use apps rather consistently – they type a certain way, move the mouse in a particular manner, or browse varied content. Behavioral analysis tries to detect that anomaly by comparing ongoing activities to known ones.

// Behavioral analysis system
class BehaviorAnalysisService {
  async analyzeUserBehavior(userId, currentSession) {
    const historicalBehavior = await this.getUserBehaviorProfile(userId);
    const anomalies = [];

    const typingAnomaly = this.analyzeTypingPatterns(
      currentSession.typingData,
      historicalBehavior.typingProfile
    );
    if (typingAnomaly.score > 0.7) {
      anomalies.push({ type: 'typing_pattern', score: typingAnomaly.score, details: typingAnomaly.details });
    }

    const navigationAnomaly = this.analyzeNavigationPatterns(
      currentSession.navigationData,
      historicalBehavior.navigationProfile
    );
    if (navigationAnomaly.score > 0.6) {
      anomalies.push({ type: 'navigation_pattern', score: navigationAnomaly.score, details: navigationAnomaly.details });
    }

    const timeAnomaly = this.analyzeTimePatterns(
      currentSession.timestamp,
      historicalBehavior.timeProfile
    );
    if (timeAnomaly.score > 0.5) {
      anomalies.push({ type: 'time_pattern', score: timeAnomaly.score, details: timeAnomaly.details });
    }

    return {
      overallRiskScore: this.calculateOverallRisk(anomalies),
      anomalies,
      recommendations: this.generateRecommendations(anomalies)
    };
  }

  analyzeTypingPatterns(currentData, historicalProfile) {
    if (!currentData || !historicalProfile) return { score: 0 };
    const dwellTimeVariance = this.calculateVariance(currentData.dwellTimes, historicalProfile.averageDwellTime);
    const flightTimeVariance = this.calculateVariance(currentData.flightTimes, historicalProfile.averageFlightTime);
    const score = Math.max(dwellTimeVariance, flightTimeVariance);
    return { score, details: { dwellTimeVariance, flightTimeVariance, sampleSize: currentData.keystrokes.length } };
  }
}

This will detect suspicious changes in user behavior and typing characteristics as early warning indicators of session hijacking or insider threat.

Access from a new country or city can either be harmless or highly suspicious. Comparing login geography against historical patterns helps flag impossible travel or access from banned regions.

// Location-based access control
class LocationAccessControl {
  async validateLocationAccess(userId, ipAddress, session) {
    const location = await this.resolveLocation(ipAddress);
    const user = await getUserById(userId);
    const historicalLocations = await this.getUserLocations(userId);
    const locationRisk = this.assessLocationRisk(location, historicalLocations);

    const lastLocation = await this.getLastKnownLocation(userId);
    if (lastLocation) {
      const impossibleTravel = this.checkImpossibleTravel(lastLocation, location, session.lastActivity);
      if (impossibleTravel.detected) {
        await this.logSecurityEvent('impossible_travel', {
          userId, fromLocation: lastLocation, toLocation: location,
          timeWindow: impossibleTravel.timeWindow,
          minimumTravelTime: impossibleTravel.minimumTravelTime
        });
        return { allowed: false, reason: 'impossible_travel', requiresStepUp: true };
      }
    }

    if (user.allowedCountries && !user.allowedCountries.includes(location.country)) {
      return { allowed: false, reason: 'country_restriction', requiresStepUp: true };
    }

    const highRiskCountries = ['XX', 'YY', 'ZZ'];
    if (highRiskCountries.includes(location.country)) {
      return { allowed: true, reason: 'high_risk_location', requiresStepUp: true, additionalVerification: ['sms', 'email'] };
    }

    return { allowed: true, riskScore: locationRisk, location };
  }

  checkImpossibleTravel(fromLocation, toLocation, lastActivity) {
    const distance = this.calculateDistance(fromLocation, toLocation);
    const timeElapsed = Date.now() - lastActivity;
    const maximumSpeed = 900; // km/h
    const minimumTravelTime = (distance / maximumSpeed) * 3600000;
    return { detected: timeElapsed < minimumTravelTime, timeWindow: timeElapsed, minimumTravelTime, distance };
  }
}

This logic prevents abuse via VPNs or stolen credentials by requiring step-up verification when impossible travel or unusual locations are detected.

Step-Up Authentication

Step-up security introduces friction only when truly needed. With lower risk considered, users move freely. When risk levels rises, they're asked for stronger proofs, such as biometrics or hardware tokens.

// Step-up authentication system
class StepUpAuthenticationService {
  async evaluateStepUpRequirement(userId, requestContext, resourceSensitivity) {
    const riskFactors = await this.calculateRiskFactors(userId, requestContext);
    const stepUpRequired = this.shouldRequireStepUp(riskFactors, resourceSensitivity);

    if (stepUpRequired.required) {
      return {
        required: true,
        methods: this.selectAuthenticationMethods(riskFactors, stepUpRequired.level),
        expiresIn: this.calculateStepUpDuration(stepUpRequired.level),
        reason: stepUpRequired.reason
      };
    }

    return { required: false };
  }

  async calculateRiskFactors(userId, context) {
    return {
      deviceTrust: await this.getDeviceTrustScore(userId, context.deviceFingerprint),
      locationRisk: await this.getLocationRiskScore(userId, context.ipAddress),
      behaviorAnomaly: await this.getBehaviorAnomalyScore(userId, context.sessionData),
      timeSinceLastAuth: Date.now() - context.lastAuthTime,
      resourceSensitivity: context.resourceSensitivity || 'medium'
    };
  }

  shouldRequireStepUp(riskFactors, sensitivity) {
    let score = 0;
    if (riskFactors.deviceTrust < 70) score += 30;
    if (riskFactors.deviceTrust < 40) score += 20;
    if (riskFactors.locationRisk > 0.6) score += 25;
    if (riskFactors.locationRisk > 0.8) score += 15;
    if (riskFactors.behaviorAnomaly > 0.5) score += 20;
    if (riskFactors.behaviorAnomaly > 0.7) score += 10;
    const hours = riskFactors.timeSinceLastAuth / (1000 * 60 * 60);
    if (hours > 8) score += 10;
    if (hours > 24) score += 15;

    score *= { low: 0.7, medium: 1.0, high: 1.3, critical: 1.6 }[sensitivity] || 1.0;

    if (score >= 80) return { required: true, level: 'high', reason: 'high_risk_detected' };
    if (score >= 50) return { required: true, level: 'medium', reason: 'moderate_risk_detected' };
    if (score >= 25) return { required: true, level: 'low', reason: 'low_risk_detected' };
    return { required: false };
  }

  selectAuthenticationMethods(riskFactors, level) {
    const methods = [];
    if (level === 'high') {
      methods.push('hardware_token', 'biometric');
      if (riskFactors.deviceTrust < 30) methods.push('admin_approval');
    } else if (level === 'medium') {
      methods.push('totp', 'sms');
      if (riskFactors.locationRisk > 0.7) methods.push('email_verification');
    } else if (level === 'low') {
      methods.push('totp');
    }
    return methods;
  }
}

The service uses this balancing technique between critical resources and risks while keeping normal workflows intact when things look safe.

Security Monitoring

Security monitoring provides the observability layer that’s essential for detecting, analyzing, and responding to threats in real time. A strong system must log every authentication event, highlight anomalies, and allow for rapid and automated response to threats. This phase further builds trust by constantly evaluating access patterns and acting on them when signals of risk emerge.

Logging is visibility at its base. These days, every authentication attempt, be it successful, failed, or suspicious, needs to be logged with exhaustive context. This very information helps forensic analysis, alerting, and compliance reporting.

// Comprehensive authentication event logging
class AuthenticationLogger {
  async logAuthenticationEvent(eventType, userId, context, result) {
    const logEntry = {
      timestamp: new Date().toISOString(),
      eventType,
      userId,
      sessionId: context.sessionId,
      ipAddress: context.ipAddress,
      userAgent: context.userAgent,
      deviceFingerprint: context.deviceFingerprint,
      location: context.location,
      authenticationMethod: context.authMethod,
      result: result.success ? 'success' : 'failure',
      failureReason: result.failureReason,
      riskScore: result.riskScore,
      additionalFactorsRequired: result.stepUpRequired,
      processingTime: result.processingTime,
      correlationId: context.correlationId
    };

    // Store in multiple destinations for redundancy
    await Promise.all([
      this.writeToDatabase(logEntry),
      this.sendToLogAggregator(logEntry),
      this.updateRealTimeMetrics(logEntry)
    ]);

    // Trigger real-time alerts for critical events
    if (this.isCriticalEvent(logEntry)) {
      await this.triggerSecurityAlert(logEntry);
    }
  }

  isCriticalEvent(logEntry) {
    const criticalConditions = [
      logEntry.result === 'failure' && logEntry.failureReason === 'brute_force_detected',
      logEntry.riskScore > 80,
      logEntry.eventType === 'impossible_travel_detected',
      logEntry.eventType === 'account_takeover_suspected'
    ];

    return criticalConditions.some(condition => condition);
  }

  async generateSecurityReport(userId, timeRange) {
    const events = await this.getAuthenticationEvents(userId, timeRange);

    const analysis = {
      totalEvents: events.length,
      successfulLogins: events.filter(e => e.result === 'success').length,
      failedAttempts: events.filter(e => e.result === 'failure').length,
      uniqueDevices: new Set(events.map(e => e.deviceFingerprint)).size,
      uniqueLocations: new Set(events.map(e => e.location?.country)).size,
      averageRiskScore: events.reduce((sum, e) => sum + e.riskScore, 0) / events.length,
      timePatterns: this.analyzeTimePatterns(events),
      locationPatterns: this.analyzeLocationPatterns(events),
      devicePatterns: this.analyzeDevicePatterns(events)
    };

    return analysis;
  }
}

In the above code, the class logs detailed authentication events such as the approximate device and location from which it was initiated, the authentication methods used, and the risk score.

From a security perspective, it’s envisaged to generate security reports with the advantage of flagging critical events such as brute-force attempts or logins from suspicious geographies that can send real-time alerts.

Monitoring authentication events isn’t enough – the system must be able to interpret patterns and flag suspicious behavior. This detection system combines static rule-based checks with dynamic anomaly detection powered by machine learning. It identifies threats like brute-force attacks, credential stuffing, and unusual geographic access, then escalates them automatically for further action.

The following code performs real-time threat detection by analyzing recent authentication events and contextual data. Here's what it does, broken down clearly:

// Suspicious activity detection system
class SuspiciousActivityDetector {
  constructor() {
    this.detectionRules = this.initializeDetectionRules();
    this.mlModel = this.loadAnomalyDetectionModel();
  }

  async analyzeActivity(userId, recentEvents, context) {
    const suspiciousPatterns = [];

    // Rule-based detection
    const ruleViolations = await this.checkDetectionRules(userId, recentEvents);
    suspiciousPatterns.push(...ruleViolations);

    // ML-based anomaly detection
    const anomalies = await this.detectAnomalies(userId, recentEvents, context);
    suspiciousPatterns.push(...anomalies);

    // Threat intelligence correlation
    const threatMatches = await this.correlateThreatIntelligence(context);
    suspiciousPatterns.push(...threatMatches);

    if (suspiciousPatterns.length > 0) {
      await this.escalateSuspiciousActivity(userId, suspiciousPatterns);
    }

    return {
      suspicious: suspiciousPatterns.length > 0,
      patterns: suspiciousPatterns,
      riskScore: this.calculateSuspiciousActivityRisk(suspiciousPatterns)
    };
  }

  initializeDetectionRules() {
    return [
      {
        name: 'brute_force_detection',
        condition: (events) => {
          const failedAttempts = events.filter(e =>
            e.result === 'failure' &&
            Date.now() - new Date(e.timestamp).getTime() < 300000 // 5 minutes
          );
          return failedAttempts.length >= 5;
        },
        severity: 'high',
        action: 'temporary_lockout'
      },
      {
        name: 'credential_stuffing',
        condition: (events) => {
          const recentFailures = events.filter(e =>
            e.result === 'failure' &&
            Date.now() - new Date(e.timestamp).getTime() < 3600000 // 1 hour
          );
          const uniqueUsernames = new Set(recentFailures.map(e => e.username));
          return uniqueUsernames.size >= 10;
        },
        severity: 'medium',
        action: 'rate_limiting'
      },
      {
        name: 'suspicious_location_pattern',
        condition: (events) => {
          const locations = events.map(e => e.location?.country).filter(Boolean);
          const uniqueCountries = new Set(locations);
          return uniqueCountries.size >= 3 && events.length >= 5;
        },
        severity: 'medium',
        action: 'enhanced_verification'
      }
    ];
  }

  async detectAnomalies(userId, events, context) {
    const features = this.extractFeatures(events, context);
    const anomalyScore = await this.mlModel.predict(features);

    if (anomalyScore > 0.7) {
      return [{
        type: 'ml_anomaly',
        score: anomalyScore,
        features: features,
        description: 'Machine learning model detected anomalous behavior pattern'
      }];
    }

    return [];
  }
}

This class applies multiple techniques to detect threats. It first evaluates authentication history using static rules for brute-force attempts, large-scale credential reuse, or location anomalies. It then passes behavioral data through a trained ML model to spot subtle patterns missed by rules. If any suspicious pattern is detected, it returns a structured risk report and initiates escalation.

Automating Threat Response

Most times, systems respond in real-time. Automated threat response follows predefined actions and includes locking an account, alerting users, or blocking an IP, among others, when a high-risk event occurs.

// Automated threat response system
class AutomatedThreatResponse {
  constructor() {
    this.responsePlaybooks = this.initializeResponsePlaybooks();
    this.escalationPolicies = this.loadEscalationPolicies();
  }

  async processSecurityEvent(event) {
    const threatLevel = this.assessThreatLevel(event);
    const applicablePlaybooks = this.selectPlaybooks(event, threatLevel);

    const responses = [];
    for (const playbook of applicablePlaybooks) {
      const response = await this.executePlaybook(playbook, event);
      responses.push(response);
    }

    if (threatLevel === 'critical' || responses.some(r => !r.success)) {
      await this.escalateToHuman(event, responses);
    }

    return {
      event,
      threatLevel,
      responses,
      timestamp: new Date()
    };
  }

  initializeResponsePlaybooks() {
    return [
      {
        name: 'brute_force_response',
        triggers: ['brute_force_detected'],
        actions: [
          { type: 'temporary_lockout', duration: 900 },
          { type: 'rate_limiting', factor: 10 },
          { type: 'notify_user', method: 'email' },
          { type: 'log_security_event', level: 'high' }
        ]
      },
      {
        name: 'account_takeover_response',
        triggers: ['impossible_travel', 'behavior_anomaly_high'],
        actions: [
          { type: 'terminate_all_sessions' },
          { type: 'require_password_reset' },
          { type: 'notify_user', method: 'multiple' },
          { type: 'freeze_account', duration: 7200 }
        ]
      }
    ];
  }

  async executePlaybook(playbook, event) {
    const execution = {
      playbookName: playbook.name,
      eventId: event.id,
      actions: [],
      success: true
    };

    for (const action of playbook.actions) {
      try {
        const result = await this.executeAction(action, event);
        execution.actions.push(result);
        if (!result.success) {
          execution.success = false;
          break;
        }
      } catch (err) {
        execution.success = false;
        execution.error = err.message;
      }
    }

    return execution;
  }

  async executeAction(action, event) {
    switch (action.type) {
      case 'temporary_lockout':
        await this.lockoutUser(event.userId, action.duration);
        return { success: true, type: action.type };
      case 'notify_user':
        await this.notifyUser(event.userId, action.method, event);
        return { success: true, type: action.type };
      default:
        return { success: false, type: action.type, error: 'Unknown action' };
    }
  }
}

Here, the system uses playbooks – predefined actions to be taken in response to threats. For example, locks user from further brute-force attempts for some time and sends them an email notification. Freezing the account and ending all sessions are some reactive measures you can take if suspicious behavior indicates a takeover. These measures ensure fast and consistent action to mitigate damage even before humans can get involved.

Conclusion

Zero-trust authentication creates a strong line of distinction going against classic perimeter-based security. It must be painstakingly planned, implemented in layers, and constantly improved. This article offers a structured path, from basic MFA to intelligent behavioral monitoring and automated threat response.

Complementing the improvement of security, zero-trust promises better user experience, compliance readiness, and decreased incident risk. When organizations maintain a perpetual position of zero trust, we can see an actual positive impact on their ability to detect, prevent, and respond to threats in real time.

To have long-term success with this approach, you’ll need to continuously monitor your setup, perform periodic assessments, and be responsive to evolving attack patterns. Feedback loops and performance data are essential to keep the system secure yet user-friendly.

As threats grow more sophisticated, so must our defenses. ZTA provides a durable foundation – ready to evolve with emerging technologies like adaptive biometrics and AI-driven risk engines. Organizations investing in it today will be better equipped to meet tomorrow’s security and usability demands.

With the release of Swift 6 and its new Embedded Swift compilation mode, developers now have access to a modern, memory-safe, and performant alternative that’s tailored specifically for resource-constrained systems.

While languages like Rust have also emerged to address these issues, Embedded Swift brings the clarity and safety of Swift to microcontroller environments, without giving up on determinism, binary size, or hardware access.

This article introduces Embedded Swift and explores how it compares to traditional C/C++ development. We’ll cover its key features, programming and memory models, how to set up the toolchain for STM32 microcontrollers, and how to link Swift with existing C drivers.

Along the way, we’ll examine performance trade-offs, growing ecosystem support, and the broader industry movement toward memory-safe languages. As I hope you’ll see, Swift is a serious contender in the future of embedded development.

Prerequisites

To get the most out of this article, you should have a basic understanding of programming in Swift and C. Familiarity with embedded hardware platforms and firmware development concepts will also be helpful.

If you're new to embedded systems, consider reviewing this introductory guide to embedded firmware to build foundational knowledge before diving into Embedded Swift.

Scope

This article is intended as a practical introduction to Embedded Swift. It covers:

• An overview of Embedded Swift and its key language features

• Swift’s programming and memory model in an embedded context

• Setting up the Embedded Swift toolchain on macOS for STM32 microcontrollers

• Interoperability with C code and linking to existing low-level drivers

• A look at memory and instruction-level performance

• Future directions and use cases for Embedded Swift

Note that this article does not provide a full tutorial on the Swift language itself. While the primary focus is on STM32, similar principles apply to other supported platforms such as ESP32, Raspberry Pi Pico, and nRF52.

Table of Contents:

• What is Swift? What is Embedded Swift?

• Swift Programming Model

• Swift Memory Management

• Memory and Instruction Cycle Comparison

• How to Setup Embedded Swift

• C-Swift Linkages:

• Future Work

• Conclusion

What is Swift? What is Embedded Swift?

Swift is a modern programming language developed by Apple that combines the performance of compiled languages with the expressiveness and safety of modern language design. While Swift was originally created for iOS and macOS development, it has evolved into a powerful general-purpose language used in server-side development, systems programming, and increasingly, embedded systems.

Embedded Swift is a special compilation mode introduced in Swift 6 that brings the benefits of Swift to resource-constrained platforms like microcontrollers. It lets developers use a safe, high-level language while still producing compact, deterministic, and performant binaries suitable for embedded applications.

Key Features of Swift

Embedded Swift retains many of the powerful language features that make Swift an attractive alternative to C/C++ in embedded development:

Type Safety: Swift uses a strong static type system, which prevents many programming errors at compile time. Unlike C, where type mismatches can result in undefined behavior, Swift ensures all types are used correctly before code even runs.

Strict Type Checking: Swift doesn't allow implicit type conversions that could lose data or cause unexpected behavior. For example:

// This won't compile in Swift
let integer: Int = 42
let decimal: Double = 3.14
let result = integer + decimal  // Error: Cannot convert value of type 'Int' to expected argument type 'Double'

// You must be explicit about conversions
let result = Double(integer) + decimal  // Correct

Non-nullable Types by Default: In C, pointers can be null by default, which introduces risk. In Swift, variables cannot be nil unless explicitly marked as optionals:

var name: String = "John"
name = nil  // Compile error - String cannot be nil

var optionalName: String? = "John"
optionalName = nil  // This is allowed

Memory Safety via ARC (Covered in detail later):

Swift manages memory automatically using Automatic Reference Counting (ARC). Unlike manual memory management in C/C++, ARC handles object lifecycles efficiently without unpredictable garbage collection pauses. We'll cover ARC and its impact in embedded contexts in a dedicated section later.

Modern Syntax:
Swift's syntax is clean, consistent, and designed for readability. It supports modern paradigms including:

• Functional programming (map, filter, reduce)

• Generics (type-safe abstractions)

• Protocol-Oriented Programming (discussed in the next section)

These features allow you to write more expressive and maintainable code compared to procedural C or inheritance-heavy C++.

Performance:
Swift is designed to perform on par with C++ in many scenarios. Optimizations such as inlining, dead code elimination, and static dispatch help ensure that high-level abstractions don’t compromise performance. In embedded mode, Swift disables features like runtime reflection and dynamic dispatch to further reduce overhead.

To fully leverage Swift for embedded development, it's important to understand its programming model. Unlike C’s procedural approach or C++’s class-heavy design, Swift promotes protocol-oriented programming and composition, which offers both flexibility and safety in embedded system design.

Swift Programming Model

Swift embraces a multi-paradigm programming model that blends object-oriented, functional, and protocol-oriented programming, all underpinned by strong type safety and memory safety.

For embedded developers coming from C or C++, this model may feel different at first. But it provides a more modular and testable way to build complex systems, something especially valuable in embedded applications where hardware abstraction and strict reliability are critical.

Protocol-Oriented Programming (POP)

Swift emphasizes protocols over inheritance, encouraging developers to define behaviors through protocols and implement them using value types like struct and enum, rather than relying heavily on classes.

This philosophy favors composition over inheritance, allowing you to build complex functionality by combining smaller, well-defined components.

Key Concepts:

• protocol defines required behavior.

• Protocol extensions provide default behavior.

• Prefer value semantics using struct.

Example:

protocol Speakable {
    func speak()
}

extension Speakable {
    func speak() {
        print("Default sound")
    }
}

struct Dog: Speakable {
    func speak() {
        print("Woof!")
    }
}

Embedded Swift uses protocols with static dispatch. With static dispatch, the compiler knows the exact memory address of the function to call and can generate a direct jump instruction. There's no runtime lookup, no indirection, and no uncertainty.

Why POP Matters for Embedded Systems

First, you get flexible hardware extraction. Protocols make it easy to define interfaces for hardware components, allowing for mock implementations during testing or platform-specific variations.

Second, you have nice low overhead. Embedded Swift uses static dispatch for protocols, meaning there’s no runtime lookup, and calls are resolved at compile time for maximum performance.

Also, struct and enum types avoid heap allocations, making code more efficient and predictable in low-memory environments.

Now that we’ve explored how Swift’s programming model enables safer and more modular embedded code, let’s turn to another critical piece of the puzzle: memory management. Swift’s use of Automatic Reference Counting (ARC) replaces manual memory handling and offers important benefits, and tradeoffs, for embedded systems.

Swift Memory Management

One of Swift’s most impactful features, especially in the context of embedded systems, is its use of Automatic Reference Counting (ARC) for memory management. Unlike C/C++, where memory must be manually allocated and freed using malloc and free, Swift automates this process while maintaining deterministic performance.

This automation significantly reduces the risk of common memory-related bugs like leaks, dangling pointers, or use-after-free errors, all of which are notorious in low-level C code.

How ARC works

Swift supports ARC not only for the Cocoa Touch API's but for all APIs, providing a streamlined approach to memory management. Unlike garbage collection systems that can cause unpredictable pauses, ARC works deterministically at compile time and runtime to manage memory.

ARC automatically tracks and manages the lifetime of objects in memory based on how many references point to them.

• Reference Counting: Every object has a counter that tracks how many strong references point to it.

• Retain / Release: The compiler inserts retain and release calls automatically during assignment and deinitialization.

• Immediate Deallocation: When the reference count reaches zero, the object is deallocated immediately.

• Deterministic: Unlike garbage collectors, ARC doesn’t introduce unpredictable pauses or runtime scanning.

Swift offers multiple reference types to give you precise control over memory behavior and prevent cycles:

Strong References (default)

• Keeps the referenced object alive.

• Used in most cases.

class MotorController {
    var sensor: SensorData?  // Strong reference

    func updateReading(newData: SensorData) {
        self.sensor = newData  // Previous sensor data automatically deallocated
    }
}

Weak References

• Used to break reference cycles (especially in two-way object relationships).

• Automatically becomes nil when the referenced object is deallocated.

class Device {
    var controller: MotorController?

    deinit {
        print("Device deallocated")
    }
}

class MotorController {
    weak var device: Device?  // ← Weak reference breaks the cycle

    deinit {
        print("MotorController deallocated")
    }
}

func breakCycle() {
    let device = Device()
    let controller = MotorController()

    device.controller = controller
    controller.device = device  // ← This is now a weak reference

    // When this function ends, both objects are properly deallocated
}

breakCycle()
// Output:
// Device deallocated
// MotorController deallocated

Unowned References

• Non-optional version of weak.

• Assumes the object will never be deallocated while still in use.

• More lightweight than weak, but unsafe if misused.

class SensorSystem {
    unowned let controller: MotorController  // unowned reference

    init(controller: MotorController) {
        self.controller = controller
    }
}

class MotorController {
    var sensorSystem: SensorSystem?

    func setupSensors() {
        sensorSystem = SensorSystem(controller: self)
    }

    deinit {
        print("MotorController deallocated")
    }
}

func testUnowned() {
    let controller = MotorController()
    controller.setupSensors()
    // sensorSystem deallocates before controller ends
}

testUnowned()
// Output: MotorController deallocated

ARC Overhead in Embedded Systems

While ARC provides safety benefits, it does introduce some overhead compared to manual memory management:

Memory Overhead:

ARC-managed class instances in Swift typically include an additional 4 or 8 bytes to store reference count metadata, depending on the system architecture, 4 bytes on 32-bit systems and 8 bytes on 64-bit systems. This metadata allows the runtime to track how many active references exist to a given object and deallocate it when no references remain. When developers use weak or unowned references, the memory footprint increases further. These references require additional data structures, such as side tables or tracking mechanisms, to manage object liveness and cleanup. In the case of weak references specifically, Swift maintains zeroing weak reference tables that automatically null out pointers once the referenced object is deallocated, ensuring memory safety.

CPU Overhead:

ARC introduces some runtime overhead due to retain and release operations, which are inserted automatically during reference assignments. These operations involve incrementing or decrementing the reference count and are especially common in code that passes objects between functions or stores them in collections. To ensure thread safety, these updates are typically implemented using atomic operations, which add further instruction cycles. In complex object graphs, ARC may also engage in cycle detection and cleanup through the use of weak references to prevent memory leaks caused by strong reference cycles. While Swift's ARC provides deterministic and efficient memory management, it does so with both memory and CPU costs that developers should consider carefully, especially in performance-critical embedded systems.

Type Safety and Error Prevention

Swift's type system prevents many common errors that plague C/C++ programs:

• Buffer Overflows: Swift arrays are bounds-checked, preventing buffer overflow vulnerabilities that are common in C.

• Null Pointer Dereferences: Swift's optional types make null pointer dereferences impossible at compile time.

• Use After Free: Swift's ownership model prevents use-after-free errors that can cause crashes or security vulnerabilities.

Now that we’ve covered Swift's memory model and ARC behavior, let’s explore how it compares to C in terms of memory usage and instruction cycles, a crucial aspect when evaluating Embedded Swift for real-world deployment.

Memory and Instruction Cycle Comparison

Understanding the performance characteristics of Swift versus C is essential for embedded systems, where every instruction cycle and byte of memory matters. While Swift brings advantages like safety and expressiveness, these benefits come with certain trade-offs in terms of memory usage and runtime behavior that embedded developers must evaluate carefully.

Memory Management:

Swift uses Automatic Reference Counting (ARC) to manage memory. ARC tracks the number of references to each object and deallocates it when no references remain. This eliminates the need for explicit free() calls but introduces overhead.

C, in contrast, uses manual memory management. Developers allocate memory using malloc and release it using free, or rely on the stack for most short-lived data.

The table below provides the memory management comparison between Swift and C:

Swift ensures safety through deterministic cleanup and predictable memory usage. But this comes at the cost of added memory and CPU overhead.

C’s approach offers complete control over memory layout and minimal runtime cost, but increases the risk of memory leaks and fragmentation without disciplined practices.

Instruction Cycle Analysis

The safety features in Swift, such as bounds checking, optional unwrapping, and ARC updates, translate into additional CPU instructions. While this can impact performance, the Swift compiler is aggressive about optimization in release builds. For example, inlining and ARC elision can remove much of the overhead in performance-critical paths.

C has no built-in safety checks, allowing it to generate highly efficient, predictable code. Developers can even use inline assembly for tight control over performance.

The table below provides the instruction cycle comparison between Swift and C:

Although Swift inserts extra instructions for safety, much of this cost can be mitigated through compiler optimization.

C has no such features by default, making it ideal for applications where performance must be tightly controlled and the developer is willing to take full responsibility for safety.

Instruction Count Comparison: Swift vs C Loop Performance

When evaluating Swift and C for embedded use, it's helpful to analyze instruction-level performance on basic operations, such as a loop that processes an array of floating-point numbers. This gives us a concrete sense of the computational cost of each language's safety and abstraction features.

Let’s consider a simple example: summing an array of Float values and returning the average. In Swift, the code uses a high-level for-in loop over an array:

Simple loop performance:

// Swift loop with safety checks
func processData(_ data: [Float]) -> Float {
    var sum: Float = 0.0
    for value in data {  // Iterator with bounds checking
        sum += value     // Safe arithmetic
    }
    return sum / Float(data.count)  // Safe division
}
// Estimated: ~8-10 instructions per iteration

Although elegant and safe, this loop includes several safety mechanisms:

1. Bounds checking on every array access

2. Reference counting if data is passed as a reference type

3. Overflow protection in debug mode

4. Optional handling or runtime checks if data might be empty

These checks introduce runtime overhead, resulting in an estimated 8–10 instructions per iteration on most platforms (depending on optimization level and target architecture). In release builds, Swift aggressively inlines and strips redundant checks, but some level of abstraction cost remains, especially compared to raw memory access in C.

Now, compare that to its equivalent in C:

// C loop without safety checks
float process_data(float* data, int count) {
    float sum = 0.0f;
    for (int i = 0; i < count; i++) {  // Direct pointer arithmetic
        sum += data[i];                // Direct memory access
    }
    return sum / count;  // Direct division (no safety check)
}
// Estimated: ~4-5 instructions per iteration

This version performs direct memory access with pointer arithmetic, no bounds checks, and no type safety. The C code is lower-level, with fewer runtime checks, and compiles down to just 4–5 instructions per iteration, depending on the target CPU and compiler flags. It is lean and fast, ideal for cycles-per-instruction-critical scenarios.

The table below shows the comparison of single loop performance between Swift and C:

Now that we’ve compared Swift and C in terms of memory and cycle costs, let’s move into the practical side: how to set up Embedded Swift on an STM32 platform and get started with real-world development.

How to Setup Embedded Swift

In this section, we'll walk through how to configure and use Embedded Swift for development on STM32 microcontrollers. STM32 is a popular family of ARM Cortex-M–based microcontrollers, commonly used in industrial, consumer, and IoT applications.

Prerequisites

Required Software:

• Swift Development Snapshot (includes the Embedded Swift toolchain)

• Swiftly - Easiest way to manage and install swift toolchains

• Swiftc - Swift Compiler command-line tool

• Python3 - Required to run scripts to convert Mach-O to binary files

• Git (to clone sample repositories) like https://github.com/swiftlang/swift-embedded-examples

• A Unix-like development environment (macOS is currently best supported)

Target Hardware: This guide focuses on STM32 microcontrollers, which are widely used in embedded applications and have excellent community support.

This guide walks you through the full setup process, from installing the required Swift toolchain to flashing the final binary onto your board. We’ll begin by installing the Swift Development Snapshot using Swiftly, a simple command-line utility for managing Swift toolchains. From there, we’ll configure the build system, set up the correct board variant, customize the build script, and compile the Swift and C source code into a binary. Finally, we’ll flash the firmware onto the STM32 using standard tools

Install Swift Development Snapshot

The easiest way to install and manage Embedded Swift toolchains is by using the swiftly tool, which simplifies downloading and using Swift snapshots.

macOS Installation:

The below steps will help install the Swift embedded toolchain:

# Using Swiftly (Recommended)
curl -O https://download.swift.org/swiftly/darwin/swiftly.pkg
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory
~/.swiftly/bin/swiftly init --quiet-shell-followup
source "${SWIFTLY_HOME_DIR:-$HOME/.swiftly}/env.sh"

# Install and use development snapshot
swiftly install main-snapshot
swiftly use main-snapshot

# Verify installation
swift --version

You can clone this Github example repository:

git clone https://github.com/swiftlang/swift-embedded-examples.git 
cd swift-embedded-examples/projects/stm32-blink

The stm32-blink contains:

• Swift code that toggles GPIOs

• A C startup file with vector table

• A build.sh script that uses swiftc, clang, and a custom linker setup

Setup the STM32 Board

Tell the build script which STM32 board is being used:

export STM_BOARD=STM32F746G_DISCOVERY

You can add your own board variant by defining the appropriate memory map and compiler flags in the script.

Modify build.sh (Optional)

Ensure the script correctly locates the following:

• swiftc: should point to the toolchain you installed with Swiftly

• clang: can be macOS’s default Clang

• libBuiltin.a, crt0.s, and macho2bin.py: used to provide minimal runtime support and convert output to flashable binaries

If needed, update these paths:

SWIFT_EXEC=${SWIFT_EXEC:-$(swiftly which swiftc)}
CLANG_EXEC=${CLANG_EXEC:-$(xcrun -f clang)}
PYTHON_EXEC=${PYTHON_EXEC:-$(which python3)}

Ensure the linker flags match your target’s flash and RAM sizes.

Build and Flash the Project:

Run:

./build.sh

This compiles Swift and C code, links them, and produces a blink.bin file.

If successful, you’ll see:

.build/blink.bin  # ready to flash Step 6: Flash the Firmware to STM32

Use ST-Link tools or openocd to flash your board. Example using st-flash:

brew install stlink
st-flash write .build/blink.bin 0x8000000

You should now see an LED blinking.

Here’s a more detailed step by step approach to writing a bare metal code on STM32. For comprehensive installation guides covering other platforms (Raspberry Pi Pico, ESP32, nRF52), detailed IDE configuration, troubleshooting, and advanced examples, you can check out the official documentation:

• Complete Setup Guide: Install Embedded Swift

• Platform Examples: Swift Embedded Examples Repository

• Getting Started Tutorial: Embedded Swift on Microcontrollers

Now that we’ve set up Embedded Swift and explored how to build and run an example project, let’s look at a critical real-world scenario: interfacing Swift with low-level C drivers.

C-Swift Linkages

In many embedded projects, low-level hardware drivers are written in C because of its close-to-metal control and widespread ecosystem support. Embedded Swift supports seamless interoperability with C, which lets you reuse existing C libraries and drivers, write hardware control logic in C, and implement higher-level application logic in Swift.

This hybrid model lets you combine Swift’s safety and productivity with C’s hardware-level control, with no runtime overhead or object translation.

Let’s walk through an example where a low-level sensor driver is implemented in C and the application logic is written in Swift.

C Header File (sensor_driver.h):

This C header file defines the public interface for a low-level sensor driver. It includes standard fixed-width integer types and declares four functions:

• sensor_init(): Initializes the hardware sensor

• sensor_read_temperature() and sensor_read_humidity(): Read raw sensor values

• sensor_delay_ms(): Delays execution for a given number of milliseconds

This interface acts as a bridge between Swift and C. Swift will link to these functions by name, no wrappers or bindings required.

#ifndef SENSOR_DRIVER_H
#define SENSOR_DRIVER_H

#include <stdint.h>

// Low-level sensor driver functions
void sensor_init(void);
uint32_t sensor_read_temperature(void);
uint32_t sensor_read_humidity(void);
void sensor_delay_ms(uint32_t milliseconds);

#endif

C Implementation (sensor_driver.c):

This implementation assumes the sensor is memory-mapped at a fixed address (0x40001000). Each register, temperature, humidity, and control, is accessed by offset from that base address.

The sensor_init() function writes 0x01 to the control register, presumably enabling or starting the sensor hardware.

The sensor_read_temperature() method and sensor_read_humidity() method reads from memory-mapped registers and return the raw ADC values from the sensor.

The sensor_delay_ms() method performs a simple busy-wait loop using nop (no-operation) instructions to approximate a delay. This is suitable for short, coarse-grained delays in bare-metal contexts.

#include "sensor_driver.h"

// Hardware register addresses
#define SENSOR_BASE_ADDR    0x40001000
#define TEMP_REG_OFFSET     0x00
#define HUMIDITY_REG_OFFSET 0x04
#define CONTROL_REG_OFFSET  0x08

void sensor_init(void) {
    // Initialize sensor hardware
    volatile uint32_t* control_reg = (volatile uint32_t*)(SENSOR_BASE_ADDR + CONTROL_REG_OFFSET);
    *control_reg = 0x01; // Enable sensor
}

uint32_t sensor_read_temperature(void) {
    volatile uint32_t* temp_reg = (volatile uint32_t*)(SENSOR_BASE_ADDR + TEMP_REG_OFFSET);
    return *temp_reg;
}

uint32_t sensor_read_humidity(void) {
    volatile uint32_t* humidity_reg = (volatile uint32_t*)(SENSOR_BASE_ADDR + HUMIDITY_REG_OFFSET);
    return *humidity_reg;
}

void sensor_delay_ms(uint32_t milliseconds) {
    // Simple delay implementation
    for (uint32_t i = 0; i < milliseconds * 1000; i++) {
        __asm__("nop");
    }
}

Swift Code Using C Driver:

To use these C functions from Swift, you declare them using @_silgen_name, which tells the Swift compiler to link directly to these symbol names at runtime.

The SensorController class encapsulates sensor-related logic. In its init() method, it calls the sensor_init() function defined in C to initialize the sensor hardware.

The readSensors() method reads the raw values from the C driver, converts them into human-readable units using helper functions, stores them internally, and returns the processed values.

The convertTemperature() and convertHumidity() conversion methods apply a basic linear formula to turn raw ADC values into temperature in Celsius and humidity in percentage, respectively. These formulas would be based on the specific sensor’s datasheet.

The checkThresholds() method applies simple threshold logic, a good example of where Swift’s readability and type safety shine. You could easily expand this logic to include error bounds, state machines, or alerts.

// Import C driver functions

/*
These declarations match the C function signatures exactly. 
They allow Swift to invoke the C functions as if they were native Swift functions 
— with zero overhead.
*/
@_silgen_name("sensor_init")
func sensor_init()

@_silgen_name("sensor_read_temperature")
func sensor_read_temperature() -> UInt32

@_silgen_name("sensor_read_humidity")
func sensor_read_humidity() -> UInt32

@_silgen_name("sensor_delay_ms")
func sensor_delay_ms(_ ms: UInt32)

// Swift sensor controller using C driver
class SensorController {
    private var lastTemperature: Float = 0.0
    private var lastHumidity: Float = 0.0

    init() {
        // Initialize the C driver
        sensor_init()
    }

    func readSensors() -> (temperature: Float, humidity: Float) {
        // Read raw values from C driver
        let rawTemp = sensor_read_temperature()
        let rawHumidity = sensor_read_humidity()

        // Convert raw values to meaningful units in Swift
        let temperature = convertTemperature(rawValue: rawTemp)
        let humidity = convertHumidity(rawValue: rawHumidity)

        // Store for comparison
        lastTemperature = temperature
        lastHumidity = humidity

        return (temperature: temperature, humidity: humidity)
    }

    private func convertTemperature(rawValue: UInt32) -> Float {
        // Convert raw ADC value to Celsius
        return (Float(rawValue) * 3.3 / 4095.0 - 0.5) * 100.0
    }

    private func convertHumidity(rawValue: UInt32) -> Float {
        // Convert raw ADC value to percentage
        return Float(rawValue) * 100.0 / 4095.0
    }

    func checkThresholds() -> Bool {
        // Swift logic for threshold checking
        let tempThreshold: Float = 25.0
        let humidityThreshold: Float = 60.0

        return lastTemperature > tempThreshold || lastHumidity > humidityThreshold
    }
}

// Main application loop
func main() -> Never {
    let sensorController = SensorController()

    while true {
        // Read sensors using Swift controller with C driver
        let readings = sensorController.readSensors()

        // Process data with Swift's type safety and expressiveness
        if sensorController.checkThresholds() {
            print("Warning: Temperature: \(readings.temperature)°C, Humidity: \(readings.humidity)%")
        } else {
            print("Normal: Temperature: \(readings.temperature)°C, Humidity: \(readings.humidity)%")
        }

        // Delay using C driver function
        sensor_delay_ms(1000) // 1 second delay
    }
}

The func main() is the main event loop standard for embedded systems. It creates the sensor controller, reads sensor data in a loop, checks thresholds, and prints results accordingly. The loop includes a delay (via the C driver) to avoid hammering the sensor continuously.

In an actual embedded context, instead of using print(), you might blink an LED, send UART messages, or log data to memory.

With Embedded Swift and C now working together, let’s explore what lies ahead. The next section outlines ongoing improvements, emerging use cases, and research directions that are shaping the future of Embedded Swift.

Future Work

Embedded Swift is still a young but rapidly evolving technology. Its modern language features, type safety, and performance make it an attractive option for embedded development, and ongoing work is expanding its capabilities, reach, and ecosystem.

Ongoing Improvements

Compiler Optimizations: The Swift compiler team is actively improving code generation for embedded targets, including:

• Reducing binary size

• Minimizing ARC overhead

• Improving static dispatch performance

Hardware Support: Embedded Swift can target a wide variety of ARM and RISC-V microcontrollers, which are popular for building industrial applications. Support for additional architectures is being developed.

Tooling Enhancements: Tooling support for Embedded Swift is still evolving, but several community-driven and open-source efforts are making development more accessible:

• Build Systems: The Swift Embedded Working Group provides example projects that adapt Swift Package Manager (SwiftPM) for cross-compilation. Custom linker scripts and build helpers are available for platforms like STM32 and nRF52.

• Debugging Support: Developers can debug Embedded Swift programs using existing tools like GDB or OpenOCD, provided the build includes appropriate debug symbols. While not yet officially streamlined, this approach enables step-through debugging on real hardware.

• IDE Integration: There is no official IDE support yet, but some developers use VSCode with Swift syntax highlighting and external build tasks. These setups are still manual but serve as early prototypes for embedded workflows.

Emerging Use Cases

There are a number of emerging use cases for embedded Swift. For example, Swift’s memory safety, type guarantees, and protocol-oriented design make it ideal for secure and scalable IoT devices, especially where firmware bugs could affect user safety or privacy.

The automotive sector is also exploring Swift for infotainment systems, driver assistance features, and safety-critical logic (where deterministic execution and safety matter).

Swift’s expressive syntax and compile-time safety make it suitable for industrial automation – think real-time control loops, sensor fusion systems, and edge devices in smart manufacturing.

It’s also useful for medical devices, as it aligns well with strict medical regulations around memory safety, type guarantees, and predictable resource usage.

Community and Ecosystem

Open Source Projects

The Swift Embedded working group maintains example repositories showcasing how to use Embedded Swift on microcontrollers such as STM32, nRF52, and ESP32. Early-stage libraries for UART, GPIO, and basic peripherals are emerging, though the ecosystem is still young compared to C or Rust.

Learning Resources

While Embedded Swift is not yet widely taught in formal curricula, community tutorials and exploratory projects (for example, Swift for Arduino) are lowering the barrier for hobbyists and independent learners. As tooling matures, educational adoption is likely to follow.

Industry Interest

Embedded Swift is beginning to draw attention from developers and companies looking for safer, more maintainable alternatives to C. Although large-scale adoption remains limited, use cases like rapid prototyping, IoT development, and internal experimentation are gaining traction.

Conclusion

Embedded Swift represents a major step forward in embedded programming. By combining the power and safety of Swift with the low-level control needed for microcontrollers, it offers an exciting alternative to traditional C and C++ development.

While C will remain essential for hardware-level programming and performance-critical paths, Swift brings compelling advantages to many embedded scenarios:

• Memory safety: Swift eliminates entire categories of bugs such as buffer overflows, use-after-free, and null pointer dereferencing.

• Type safety: Many logic errors are caught at compile time, long before they can cause runtime failures.

• Modern language features: Developers can use functional paradigms, generics, and protocol-oriented design even in embedded code.

• C interoperability: Swift works seamlessly with existing C libraries, allowing gradual adoption without rewriting low-level drivers.

• Developer productivity: Clear syntax, automatic memory management, and strong tooling lead to faster development and easier maintenance.

Government and regulatory bodies are increasingly encouraging or mandating the use of memory-safe programming languages to reduce vulnerabilities in critical software systems. For example:

• In 2022, the U.S. National Security Agency (NSA) recommended moving away from unsafe languages like C/C++ for new software projects, promoting memory-safe alternatives.

• In June 2025, the NSA and CISA released a joint Cybersecurity Information Sheet titled “Memory Safe Languages: Reducing Vulnerabilities in Modern Software Development”, which emphasized that memory safety flaws remain a persistent risk, and organizations should develop strategies to adopt memory-safe programming languages in new systems.

• The U.S. Cybersecurity and Infrastructure Security Agency (CISA) and NIST have echoed similar guidance in the context of national cybersecurity.

While these documents do not mention Swift explicitly, Swift's strong type system, ARC-based memory model, and compile-time safety guarantees align closely with the goals outlined in these recommendations. As such, it offers a practical, developer-friendly path toward safer embedded development.

Swift may not be the right fit for every embedded system. In applications where every byte of memory or instruction cycle is critical, real-time guarantees are hard requirements, or toolchain maturity is essential (for example, RTOS integration, static analyzers), C or Rust may still be preferred.

But in many modern embedded applications, especially those involving rapid prototyping, fast product iteration, safety-critical or maintainable firmware, and interoperability with existing C codebases, Swift offers a highly productive and safe development experience.

Embedded Swift is still maturing, but its momentum is undeniable. With ongoing compiler work, community-driven examples, and growing interest from developers, it’s poised to play a major role in the future of embedded systems.

Whether you're building an IoT device, a piece of industrial equipment, or a proof-of-concept wearable, Swift can help you write safer, more expressive firmware, without giving up performance or control.

Swift can be especially powerful during the prototyping phase, when the primary goal is to validate functionality quickly and safely. And with its increasing support for multiple hardware platforms, it offers a strong foundation for bringing modern software development practices to the embedded world.

You’ve been grinding through tutorials. You've built a portfolio site, maybe deployed a few projects on GitHub. And now you're trying to land a job or join a team.

Then the interviews start.

Suddenly, people ask:

• "Are you familiar with Agile?"

• "Have you worked in a Scrum environment?"

• "What’s your experience with sprints?"

Cue the imposter syndrome. Because no one teaches this stuff in JavaScript 101.

This guide is for you.

I’ll help make the Scrum process – a very common way developers work together – make actual sense. I’ll walk you through the basics, but also tell you what developers actually do, how standups feel when you're new, and what’s expected of you when you’re no longer coding in a vacuum.

Let’s break it down.

Here’s what we’ll cover:

• What Even Is Scrum?

• The Three Roles in Scrum (and Who Does What)

• The Scrum Rhythm: What a Sprint Actually Looks Like

• Who attends the Ceremonies:

• Standups: Where You Talk Like a Human, Not a Robot

• Sprint Planning

• What’s a User Story and Why Does It Sound Like a Children’s Book?

• What Counts as “Done”? Definition of Done and Why It’s Important

• Demos, Retros, and Saying the Hard Things

• Tools You Might Encounter

• If You’re Preparing for a Job, Here’s What You Can Do

• Final Thoughts

What Even Is Scrum?

Scrum is not a tool. It’s not a software. It’s not some elite thing only PMs care about.

It’s a lightweight framework that helps software teams build things incrementally, together, in short focused cycles called sprints.

Scrum is used by everyone from FAANG teams to indie dev shops because it helps:

• Keep teams aligned

• Deliver working software fast

• Course-correct often

• Spot problems early (before they go nuclear)

It’s the opposite of the old-school “build for a year and pray it works” model.

The Three Roles in Scrum (and Who Does What)

Scrum officially defines three roles. Here's what that means in practice:

1. Product Owner (PO)

Think: Vision-holder. They decide what the team builds and why. A product owner:

• Writes user stories (think of these as feature requests written from a user’s point of view)

• Prioritizes the work

• Clarifies what success looks like

• Says “yes” or “not yet” to features

2. Scrum Master (SM)

Think: Air-traffic controller meets therapist. They make sure the process works. The are master Facilitators, like between Dev and PO’s. A Scrum Master:

• Facilitates meetings

• Removes blockers (“Your AWS access is stuck? I’ll escalate it.”)

• Coaches the team on Scrum practices

• Doesn’t manage people – manages flow

3. Developers (YOU!)

Think: Builders. You write code, test it, ship it, fix it, and improve it. You also:

• Break down stories into tasks

• Pick up work from the team board (like Jira or Trello)

• Communicate progress

• Demo what you’ve built at the end of the sprint

You might also work with designers, testers, or DevOps folks – but within Scrum, you’re all “developers” building a product together.

The Scrum Rhythm: What a Sprint Actually Looks Like

Image Source: https://www.invensislearning.com/blog/what-are-scrum-ceremonies/

Understanding the Scrum Cycle

So, what does it actually look like when a team uses Scrum to build software?

Let’s walk through a full sprint – not just the buzzwords, but what really happens when a group of humans tries to plan, build, review, and improve together. Think of this as your backstage pass to the rhythm of modern teamwork.

📦 Step 1: Build and Refine the Product Backlog

Before any coding starts, the team needs to agree on what they might build – not just this week, but in the near future too.

That’s where the Product Backlog comes in. This is a big, running list of everything the product might need – features, bug fixes, improvements, ideas, and maybe a few wild dreams. It’s like the wishlist for the product, but more organized (ideally).

The Product Owner is responsible for maintaining and prioritizing this list. They decide what’s most important to work on based on customer needs, business goals, and feedback.

But the PO doesn’t do this in isolation. Enter the Backlog Refinement meeting.

In these sessions, the Scrum Team – that’s the PO, the Scrum Master (SM), and the Developers – come together to:

• Review the most important upcoming items

• Clarify any vague or confusing parts of each task

• Break big items down into smaller, buildable pieces called user stories

• Estimate effort (how much time or complexity is involved for each story)

This meeting makes sure the team isn’t caught off guard in the sprint – that they understand the work ahead and can actually start sprinting when the time comes.

🧭 Step 2: Sprint Planning – What Are We Building This Time?

Now that we’ve got a solid backlog, it’s time to pick what to build right now.

At the start of each sprint (which typically lasts 1 to 4 weeks), the team holds a Sprint Planning meeting. This meeting sets the stage for the entire sprint – it’s like the huddle before the big game.

In Sprint Planning, the team:

• Reviews the top items from the backlog

• Discusses what can realistically be completed based on their availability and capacity

• Chooses a handful of these stories to commit to

• Defines a Sprint Goal – a simple statement that captures the purpose of this sprint

For example, the Sprint Goal might be:
🎯 “Allow users to reset their passwords.”

Every user story chosen should contribute to that goal. The collection of these stories becomes the Sprint Backlog – basically, the to-do list for the sprint.

So when we say:

“The team selects an ordered list of user stories to comprise the Sprint Backlog for the next sprint, which will be achievable to satisfy the Sprint Goal...”

We’re really just saying:
👉 “Pick a realistic number of important tasks that, if completed, will help us hit our target for the sprint.”

Not too vague. Not too ambitious. Just achievable and focused.

☀️ Step 3: Daily Standups – Stay in Sync

Now the sprint is underway! But how does everyone stay aligned and avoid working in silos?

That’s where the Daily Standup comes in. Every day – usually in the morning – the team has a quick check-in (about 15 minutes) where each person answers three questions:

1. What did I do yesterday?

2. What am I working on today?

3. Is anything blocking me? (that is, am I stuck?)

Example:

“Yesterday I set up the login API integration. Today I’ll work on the UI validation. I’m blocked on getting access to the staging database — may need help.”

These standups keep the team in sync and surface blockers early so they can be addressed quickly. They’re not about micromanaging or showing off. They’re about visibility and support.

📉 What’s a Sprint Burndown Chart?

You might hear your team mention a “burndown chart.” No, this isn’t about things going down in flames (hopefully).

A Sprint Burndown Chart is a graph that shows how much work is left in the sprint – day by day.

• The y-axis is the amount of work remaining (often measured in story points or tasks)

• The x-axis is the number of days left in the sprint

The line should ideally trend downward as work gets completed – hence “burning down.” If it flattens out or slopes up, that’s a red flag that the team might be stuck, behind schedule, or not updating the board.

Think of it as a visual heartbeat of the sprint. You can learn more via a practical example in this video.

🖥️ Step 4: Sprint Review – Show What You’ve Built

At the end of the sprint, the team holds a Sprint Review (also called a “demo”). This is where you show what was actually built during the sprint.

• The Developers demo working features – live, not just screenshots

• The Product Owner reviews whether the Sprint Goal was achieved

• Stakeholders may ask questions, give feedback, or suggest tweaks

This meeting isn’t just for show – it’s a feedback loop. It helps the team validate that what they built is useful, usable, and meets expectations. If changes are needed, those get added to the backlog for future sprints.

🔍 Step 5: Sprint Retrospective – Look Back to Move Forward

Once the review is done, the team shifts focus from what they built to how they worked together.

Enter the Sprint Retrospective – a meeting to reflect on the process, not the product.

The team discusses:

• ✅ What went well

• ❌ What didn’t go so well

• 🔁 What could be improved next time

This isn’t about pointing fingers. It’s about learning, adapting, and continuously improving how the team collaborates.

The Scrum Master often facilitates this meeting and helps turn feedback into action items for the next sprint. For example:

“We underestimated testing time. Next sprint, let’s budget for QA earlier.”

The best teams take retros seriously. Why? Because even if your code is perfect, your process needs tuning too – and small process changes often lead to big gains.

♻️ Scrum Is a Loop

Here’s the rhythm:

1. Plan the sprint

2. Check in daily

3. Build and demo the product

4. Reflect and improve

Then do it all over again – with slightly better coordination and slightly more trust each time.

It’s not about being fast. It’s about being intentional, consistent, and collaborative.

Example Sprint

Let’s say, for example, that your team does 4-week sprints. (Keep in mind that Sprints can differ by team, nature of product, release cycles, and so on.)

Here’s the rough beat:

Scrum works in loops. Every 2-4 weeks (depending on your cadence and sprint cycle), your team should have working, demo-able software to show for it – even if it’s small.

And no, it’s not about “speed.” It’s about consistency, communication, and collaboration.

Who attends the Ceremonies:

Now lets dive deeper and understand practically how each of these ceremonies work:

Standups: Where You Talk Like a Human, Not a Robot

So how does the team actually stay connected day to day? That’s where standups come in.

Every morning, your team meets briefly – usually on Zoom or in a circle – and you answer 3 questions:

1. What did I work on yesterday?

2. What will I work on today?

3. What’s blocking me? Any impediments?

Example:

"Yesterday I cleaned up the signup validation logic. Today I’m working on the email verification flow. I’m stuck on SendGrid config – might need help setting up credentials."

It’s not about impressing anyone. It’s about keeping everyone in sync. Some days you’ll say, “I spent the whole day debugging a CSS bug that turned out to be a semicolon.” That’s okay.

How does it work?

The Scrum Master gathers everyone in a huddle room, the PO and Dev Team included, and opens the the Standup. They are the facilitator of the ceremony. Everyone gets a chance to answer the 3 questions above (usually about 2-5 minutes each). It’s not a full report – it’s quick. When one person is done, they pass it on to someone else.

This ensures there is team cohesion and transparency.

Here is a video example of a standup.

Sprint Planning

The goal of the planning meeting is to answer the questions “What are we going to work on, and how are we going to do it?” It is critical for the team to have a shared goal and a shared commitment to this goal before beginning this ceremony.

Participants should:

• Measure growth

• Sync with the Scrum Master

• Sync with the Product Owner

Sprint planning happens just before the sprint starts, and usually lasts for an hour or two. In this meeting, the team goes over a collection of user stories and discuss, plan, measure, and prioritize. This is where they decide what is going to be in scope for their upcoming sprint cycle.

The Product Owner will have a prioritized view of things in the backlog. They work with the team on each object or customer experience. Together, as a group they go through and make calculations, deciding to what they can commit.

What’s a User Story and Why Does It Sound Like a Children’s Book?

So you might be wondering: how do you know what to work on? What to build? So much work, so little time? Thats where user stories come in.

In Scrum, teams don’t just write vague tasks like “code the login.” Instead, they write user stories – short, human-centered feature descriptions that describe what the user needs, why they need it, and what success looks like.

Here’s an example:

As a user, I want to be able to reset my password, so I can access my account if I forget it.

User stories are the scaffolding of teamwork. They’re written with empathy, not just efficiency. And each one comes with acceptance criteria – a checklist that clarifies what “done” actually means:

• A “Forgot Password” link is visible

• Clicking it shows a form

• An email gets sent with a reset link

Once a story is agreed upon, developers break it down into tasks, like “build form,” “hook into backend,” or “handle email validation.” It’s collaborative, not prescriptive. And user stories have priority so you know what’s the most important and what’s the least.

A helpful rule of thumb many teams use is the Gherkin-style "Given–When–Then" format:

• Given some initial context

• When an event occurs

• Then a specific outcome should happen

This ensures that everyone – devs, testers, and product owners – shares the same understanding of behavior and expectations.

Here is a great video example thats outlines how to draft effective and powerful user stories.

What Counts as “Done”? Definition of Done and Why It’s Important

Now you might be wondering – how do I know when a task is done and can be closed out?

The Definition of Done is a type of documentation in the form of a team agreement. The Definition of Done identifies the conditions that need to be achieved in order for the product to be considered done (as in potentially shippable).

This is how we know that we "did the thing right". Meaning, we built the correct level of quality into the product. The Definition of Done is not the same as the acceptance criteria, which are written by the product owner to help us know we did the "right thing".

Every team has a Definition of Done – it’s not just “I pushed code.” It could mean:

• Code is written

• Reviewed by a peer

• Merged into main

• Tested on staging

• Possibly deployed

This clarity keeps teams honest and accountable. No “it works on my machine” energy here. The DoD sets a quality bar. It prevents ambiguity, rework, and “it works on my machine” moments. When every card on the board passes the same finish line, teams move faster – and trust each other more.

Everyone should know what done is in a team. Either its Done as per DoD standards or its not.

Here is a beautiful video highlighting the impotence of DoD.

Demos, Retros, and Saying the Hard Things

Once you’ve built the product, then comes demos (showcasing your work) and retros (analysis as a team on what when well and what areas to improve on).

In the retro, everyone’s encouraged to speak up:

• What went well?

• What didn’t?

• What should we try next time?

Example:

“We missed a lot of stories because we didn’t account for testing time. Maybe we buffer next sprint with fewer tasks.”

The goal is not to blame – it’s to improve. Over time, this feedback loop becomes gold. The Scrum Master usually facilitates, collects feedback (via tools like Parabol, Miro, or sticky notes), and helps turn insights into actionable experiments for the next sprint.

Over time, retros become the heartbeat of team evolution.

Here is a video highlighting the importance of a Retro and Sprint Review.

🧠 Why Retrospection Matters More Than You Think

The Sprint Retrospective is more than just another meeting. It’s a mirror for your team – a safe, structured space to pause, reflect, and improve together.

You discuss:

✅ what went well

❌ what did not go well

🔁 what could we do better next time

Great teams don't just deliver great software, they continually deliver better ways of working.

This is why many experienced Scrum practitioners consider the retro to be the most important event in Scrum. Code is deployed once, but process improvements grow exponentially, sprint after sprint.

Tools You Might Encounter

Scrum doesn’t require software, but real-world teams use a variety of tools:

• Jira – Tracks sprints, issues, velocity

• Trello – Simple board, good for small teams

• Slack – Where standups often happen if async

• Notion / Confluence – Docs, retros, notes

• GitHub Projects – Lightweight planning for devs

Don’t worry if you’re not fluent in these yet. They’re tools – you’ll learn them on the job.

If You’re Preparing for a Job, Here’s What You Can Do

• ✍️ Practice writing user stories from your side projects

• 🧪 Run a mini-sprint: Plan your weekend project, set goals, and “review” it at the end

• 🤝 Contribute to an open-source project that uses Scrum or Agile workflows

• 🧾 Write about what you learned – maybe as a tutorial (hint hint)

Final Thoughts

So to recap, Scrum is a simple yet powerful way for teams to work together, stay organized, and deliver results quickly. It runs in short cycles called sprints, where the team plans what to do, checks in daily, shows their progress at the end, and reflects on how to improve.

The four key ceremonies – Sprint Planning, Daily Scrum, Sprint Review, and Sprint Retrospective – help keep everyone aligned and focused. With clear roles and regular feedback, Scrum makes it easier to handle changes, solve problems early, and continuously get better as a team.

But scrum isn’t a magic spell. It’s just a way for humans to build complex things – together – without falling apart.

You don’t need to be a Scrum Master. You don’t need a certification. But if you understand how sprints work, what’s expected of you, and how to show up to meetings with clarity and candor, you’re 10 steps ahead of most.

Scrum helps teams talk, plan, build, and learn. And now? You can too.

If you liked this, please do share. You never know who it might help out.

Until then…keep learning, unlearning, and relearning!!!

In this guide, you'll build a complete Task Manager app with user authentication, protected routes, and full CRUD functionality, built with React on the frontend and Express/MongoDB on the backend.

This article will serve as your hands-on, code-first guide to building, securing, and deploying a MERN application, drawing from my own practical experience. Every section has code you can run, and I’ll give concise explanations along the way.

It doesn’t matter if you're just getting started with MERN or looking to level up your architecture and production deployment knowledge – this article is designed to get you from zero to production with confidence.

Table of Contents

• Prerequisites

  • Tools & Tech Stack

  • Skills & Setup

• Project Setup: Laying the Groundwork

  • Project Structure

  • Code Quality: Linting and Formatting

• Testing: Ensuring Robustness

  • Backend Testing (Node.js/Express.js)

  • Frontend Testing (React Testing Library + Cypress)

• How to Build the Task Manager

  • Backend Implementation (Node.js/Express.js)

  • Frontend Implementation (React)

• Deployment: From Localhost to Live

  • Backend Deployment (Node.js/Express.js)

  • Frontend Deployment (React)

  • Database Deployment (MongoDB Atlas)

  • 1. .env Configuration Example

  • 2. Connect to MongoDB in app.js

  • Other Deployment Options

• Security Best Practices: Fortifying Your Application

  • Setup Input Validation and Sanitization

  • Add Authentication and Authorization

  • Implement Rate Limiting

  • Setup CORS Configuration (Cross-Origin Resource Sharing)

  • Use Environment Variables

• Monitoring and Logging with Winston and Morgan

  • Frontend Error Monitoring (Sentry)

• Conclusion

Prerequisites

Before jumping in the project, here’s what you’ll need to get the most out of this tutorial:

Tools & Tech Stack

You’ll be using the following technologies throughout the project:

• Node.js & npm – Backend runtime and package manager

• Express.js – Web framework for Node

• MongoDB Atlas – Cloud-hosted NoSQL database

• Mongoose – ODM for MongoDB

• React – Frontend UI library

• React Router – For client-side routing

• Axios – For making API requests

• Jest & Supertest – For backend tests

• React Testing Library & Cypress – For Frontend unit and E2E tests

• ESLint + Prettier – For code formatting, linting

• Husky – To setup pre-commit hooks

• Helmet, Joi, express-rate-limit, cors – For security, validation, and best practices

• PM2 & NGINX – For backend deployment

• Sentry – For error monitoring

Skills & Setup

• Basic knowledge of JavaScript, React, and Node.js

• Familiarity with REST APIs and HTTP request/response flows

• Git and a GitHub account for version control

• A free MongoDB Atlas account

• Node.js and npm installed locally (Node 18+ recommended)

Project Setup: Laying the Groundwork

A well-structured project is crucial for maintainability. We'll adopt a clear separation between the front end and the back end here.

Project Structure

This structure clearly separates the React front end (client/) from the Node.js/Express.js back end (server/), promoting modularity and easier management.

my-mern-app/                # Root folder
├── client/                 # React frontend
│   ├── public/
│   ├── src/
│   │   ├── components/
│   │   ├── pages/
│   │   ├── App.js
│   │   └── index.js
│   └── package.json
├── server/                 # Node.js/Express.js backend
│   ├── config/
│   ├── controllers/
│   ├── models/
│   ├── routes/
│   ├── services/
│   ├── app.js
│   └── package.json

Code Quality: Linting and Formatting

Consistency is key when you’re building a production-grad application like this one. We'll use ESLint with Airbnb style and Prettier for automated code quality and formatting.

To install these tools, run this in your terminal:

npm install --save-dev eslint prettier eslint-config-airbnb-base eslint-plugin-prettier

And here are some setups with their recommended configurations:

This configuration sets up ESLint for a Node.js project using the Airbnb and Prettier style guides, with custom rules to relax strict linting constraints like allowing console.log and disabling mandatory function names.

.eslintrc.js (server-side example)

module.exports = {

  env: {

    node: true,

    commonjs: true,

    es2021: true,

  },

  extends: ["airbnb-base", "prettier"],

  plugins: ["prettier"],

  parserOptions: {

    ecmaVersion: 12,

  },

  rules: {

    "prettier/prettier": "error",

    "no-console": "off",

    "func-names": "off",

    "no-process-exit": "off",

    "class-methods-use-this": "off",

    "import/no-extraneous-dependencies": "off",

  },

};

.prettierrc

This config enforces consistent formatting: add semicolons, use trailing commas where valid, and prefer single quotes for strings.

{

  "semi": true,

  "trailingComma": "all",

  "singleQuote": true

}

Version Control: Git Essentials

Git is indispensable. You can use feature branches and pull requests for collaborative development, making it easier to work on large projects with coworkers. Consider using Husky for pre-commit hooks to enforce linting and testing.

Install Husky:

Install Husky to easily manage Git hooks, allowing you to automate tasks like linting and testing before commits.

npm install husky --save-dev

package.json (add script)

This package.json file sets up a Node.js project named my-mern-app, and configures a prepare script to install Git hooks using Husky (v7). It's ready for adding pre-commit automation, such as linting or testing.

{

  "name": "my-mern-app",

  "version": "1.0.0",

  "description": "",

  "main": "index.js",

  "scripts": {

    "prepare": "husky install"

  },

  "keywords": [],

  "author": "",

  "license": "ISC",

  "devDependencies": {

    "husky": "^7.0.0"

  }

}

Create a pre-commit hook

The below command sets up a pre-commit hook that automatically runs your tests and linter before each commit, ensuring code quality and preventing errors from entering your codebase.

npx husky add .husky/pre-commit "npm test && npm run lint"

Testing: Ensuring Robustness

Automated testing is vital. We'll cover unit, integration, and end-to-end testing in this guide.

Backend Testing (Node.js/Express.js)

You’ll use Jest for unit testing and Supertest for API integration tests.

Install them like this:

npm install --save-dev jest supertest

You’ll use Jest to write unit tests for your JavaScript code and Supertest to test HTTP requests against your Express.js API.

Example Test (server/tests/auth.test.js):

This test suite uses Supertest to simulate API calls for user registration and login, asserting that the responses have the expected status codes and properties.

const request = require('supertest');

const app = require('../app'); // Your Express app instance

describe('Auth API', () => {

  it('should register a new user', async () => {

    const res = await request(app)

      .post('/api/auth/register')

      .send({

        username: 'testuser',

        email: 'test@example.com',

        password: 'password123',

      });

    expect(res.statusCode).toEqual(201);

    expect(res.body).toHaveProperty('_id');

  });


  it('should login an existing user', async () => {

    const res = await request(app)

      .post('/api/auth/login')

      .send({

        email: 'test@example.com',

        password: 'password123',

      });

    expect(res.statusCode).toEqual(200);

    expect(res.headers['set-cookie']).toBeDefined();

  });

});

Frontend Testing (React Testing Library + Cypress)

You’ll use Jest and the React Testing Library for unit/integration tests, and Cypress for E2E tests.

You can install these like this:

npm install --save-dev @testing-library/react @testing-library/jest-dom jest cypress

React Testing Library will help you test your React components, and Cypress will provide comprehensive end-to-end testing of your frontend application.

Example Component Test (client/src/components/Button.test.js):

This unit test uses the React Testing Library to render a Button component and verifies that the specified text content is present in the rendered output.

import React from 'react';

import { render, screen } from '@testing-library/react';

import Button from './Button';


test('renders button with text', () => {

  render(<Button>Click Me</Button>);

  const buttonElement = screen.getByText(/Click Me/i);

  expect(buttonElement).toBeInTheDocument();

});

The following Cypress test simulates a complete user authentication flow, from registration to login and logout, asserting expected URL changes and page content.

Example E2E Test (cypress/e2e/auth.cy.js)

describe('Authentication Flow', () => {

  it('should allow a user to register and login', () => {

    cy.visit('/register');

    cy.get('input[name="username"]').type('e2euser');

    cy.get('input[name="email"]').type('e2e@example.com');

    cy.get('input[name="password"]').type('password123');

    cy.get('button[type="submit"]').click();

    cy.url().should('include', '/dashboard');

    cy.contains('Welcome, e2euser');

    cy.get('button').contains('Logout').click();

    cy.url().should('include', '/login');

    cy.get('input[name="email"]').type('e2e@example.com');

    cy.get('input[name="password"]').type('password123');

    cy.get('button[type="submit"]').click();

    cy.url().should('include', '/dashboard');

  });

});

How to Build the Task Manager

We'll build a simple Task Manager with user authentication and CRUD operations for tasks so you can see how the whole thing comes together.

Backend Implementation (Node.js/Express.js)

Dependencies

Start by installing our core backend libraries: Express for routing, Mongoose for MongoDB interactions, dotenv for environment variables, bcrypt/jsonwebtoken/cookie-parser for secure authentication, and helmet for setting secure HTTP headers:

npm install express mongoose dotenv bcryptjs jsonwebtoken cookie-parser

server/app.js (Entry Point)

Next, we’ll set up the first or the main entry point for the backend. This is the main Express.js application file, which configures middleware, establishes a MongoDB connection, and sets up API routes for authentication and task management.

const express = require('express');

const mongoose = require('mongoose');

const dotenv = require('dotenv');

const cookieParser = require('cookie-parser');

const helmet = require('helmet');

const authRoutes = require('./routes/authRoutes');

const taskRoutes = require('./routes/taskRoutes');

const { notFound, errorHandler } = require('./middleware/errorMiddleware');

dotenv.config();


const app = express();

app.use(helmet());

app.use(express.json());

app.use(cookieParser());


mongoose.connect(process.env.MONGO_URI)

  .then(() => console.log('MongoDB connected!'))

  .catch(err => console.error('MongoDB connection error:', err));


app.use('/api/auth', authRoutes);

app.use('/api/tasks', taskRoutes);


app.get('/', (req, res) => {

  res.send('MERN Task Manager API is running!');

});


app.use(notFound);

app.use(errorHandler);


const PORT = process.env.PORT || 5000;

app.listen(PORT, () => {

  console.log(`Server running on port ${PORT}`);

});

server/.env

To avoid hardcoding secrets, we’ll add a .env file where we can securely store environment variables, such as our database URI and JWT secret. This file stores sensitive environment variables such as your MongoDB connection string, server port, and JWT secret, keeping them secure and separate from your codebase.

MONGO_URI=your_mongodb_connection_string_here

PORT=5000

JWT_SECRET=supersecretjwtkey

server/models/User.js

Now, let’s define our User model using MongoDB. This schema includes fields for username, email, and password, with pre-save hooks for password hashing and a method for password comparison.

const mongoose = require('mongoose');

const bcrypt = require('bcryptjs');


const UserSchema = new mongoose.Schema({

  username: {

    type: String,

    required: true,

    unique: true,

  },

  email: {

    type: String,

    required: true,

    unique: true,

  },

  password: {

    type: String,

    required: true,

  },

});

UserSchema.pre('save', async function (next) {

  if (!this.isModified('password')) {

    next();

  }

  const salt = await bcrypt.genSalt(10);

  this.password = await bcrypt.hash(this.password, salt);

});


UserSchema.methods.matchPassword = async function (enteredPassword) {

  return await bcrypt.compare(enteredPassword, this.password);

};


module.exports = mongoose.model('User', UserSchema);

server/models/Task.js

Next, we’ll create the Task model. This schema defines the Task model, which links each task to a user and includes fields for title, description, completion status, and creation timestamp.

const mongoose = require('mongoose');


const TaskSchema = new mongoose.Schema({

  user: {

    type: mongoose.Schema.Types.ObjectId,

    ref: 'User',

    required: true,

  },

  title: {

    type: String,

    required: true,

    trim: true,

  },

  description: {

    type: String,

    trim: true,

  },

  completed: {

    type: Boolean,

    default: false,

  },

  createdAt: {

    type: Date,

    default: Date.now,

  },

});


module.exports = mongoose.model('Task', TaskSchema);

server/controllers/authController.js

Let’s build out the authentication controller. This controller handles user authentication flows, including registration, login, logout, and fetching user profiles, using JWTs and secure HTTP-only cookies.

const User = require('../models/User');

const jwt = require('jsonwebtoken');

const generateToken = (id) => {

  return jwt.sign({ id }, process.env.JWT_SECRET, {

    expiresIn: '1h',

  });

};

exports.registerUser = async (req, res) => {

  const { username, email, password } = req.body;

  try {

    const userExists = await User.findOne({ email });

    if (userExists) return res.status(400).json({ message: 'User already exists' });

    const user = await User.create({ username, email, password });

    if (user) {

      const token = generateToken(user._id);

      res.cookie('token', token, { httpOnly: true, secure: process.env.NODE_ENV === 'production', maxAge: 3600000 });

      res.status(201).json({ id: user.id, username: user.username, email: user.email });

    } else {

      res.status(400).json({ message: 'Invalid user data' });

    }

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};


exports.loginUser = async (req, res) => {

  const { email, password } = req.body;

  try {

    const user = await User.findOne({ email });

    if (user && (await user.matchPassword(password))) {

      const token = generateToken(user._id);

      res.cookie('token', token, { httpOnly: true, secure: process.env.NODE_ENV === 'production', maxAge: 3600000 });

      res.json({ id: user.id, username: user.username, email: user.email });

    } else {

      res.status(401).json({ message: 'Invalid email or password' });

    }

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};


exports.logoutUser = (req, res) => {

  res.cookie('token', '', { httpOnly: true, expires: new Date(0) });

  res.status(200).json({ message: 'Logged out successfully' });

};


exports.getUserProfile = async (req, res) => {

  try {

    const user = await User.findById(req.user._id).select('-password');

    if (user) {

      res.json(user);

    } else {

      res.status(404).json({ message: 'User not found' });

    }

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};

server/controllers/taskController.js

Now it’s time to implement the task controller. This controller provides the logic for fetching, creating, updating, and deleting tasks, ensuring that users can only interact with their tasks.

const Task = require('../models/Task');


exports.getTasks = async (req, res) => {

  try {

    const tasks = await Task.find({ user: req.user._id });

    res.status(200).json(tasks);

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};


exports.createTask = async (req, res) => {

  const { title, description } = req.body;

  if (!title) return res.status(400).json({ message: 'Please add a title' });

  try {

    const task = await Task.create({ title, description, user: req.user._id });

    res.status(201).json(task);

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};


exports.updateTask = async (req, res) => {

  try {

    const task = await Task.findById(req.params.id);

    if (!task) return res.status(404).json({ message: 'Task not found' });

    if (task.user.toString() !== req.user._id.toString()) return res.status(401).json({ message: 'Not authorized' });


    const updatedTask = await Task.findByIdAndUpdate(req.params.id, req.body, { new: true, runValidators: true });

    res.status(200).json(updatedTask);

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};


exports.deleteTask = async (req, res) => {

  try {

    const task = await Task.findById(req.params.id);

    if (!task) return res.status(404).json({ message: 'Task not found' });

    if (task.user.toString() !== req.user._id.toString()) return res.status(401).json({ message: 'Not authorized' });


    await Task.deleteOne({ _id: req.params.id });

    res.status(200).json({ message: 'Task removed' });

  } catch (error) {

    res.status(500).json({ message: error.message });

  }

};

server/middleware/authMiddleware.js

To protect private routes, we will create a middleware that verifies the JWT from the request's cookies, ensuring that only authenticated users can access specific endpoints.

const jwt = require('jsonwebtoken');

const User = require('../models/User');

exports.protect = async (req, res, next) => {

  let token;

  if (req.cookies.token) {

    try {

      token = req.cookies.token;

      const decoded = jwt.verify(token, process.env.JWT_SECRET);

      req.user = await User.findById(decoded.id).select('-password');

      next();

    } catch (error) {

      res.status(401).json({ message: 'Not authorized, token failed' });

    }

  } else {

    res.status(401).json({ message: 'Not authorized, no token' });

  }

};

server/middleware/errorMiddleware.js

To handle errors cleanly across our backend, we’ll add global error-handling middleware that can handle 404 Not Found errors and provide a centralized error-handling mechanism for consistent API error responses.

exports.notFound = (req, res, next) => {

  const error = new Error(`Not Found - ${req.originalUrl}`);

  res.status(404);

  next(error);

};


exports.errorHandler = (err, req, res, next) => {

  const statusCode = res.statusCode === 200 ? 500 : res.statusCode;

  res.status(statusCode);

  res.json({

    message: err.message,

    stack: process.env.NODE_ENV === 'production' ? null : err.stack,

  });

};

server/routes/authRoutes.js

Next, let’s define our authentication routes. These endpoints enable user authentication and map HTTP methods to their corresponding controller functions.

const express = require('express');

const { registerUser, loginUser, logoutUser, getUserProfile } = require('../controllers/authController');

const { protect } = require('../middleware/authMiddleware');


const router = express.Router();


router.post('/register', registerUser);

router.post('/login', loginUser);

router.get('/logout', logoutUser);

router.get('/profile', protect, getUserProfile);


module.exports = router;

server/routes/taskRoutes.js

Now we’ll add the routes for task operations. This file defines the API routes for task management, applying the protect middleware to secure all task-related operations.

const express = require('express');

const { getTasks, createTask, updateTask, deleteTask } = require('../controllers/taskController');

const { protect } = require('../middleware/authMiddleware');

const router = express.Router();

router.route('/').get(protect, getTasks).post(protect, createTask);

router.route('/:id').put(protect, updateTask).delete(protect, deleteTask);

module.exports = router;

Frontend Implementation (React)

Dependencies

Now, you’ll need to initialize a new React project and install your essential libraries: Axios for HTTP requests, React Router for navigation, and React Toastify for displaying notifications.

npm install axios react-router-dom react-toastify

client/src/index.js

Let’s start the frontend by setting up the entry point. Here we are rendering the main App component and wrapping it with AuthProvider to provide authentication context globally.

import React from 'react';

import ReactDOM from 'react-dom/client';

import './index.css';

import App from './App';

import { AuthProvider } from './context/AuthContext';


const root = ReactDOM.createRoot(document.getElementById('root'));

root.render(

  <React.StrictMode>

    <AuthProvider>

      <App />

    </AuthProvider>

  </React.StrictMode>

);

client/src/App.js

Next, we’ll define our main App component. This sets up the client-side routing for the application, and defines public and private routes, and includes a navigation bar and toast notification system.

import React from 'react';

import { BrowserRouter as Router, Routes, Route } from 'react-router-dom';

import { ToastContainer } from 'react-toastify';

import 'react-toastify/dist/ReactToastify.css';


import Navbar from './components/Navbar';

import Register from './pages/Register';

import Login from './pages/Login';

import Dashboard from './pages/Dashboard';

import PrivateRoute from './components/PrivateRoute';


function App() {

  return (

    <Router>

      <Navbar />

      <ToastContainer />

      <div className="container">

        <Routes>

          <Route path="/register" element={<Register />} />

          <Route path="/login" element={<Login />} />

          <Route path="/dashboard" element={<PrivateRoute />}>

            <Route index element={<Dashboard />} />

          </Route>

          <Route path="/" element={<h1>Welcome to Task Manager!</h1>} />

        </Routes>

      </div>

    </Router>

  );

}

export default App;

client/src/context/AuthContext.js

We’ll create an authentication context that manages the global authentication state. It provides functions for user login, registration, and logout, and automatically loads user data on component mount.

import React, { createContext, useState, useEffect } from 'react';

import axios from 'axios';

const AuthContext = createContext();

export const AuthProvider = ({ children }) => {

  const [user, setUser] = useState(null);

  const [loading, setLoading] = useState(true);


  useEffect(() => {

    const loadUser = async () => {

      try {

        const res = await axios.get('/api/auth/profile');

        setUser(res.data);

      } catch (err) {

        setUser(null);

      } finally {

        setLoading(false);

      }

    };

    loadUser();

  }, []);


  const login = async (email, password) => {

    try {

      const res = await axios.post('/api/auth/login', { email, password });

      setUser(res.data);

      return true;

    } catch (err) {

      console.error(err.response.data.message);

      return false;

    }

  };


  const register = async (username, email, password) => {

    try {

      const res = await axios.post('/api/auth/register', { username, email, password });

      setUser(res.data);

      return true;

    } catch (err) {

      console.error(err.response.data.message);

      return false;

    }

  };


  const logout = async () => {

    try {

      await axios.get('/api/auth/logout');

      setUser(null);

    } catch (err) {

      console.error(err);

    }

  };


  return (

    <AuthContext.Provider value={{ user, loading, login, register, logout }}>

      {children}

    </AuthContext.Provider>

  );

};


export default AuthContext;

client/src/components/Navbar.js

Here’s a dynamic navigation bar component that dynamically displays links based on the user's authentication status, showing either login/register options or a welcome message and logout button.

import React, { useContext } from 'react';

import { Link } from 'react-router-dom';

import AuthContext from '../context/AuthContext';


const Navbar = () => {

  const { user, logout } = useContext(AuthContext);


  return (

    <nav>

      <h1>Task Manager</h1>

      <div>

        {user ? (

          <>

            <span>Welcome, {user.username}</span>

            <button onClick={logout}>Logout</button>

            <Link to="/dashboard">Dashboard</Link>

          </>

        ) : (

          <>

            <Link to="/login">Login</Link>

            <Link to="/register">Register</Link>

          </>

        )}

      </div>

    </nav>

  );

};


export default Navbar;

client/src/components/PrivateRoute.js

To protect certain pages, we can create a Private Route component. This will be a guard for private routes, ensuring that only authenticated users can access them and redirecting unauthenticated users to the login page.

import React, { useContext } from 'react';

import { Navigate, Outlet } from 'react-router-dom';

import AuthContext from '../context/AuthContext';


const PrivateRoute = () => {

  const { user, loading } = useContext(AuthContext);


  if (loading) {

    return <div>Loading...</div>; // Or a spinner

  }


  return user ? <Outlet /> : <Navigate to="/login" replace />;

};

export default PrivateRoute;

client/src/pages/Register.js

Now, let’s create the Register component, which provides a user registration form, handles input state and form submission, and displays success or error messages using toast notifications.

import React, { useState, useContext } from 'react';

import { useNavigate } from 'react-router-dom';

import { toast } from 'react-toastify';

import AuthContext from '../context/AuthContext';


const Register = () => {

  const [username, setUsername] = useState('');

  const [email, setEmail] = useState('');

  const [password, setPassword] = useState('');

  const { register } = useContext(AuthContext);

  const navigate = useNavigate();


  const handleSubmit = async (e) => {

    e.preventDefault();

    const success = await register(username, email, password);

    if (success) {

      toast.success('Registration successful!');

      navigate('/dashboard');

    } else {

      toast.error('Registration failed. Please try again.');

    }

  };


  return (

    <div>

      <h2>Register</h2>

      <form onSubmit={handleSubmit}>

        <div>

          <label>Username:</label>

          <input type="text" value={username} onChange={(e) => setUsername(e.target.value)} required />

        </div>

        <div>

          <label>Email:</label>

          <input type="email" value={email} onChange={(e) => setEmail(e.target.value)} required />

        </div>

        <div>

          <label>Password:</label>

          <input type="password" value={password} onChange={(e) => setPassword(e.target.value)} required />

        </div>

        <button type="submit">Register</button>

      </form>

    </div>

  );

};


export default Register;