Yet most developers have never built one.

That's a missed opportunity. CLI tools are one of the most practical things a developer can ship. They automate repetitive workflows, standardise processes across teams, and, when published, become tangible artifacts that the developer community can discover, install, and use.

In this handbook, you'll go from zero to building a fully distributed Dart CLI tool. We'll start with the fundamentals – how CLIs work, how Dart receives and processes terminal input, and the core syntax you need to know. Then we'll build three progressively complex CLIs, starting with the basics and finishing with a real-world API request runner. Finally, we will cover every distribution path available, from pub.dev to compiled binaries, Homebrew taps, Docker, and local team activation.

By the end of the guide, you'll understand both how to build a CLI tool in Dart as well as how to ship it so other developers can actually use it.

Table of Contents

• Prerequisites

• What is a CLI and Why Should You Build One?

• CLI Syntax Anatomy

• How Dart Receives Terminal Input

• Core CLI Concepts in Dart

  • stdout, stderr, and stdin

  • Exit Codes

  • Environment Variables

  • File and Directory Operations

  • Running External Processes

  • Platform Detection

  • Async in CLI

• Setting Up Your Dart CLI Project

• CLI 1 — Hello CLI: The Fundamentals

• CLI 2 — dart_todo: A Terminal Task Manager

  • Introducing the args Package

  • Building dart_todo

• CLI 3 — dart_http: A Lightweight API Request Runner

  • Building dart_http

• Adding Color and Polish to Your CLI

• Testing Your CLI Tool

• Deploying and Distributing Your CLI

  • Mode 1: pub.dev — Public Package Distribution

  • Mode 2: Local Path Activation

  • Mode 3: Compiled Binary via GitHub Releases

  • Mode 4: Homebrew Tap

  • Mode 5: Docker

• Choosing the Right Distribution Mode

• Conclusion

Prerequisites

Before starting, you should have:

• Dart SDK installed (dart --version should work in your terminal)

• Basic familiarity with Dart syntax

• Comfort with the terminal and running commands

• A pub.dev account (for the publishing section)

• A GitHub account (for the binary distribution section)

What is a CLI and Why Should You Build One?

A CLI (or Command Line Interface) is a program you interact with entirely through text commands in a terminal, rather than through buttons and screens in a graphical interface.

Many of the tools you likely already rely on as a developer are CLI tools:

flutter build apk
git commit -m "fix: auth flow"
dart pub get
npm install

Flutter, Git, Dart, npm – all CLIs. You are already a CLI user every single day. This article is about becoming a CLI builder.

There are three strong reasons to build CLI tools as a developer:

1. Automating repetitive work: Anything you type more than twice a week is a candidate for automation. Generating boilerplate folder structures, running sequences of commands, scaffolding files, checking environments before a build a CLI turns a seven-step manual process into a single command.

2. Standardising team workflows: Instead of a README that says "run these commands in this order," you ship one command that does all of it – consistently, every time, with no room for human error or a missed step.

3. Building and publishing tooling. A published Dart CLI package is a tangible artifact. It shows up on pub.dev, gets installed and used by other developers, and communicates real engineering depth in a way that a portfolio or resume cannot.

CLI Syntax Anatomy

Before writing a single line of code, it helps to understand the structure of a CLI command. Every command follows a consistent pattern:

tool [subcommand] [arguments] [options/flags]

Breaking down a real example:

flutter build apk --release --obfuscate
│       │     │   │
tool    sub   arg  flags

• Tool — the program itself (flutter, dart, git)

• Subcommand — the action being performed (build, run, pub)

• Arguments — what the action operates on (apk, main.dart, a filename)

• Flags and Options — modifiers that change behaviour

There are two types of options:

--release              # Boolean flag — either present or absent

--output=build/app     # Key-value option — name and a value
-v                     # Short flag — single hyphen, single character

This is the anatomy your CLIs will follow. Understanding it before writing any code means you will design your commands intentionally rather than stumbling into structure by accident.

How Dart Receives Terminal Input

In Dart, everything the user types after your tool name is passed into your program through the main function:

void main(List<String> args) {
  print(args);
}

Run it:

dart run bin/mytool.dart hello world --name=Seyi
# [hello, world, --name=Seyi]

That List<String> args is just a list of strings. Each word or flag the user typed becomes an element in that list. Everything else you build on top of a CLI subcommands, flags, validation — is ultimately just processing this list.

Core CLI Concepts in Dart

Before building anything, there's a set of foundational concepts that every CLI developer needs to understand. These are the building blocks that everything else sits on top of.

stdout, stderr, and stdin

Most developers use print() for all output when they start building CLIs. That works for learning but it's incorrect in production.

There are two separate output streams in a terminal program:

• stdout — regular output, meant for the user

• stderr — error output, meant for diagnostic messages and failures

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Error: no arguments provided');
    exit(1);
  }

  stdout.writeln('Processing: ${args[0]}');
}

Keeping these separate matters because users can redirect stdout to a file without errors polluting it:

dart run bin/tool.dart > output.txt
# Errors still appear in the terminal
# Normal output goes cleanly to the file

Tools like git, flutter, and curl all do this correctly. Your CLI should too.

stdin is the third stream — reading input from the user interactively at runtime:

import 'dart:io';

void main() {
  stdout.write('Enter your name: ');
  final name = stdin.readLineSync();

  if (name == null || name.trim().isEmpty) {
    stderr.writeln('Error: no name provided');
    exit(1);
  }

  stdout.writeln('Hello, $name!');
}

stdout.write (without ln) keeps the cursor on the same line so the user types right after the prompt. stdin.readLineSync() blocks until the user presses Enter and returns the typed string, or null if the stream closes unexpectedly. Always handle the null case.

Exit Codes

Every program returns an exit code when it finishes. This is how the shell – and any script or CI system calling your tool – knows whether it succeeded or failed.

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Error: please provide an argument');
    exit(1); // failure
  }

  stdout.writeln('Done');
  exit(0); // success — also the default if you don't call exit()
}

The conventions are:

• 0 — success

• 1 — general failure

• 2 — incorrect usage (wrong arguments, missing flags)

Exit codes are critical when your CLI is called inside shell scripts or GitHub Actions workflows. A non-zero exit code stops a pipeline immediately. That's exactly the behaviour you want from a quality gate or a validation step.

Environment Variables

Your CLI can read environment variables set in the user's shell:

import 'dart:io';

void main() {
  final token = Platform.environment['API_TOKEN'];

  if (token == null) {
    stderr.writeln('Error: API_TOKEN environment variable is not set');
    exit(1);
  }

  stdout.writeln('Token found — proceeding...');
}

Set it in the terminal and run:

export API_TOKEN=mytoken123
dart run bin/tool.dart
# Token found — proceeding...

This pattern is essential for CLI tools that interact with APIs, cloud services, or CI environments where credentials should never be hardcoded.

File and Directory Operations

Many CLI tools read from or write to the file system. Dart's dart:io library covers everything you need:

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: tool <filename>');
    exit(2);
  }

  final file = File(args[0]);

  if (!file.existsSync()) {
    stderr.writeln('Error: "${args[0]}" not found');
    exit(1);
  }

  final contents = file.readAsStringSync();
  stdout.writeln(contents);

  final output = File('output.txt');
  output.writeAsStringSync('Processed:\n$contents');
  stdout.writeln('Written to output.txt');
}

Working with directories:

import 'dart:io';

void main() {
  // Where the command was run from
  final cwd = Directory.current.path;
  stdout.writeln('Working directory: $cwd');

  // Create a directory relative to current location
  final dir = Directory('$cwd/generated');

  if (!dir.existsSync()) {
    dir.createSync(recursive: true);
    stdout.writeln('Created: ${dir.path}');
  } else {
    stdout.writeln('Already exists: ${dir.path}');
  }
}

The recursive: true flag on createSync means it creates all intermediate directories — equivalent to mkdir -p in bash.

Running External Processes

One of the most powerful things a CLI can do is call other programs. Your Dart CLI can run git, flutter, dart, or any shell command programmatically:

import 'dart:io';

void main() async {
  // Run a command and wait for it to finish
  final result = await Process.run('dart', ['pub', 'get']);

  stdout.write(result.stdout);

  if (result.exitCode != 0) {
    stderr.write(result.stderr);
    exit(result.exitCode);
  }

  stdout.writeln('Dependencies installed successfully');
}

For long-running commands where you want output to stream live as it happens:

import 'dart:io';

void main() async {
  final process = await Process.start('flutter', ['build', 'apk']);

  // Pipe output directly to the terminal in real time
  process.stdout.pipe(stdout);
  process.stderr.pipe(stderr);

  final exitCode = await process.exitCode;
  exit(exitCode);
}

Process.run — waits for completion, returns all output at once. Use for short commands.

Process.start — streams output live as it arrives. Use for long-running commands where the user needs to see progress.

Platform Detection

Sometimes your CLI needs to behave differently depending on the operating system it is running on:

import 'dart:io';

void main() {
  if (Platform.isWindows) {
    stdout.writeln('Running on Windows');
  } else if (Platform.isMacOS) {
    stdout.writeln('Running on macOS');
  } else if (Platform.isLinux) {
    stdout.writeln('Running on Linux');
  }

  // Useful for path handling across operating systems
  stdout.writeln(Platform.pathSeparator); // \ on Windows, / elsewhere
  stdout.writeln(Platform.operatingSystem); // 'macos', 'linux', 'windows'
}

This matters when your CLI creates files, resolves paths, or calls shell commands that differ between operating systems.

Async in CLI

Dart CLIs support async/await natively. Any main function can be made async:

import 'dart:io';

void main() async {
  stdout.writeln('Starting...');

  await Future.delayed(const Duration(seconds: 1)); // simulating async work

  stdout.writeln('Done');
}

Any operation involving file I/O, HTTP requests, or spawning processes will be asynchronous. Get comfortable with async main functions early — you'll use them constantly.

Setting Up Your Dart CLI Project

Create a new Dart console project:

dart create -t console my_cli_tool
cd my_cli_tool

This generates a clean structure:

my_cli_tool/
  bin/
    my_cli_tool.dart    ← entry point
  lib/                  ← shared library code
  test/                 ← tests
  pubspec.yaml
  README.md

The bin/ directory is where your executable entry point lives. The lib/ directory is where you put everything else — commands, utilities, models — that bin/ imports and uses.

Open pubspec.yaml. You'll need to add an executables block before publishing:

name: my_cli_tool
description: A sample CLI tool built with Dart
version: 1.0.0

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  my_cli_tool: my_cli_tool  # executable name: bin file name

dependencies:
  args: ^2.4.2

dev_dependencies:
  lints: ^3.0.0
  test: ^1.24.0

The executables block is what makes dart pub global activate my_cli_tool work. It tells Dart which script in bin/ to expose as a runnable command after installation.

CLI 1 — Hello CLI: The Fundamentals

This first CLI uses pure Dart — no packages. The goal is to get comfortable with args, subcommands, input validation, and exit codes before introducing any external dependencies.

Replace the contents of bin/my_cli_tool.dart:

import 'dart:io';

void main(List<String> args) {
  if (args.isEmpty) {
    printHelp();
    exit(0);
  }

  final command = args[0];

  switch (command) {
    case 'greet':
      handleGreet(args.sublist(1));
    case 'time':
      handleTime();
    case 'echo':
      handleEcho(args.sublist(1));
    case 'help':
      printHelp();
    default:
      stderr.writeln('Unknown command: "$command"');
      stderr.writeln('Run "mytool help" to see available commands.');
      exit(1);
  }
}

void handleGreet(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: mytool greet <name>');
    exit(2);
  }

  final name = args[0];
  stdout.writeln('Hello, $name! Welcome to your first Dart CLI.');
}

void handleTime() {
  final now = DateTime.now();
  stdout.writeln(
    'Current time: ${now.hour.toString().padLeft(2, '0')}:'
    '${now.minute.toString().padLeft(2, '0')}:'
    '${now.second.toString().padLeft(2, '0')}',
  );
}

void handleEcho(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: mytool echo <message>');
    exit(2);
  }

  stdout.writeln(args.join(' '));
}

void printHelp() {
  stdout.writeln('''
mytool — a simple Dart CLI

Usage:
  mytool <command> [arguments]

Commands:
  greet <name>      Greet someone by name
  time              Show the current time
  echo <message>    Echo a message back to the terminal
  help              Show this help message

Examples:
  mytool greet Seyi
  mytool echo "Hello from the terminal"
  mytool time
  ''');
}

Run it:

dart run bin/my_cli_tool.dart help

dart run bin/my_cli_tool.dart greet Seyi
# Hello, Seyi! Welcome to your first Dart CLI.

dart run bin/my_cli_tool.dart time
# Current time: 14:32:10

dart run bin/my_cli_tool.dart echo "Dart CLIs are powerful"
# Dart CLIs are powerful

dart run bin/my_cli_tool.dart unknown
# Unknown command: "unknown"
# Run "mytool help" to see available commands.

Three things this CLI demonstrates that are worth internalising:

1. Subcommands are just a switch on args[0]. The pattern is simple and scalable — add a new case to add a new command.

2. args.sublist(1) passes remaining args to the handler. When greet receives ['greet', 'Seyi'], it calls handleGreet(['Seyi']) — clean and isolated.

3. Every error path has a message and a non-zero exit code. The user always knows what went wrong and what to do next.

CLI 2 — dart_todo: A Terminal Task Manager

This CLI introduces the args package, JSON file persistence, and structured terminal output. It's meaningfully more complex than CLI 1 and reflects real patterns you will use in production tools.

Introducing the args Package

Manually parsing List<String> args works for simple cases, but breaks down quickly when you add flags like --priority=high, boolean options like --done, or commands with multiple optional arguments.

The args package handles all of that cleanly.

Add it to your pubspec.yaml:

dependencies:
  args: ^2.4.2

Run:

dart pub get

The core concept in args is the ArgParser. You define what your CLI accepts, and args handles parsing, validation, and generating help text automatically:

import 'package:args/args.dart';

void main(List<String> arguments) {
  final parser = ArgParser()
    ..addCommand('add')
    ..addCommand('list')
    ..addFlag('help', abbr: 'h', negatable: false);

  final results = parser.parse(arguments);

  if (results['help'] as bool) {
    print(parser.usage);
    return;
  }
}

For more complex CLIs with subcommands that each have their own flags, use ArgParser per command:

final parser = ArgParser();

final addCommand = ArgParser()
  ..addOption('priority', abbr: 'p', defaultsTo: 'normal');

parser.addCommand('add', addCommand);

Building dart_todo

Create a fresh project:

dart create -t console dart_todo
cd dart_todo

Update pubspec.yaml:

name: dart_todo
description: A terminal task manager built with Dart
version: 1.0.0

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  dart_todo: dart_todo

dependencies:
  args: ^2.4.2

dev_dependencies:
  lints: ^3.0.0
  test: ^1.24.0

Run dart pub get.

Create the folder structure:

dart_todo/
  bin/
    dart_todo.dart
  lib/
    models/
      task.dart
    storage/
      task_storage.dart
    commands/
      add_command.dart
      list_command.dart
      complete_command.dart
      delete_command.dart
      clear_command.dart
  pubspec.yaml

Step 1 — The Task Model (lib/models/task.dart)

class Task {
  final int id;
  final String title;
  final String priority;
  final bool isComplete;
  final DateTime createdAt;

  Task({
    required this.id,
    required this.title,
    required this.priority,
    this.isComplete = false,
    required this.createdAt,
  });

  Task copyWith({bool? isComplete}) {
    return Task(
      id: id,
      title: title,
      priority: priority,
      isComplete: isComplete ?? this.isComplete,
      createdAt: createdAt,
    );
  }

  Map<String, dynamic> toJson() => {
        'id': id,
        'title': title,
        'priority': priority,
        'isComplete': isComplete,
        'createdAt': createdAt.toIso8601String(),
      };

  factory Task.fromJson(Map<String, dynamic> json) => Task(
        id: json['id'] as int,
        title: json['title'] as String,
        priority: json['priority'] as String,
        isComplete: json['isComplete'] as bool,
        createdAt: DateTime.parse(json['createdAt'] as String),
      );
}

Step 2 — Storage (lib/storage/task_storage.dart)

This class handles reading and writing tasks to a local JSON file so they persist between CLI runs:

import 'dart:convert';
import 'dart:io';

import '../models/task.dart';

class TaskStorage {
  static final _file = File(
    '${Platform.environment['HOME'] ?? Directory.current.path}/.dart_todo.json',
  );

  static List<Task> loadAll() {
    if (!_file.existsSync()) return [];

    try {
      final content = _file.readAsStringSync();
      final List<dynamic> json = jsonDecode(content) as List<dynamic>;
      return json
          .map((e) => Task.fromJson(e as Map<String, dynamic>))
          .toList();
    } catch (_) {
      return [];
    }
  }

  static void saveAll(List<Task> tasks) {
    final json = jsonEncode(tasks.map((t) => t.toJson()).toList());
    _file.writeAsStringSync(json);
  }
}

Tasks are stored in a hidden JSON file in the user's home directory — a common pattern for CLI tools that need lightweight local persistence.

Step 3 — Commands

lib/commands/add_command.dart:

import 'dart:io';

import '../models/task.dart';
import '../storage/task_storage.dart';

void runAdd(List<String> args, String priority) {
  if (args.isEmpty) {
    stderr.writeln('Usage: dart_todo add <title> [--priority=high|normal|low]');
    exit(2);
  }

  final title = args.join(' ');
  final tasks = TaskStorage.loadAll();

  final newTask = Task(
    id: tasks.isEmpty ? 1 : tasks.last.id + 1,
    title: title,
    priority: priority,
    createdAt: DateTime.now(),
  );

  tasks.add(newTask);
  TaskStorage.saveAll(tasks);

  stdout.writeln('Added task #\({newTask.id}: "\)title" [$priority]');
}

lib/commands/list_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runList() {
  final tasks = TaskStorage.loadAll();

  if (tasks.isEmpty) {
    stdout.writeln('No tasks yet. Add one with: dart_todo add <title>');
    return;
  }

  stdout.writeln('');
  stdout.writeln('  ID   Status      Priority   Title');
  stdout.writeln('  ───  ──────────  ─────────  ────────────────────────');

  for (final task in tasks) {
    final status = task.isComplete ? 'done  ' : 'pending';
    final id = task.id.toString().padRight(4);
    final priority = task.priority.padRight(9);
    stdout.writeln('  \(id \)status  \(priority  \){task.title}');
  }

  stdout.writeln('');
}

lib/commands/complete_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runComplete(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: dart_todo complete <id>');
    exit(2);
  }

  final id = int.tryParse(args[0]);
  if (id == null) {
    stderr.writeln('Error: "${args[0]}" is not a valid task ID');
    exit(1);
  }

  final tasks = TaskStorage.loadAll();
  final index = tasks.indexWhere((t) => t.id == id);

  if (index == -1) {
    stderr.writeln('Error: No task found with ID $id');
    exit(1);
  }

  if (tasks[index].isComplete) {
    stdout.writeln('Task #$id is already complete.');
    return;
  }

  tasks[index] = tasks[index].copyWith(isComplete: true);
  TaskStorage.saveAll(tasks);

  stdout.writeln('Task #\(id marked as complete: "\){tasks[index].title}"');
}

lib/commands/delete_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runDelete(List<String> args) {
  if (args.isEmpty) {
    stderr.writeln('Usage: dart_todo delete <id>');
    exit(2);
  }

  final id = int.tryParse(args[0]);
  if (id == null) {
    stderr.writeln('Error: "${args[0]}" is not a valid task ID');
    exit(1);
  }

  final tasks = TaskStorage.loadAll();
  final index = tasks.indexWhere((t) => t.id == id);

  if (index == -1) {
    stderr.writeln('Error: No task found with ID $id');
    exit(1);
  }

  final title = tasks[index].title;
  tasks.removeAt(index);
  TaskStorage.saveAll(tasks);

  stdout.writeln('Deleted task #\(id: "\)title"');
}

lib/commands/clear_command.dart:

import 'dart:io';

import '../storage/task_storage.dart';

void runClear() {
  stdout.write('Are you sure you want to delete all tasks? (y/N): ');
  final input = stdin.readLineSync()?.trim().toLowerCase();

  if (input != 'y') {
    stdout.writeln('Cancelled.');
    return;
  }

  TaskStorage.saveAll([]);
  stdout.writeln('All tasks cleared.');
}

Step 4 — Entry Point (bin/dart_todo.dart)

import 'dart:io';

import 'package:args/args.dart';

import '../lib/commands/add_command.dart';
import '../lib/commands/clear_command.dart';
import '../lib/commands/complete_command.dart';
import '../lib/commands/delete_command.dart';
import '../lib/commands/list_command.dart';

void main(List<String> arguments) {
  final parser = ArgParser();

  // Add subcommand parsers
  final addParser = ArgParser()
    ..addOption(
      'priority',
      abbr: 'p',
      defaultsTo: 'normal',
      allowed: ['high', 'normal', 'low'],
      help: 'Task priority level',
    );

  parser
    ..addCommand('add', addParser)
    ..addCommand('list')
    ..addCommand('complete')
    ..addCommand('delete')
    ..addCommand('clear')
    ..addFlag('help', abbr: 'h', negatable: false, help: 'Show help');

  ArgResults results;

  try {
    results = parser.parse(arguments);
  } catch (e) {
    stderr.writeln('Error: $e');
    stderr.writeln(parser.usage);
    exit(2);
  }

  if (results['help'] as bool || results.command == null) {
    printHelp(parser);
    exit(0);
  }

  final command = results.command!;

  switch (command.name) {
    case 'add':
      runAdd(command.rest, command['priority'] as String);
    case 'list':
      runList();
    case 'complete':
      runComplete(command.rest);
    case 'delete':
      runDelete(command.rest);
    case 'clear':
      runClear();
    default:
      stderr.writeln('Unknown command: "${command.name}"');
      exit(1);
  }
}

void printHelp(ArgParser parser) {
  stdout.writeln('''
dart_todo — a terminal task manager

Usage:
  dart_todo <command> [arguments]

Commands:
  add <title>        Add a new task
    -p, --priority   Priority: high, normal, low (default: normal)
  list               List all tasks
  complete <id>      Mark a task as complete
  delete <id>        Delete a task
  clear              Delete all tasks

Examples:
  dart_todo add "Write the CLI article" --priority=high
  dart_todo list
  dart_todo complete 1
  dart_todo delete 2
  dart_todo clear
  ''');
}

Run it:

dart run bin/dart_todo.dart add "Write the CLI article" --priority=high
# Added task #1: "Write the CLI article" [high]

dart run bin/dart_todo.dart add "Review PR comments"
# Added task #2: "Review PR comments" [normal]

dart run bin/dart_todo.dart list
#   ID   Status      Priority   Title
#   ───  ──────────  ─────────  ────────────────────────
#   1    ⬜ pending  high       Write the CLI article
#   2    ⬜ pending  normal     Review PR comments

dart run bin/dart_todo.dart complete 1
# Task #1 marked as complete: "Write the CLI article"

dart run bin/dart_todo.dart delete 2
# Deleted task #2: "Review PR comments"

dart_todo demonstrates the patterns that form the backbone of almost every real CLI tool — argument parsing with args, JSON persistence, interactive prompts, structured output, and clean error handling across every command.

CLI 3 — dart_http: A Lightweight API Request Runner

This is the most complex CLI in this article – and the most immediately useful. dart_http lets developers make HTTP requests directly from the terminal, with pretty-printed JSON responses, response metadata, header support, and the ability to save responses to a file.

dart_http get https://jsonplaceholder.typicode.com/users/1
dart_http post https://jsonplaceholder.typicode.com/posts --body='{"title":"Hello"}'
dart_http get https://jsonplaceholder.typicode.com/users --save=users.json
dart_http get https://api.example.com/me --header="Authorization: Bearer mytoken"

Building dart_http

Create the project:

dart create -t console dart_http
cd dart_http

Update pubspec.yaml:

name: dart_http
description: A lightweight API request runner for the terminal
version: 1.0.0

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  dart_http: dart_http

dependencies:
  args: ^2.4.2
  http: ^1.2.1

dev_dependencies:
  lints: ^3.0.0
  test: ^1.24.0

Run dart pub get.

Project structure:

dart_http/
  bin/
    dart_http.dart
  lib/
    runner/
      request_runner.dart
    printer/
      response_printer.dart
    utils/
      headers_parser.dart
  pubspec.yaml

Step 1 — Headers Parser (lib/utils/headers_parser.dart)

Map<String, String> parseHeaders(List<String> rawHeaders) {
  final headers = <String, String>{};

  for (final header in rawHeaders) {
    final index = header.indexOf(':');
    if (index == -1) continue;

    final key = header.substring(0, index).trim();
    final value = header.substring(index + 1).trim();
    headers[key] = value;
  }

  return headers;
}

Step 2 — Response Printer (lib/printer/response_printer.dart)

import 'dart:convert';
import 'dart:io';

void printResponse({
  required int statusCode,
  required String body,
  required int durationMs,
  required int bodyBytes,
}) {
  final statusLabel = _statusLabel(statusCode);
  final size = _formatSize(bodyBytes);

  stdout.writeln('');
  stdout.writeln('\(statusLabel | \){durationMs}ms | $size');
  stdout.writeln('─' * 50);

  try {
    final decoded = jsonDecode(body);
    const encoder = JsonEncoder.withIndent('  ');
    stdout.writeln(encoder.convert(decoded));
  } catch (_) {
    // Not JSON — print as plain text
    stdout.writeln(body);
  }

  stdout.writeln('');
}

String _statusLabel(int code) {
  if (code >= 200 && code < 300) return '✅ $code';
  if (code >= 300 && code < 400) return '↪️  $code';
  if (code >= 400 && code < 500) return '❌ $code';
  return '$code';
}

String _formatSize(int bytes) {
  if (bytes < 1024) return '${bytes}b';
  if (bytes < 1024 * 1024) return '${(bytes / 1024).toStringAsFixed(1)}kb';
  return '${(bytes / (1024 * 1024)).toStringAsFixed(1)}mb';
}

Step 3 — Request Runner (lib/runner/request_runner.dart)

import 'dart:io';

import 'package:http/http.dart' as http;

import '../printer/response_printer.dart';

Future<void> runRequest({
  required String method,
  required String url,
  required Map<String, String> headers,
  String? body,
  String? saveToFile,
}) async {
  final uri = Uri.tryParse(url);

  if (uri == null) {
    stderr.writeln('Error: "$url" is not a valid URL');
    exit(1);
  }

  stdout.writeln('→ \({method.toUpperCase()} \)url');

  http.Response response;
  final stopwatch = Stopwatch()..start();

  try {
    switch (method.toLowerCase()) {
      case 'get':
        response = await http.get(uri, headers: headers);
      case 'post':
        response = await http.post(uri, headers: headers, body: body);
      case 'put':
        response = await http.put(uri, headers: headers, body: body);
      case 'patch':
        response = await http.patch(uri, headers: headers, body: body);
      case 'delete':
        response = await http.delete(uri, headers: headers);
      default:
        stderr.writeln('Error: unsupported method "$method"');
        exit(2);
    }
  } catch (e) {
    stderr.writeln('Error: request failed — $e');
    exit(1);
  }

  stopwatch.stop();

  printResponse(
    statusCode: response.statusCode,
    body: response.body,
    durationMs: stopwatch.elapsedMilliseconds,
    bodyBytes: response.bodyBytes.length,
  );

  if (saveToFile != null) {
    final file = File(saveToFile);
    file.writeAsStringSync(response.body);
    stdout.writeln('Response saved to $saveToFile');
  }
}

Step 4 — Entry Point (bin/dart_http.dart)

import 'dart:io';

import 'package:args/args.dart';

import '../lib/runner/request_runner.dart';
import '../lib/utils/headers_parser.dart';

void main(List<String> arguments) async {
  final parser = ArgParser();

  for (final method in ['get', 'post', 'put', 'patch', 'delete']) {
    final commandParser = ArgParser()
      ..addMultiOption('header', abbr: 'H', help: 'Request header (repeatable)')
      ..addOption('body', abbr: 'b', help: 'Request body (for POST/PUT/PATCH)')
      ..addOption('save', abbr: 's', help: 'Save response body to a file');

    parser.addCommand(method, commandParser);
  }

  parser.addFlag('help', abbr: 'h', negatable: false, help: 'Show help');

  ArgResults results;

  try {
    results = parser.parse(arguments);
  } catch (e) {
    stderr.writeln('Error: $e');
    printHelp();
    exit(2);
  }

  if (results['help'] as bool || results.command == null) {
    printHelp();
    exit(0);
  }

  final command = results.command!;
  final method = command.name!;
  final rest = command.rest;

  if (rest.isEmpty) {
    stderr.writeln('Error: please provide a URL');
    stderr.writeln('Usage: dart_http $method <url>');
    exit(2);
  }

  final url = rest[0];
  final rawHeaders = command['header'] as List<String>;
  final body = command['body'] as String?;
  final saveToFile = command['save'] as String?;

  final headers = parseHeaders(rawHeaders);

  // Default Content-Type for requests with a body
  if (body != null && !headers.containsKey('Content-Type')) {
    headers['Content-Type'] = 'application/json';
  }

  await runRequest(
    method: method,
    url: url,
    headers: headers,
    body: body,
    saveToFile: saveToFile,
  );
}

void printHelp() {
  stdout.writeln('''
dart_http — a lightweight API request runner

Usage:
  dart_http <method> <url> [options]

Methods:
  get       Send a GET request
  post      Send a POST request
  put       Send a PUT request
  patch     Send a PATCH request
  delete    Send a DELETE request

Options:
  -H, --header    Add a request header (repeatable)
  -b, --body      Request body (JSON string)
  -s, --save      Save response body to a file
  -h, --help      Show this help message

Examples:
  dart_http get https://jsonplaceholder.typicode.com/users
  dart_http get https://api.example.com/me --header="Authorization: Bearer token"
  dart_http post https://api.example.com/posts --body=\'{"title":"Hello"}\'
  dart_http get https://api.example.com/users --save=users.json
  ''');
}

Run it:

dart run bin/dart_http.dart get https://jsonplaceholder.typicode.com/users/1

# → GET https://jsonplaceholder.typicode.com/users/1
# 200 | 87ms | 510b
# ──────────────────────────────────────────────────
# {
#   "id": 1,
#   "name": "Leanne Graham",
#   "username": "Bret",
#   "email": "Sincere@april.biz"
# }

dart run bin/dart_http.dart get https://jsonplaceholder.typicode.com/users --save=users.json
# → GET https://jsonplaceholder.typicode.com/users
# 200 | 143ms | 5.3kb
# ──────────────────────────────────────────────────
# [ ... ]
# Response saved to users.json

dart run bin/dart_http.dart post https://jsonplaceholder.typicode.com/posts \
  --body='{"title":"Hello from dart_http","userId":1}'
# → POST https://jsonplaceholder.typicode.com/posts
# 201 | 312ms | 72b

Adding Color and Polish to Your CLI

The CLIs above are functional, but terminal output can be made significantly more readable with color. The ansi_styles package provides ANSI escape code support for coloring text in the terminal.

Add it to pubspec.yaml:

dependencies:
  ansi_styles: ^0.3.0

Using it:

import 'package:ansi_styles/ansi_styles.dart';

stdout.writeln(AnsiStyles.green('✅ Success'));
stdout.writeln(AnsiStyles.red('❌ Error: something went wrong'));
stdout.writeln(AnsiStyles.yellow('⚠️  Warning: check your config'));
stdout.writeln(AnsiStyles.bold('dart_http — API request runner'));
stdout.writeln(AnsiStyles.cyan('→ GET https://api.example.com/users'));

Apply color intentionally and consistently:

• Green — success states, completed operations

• Red — errors and failures

• Yellow — warnings and non-blocking issues

• Cyan — informational output, URLs, paths

• Bold — headers, tool names, important values

Avoid coloring everything. Color loses meaning when it is everywhere. Use it to draw the user's eye to what actually matters.

Testing Your CLI Tool

CLI tools are testable, and they should be tested. The most reliable approach is to test the logic inside your commands directly — not the terminal output formatting, but the behaviour.

Add test to your dev dependencies if it's not already there:

dev_dependencies:
  test: ^1.24.0

Testing command logic:

import 'package:test/test.dart';

import '../lib/models/task.dart';

void main() {
  group('Task model', () {
    test('copyWith updates isComplete correctly', () {
      final task = Task(
        id: 1,
        title: 'Write tests',
        priority: 'high',
        createdAt: DateTime.now(),
      );

      final completed = task.copyWith(isComplete: true);

      expect(completed.isComplete, isTrue);
      expect(completed.title, equals('Write tests'));
      expect(completed.id, equals(1));
    });

    test('toJson and fromJson round-trips correctly', () {
      final task = Task(
        id: 2,
        title: 'Ship the tool',
        priority: 'normal',
        createdAt: DateTime.parse('2025-01-01T00:00:00.000'),
      );

      final json = task.toJson();
      final restored = Task.fromJson(json);

      expect(restored.id, equals(task.id));
      expect(restored.title, equals(task.title));
      expect(restored.priority, equals(task.priority));
    });
  });
}

Testing the headers parser:

import 'package:test/test.dart';

import '../lib/utils/headers_parser.dart';

void main() {
  group('parseHeaders', () {
    test('parses a single header correctly', () {
      final result = parseHeaders(['Authorization: Bearer mytoken']);
      expect(result['Authorization'], equals('Bearer mytoken'));
    });

    test('parses multiple headers', () {
      final result = parseHeaders([
        'Authorization: Bearer token',
        'Accept: application/json',
      ]);
      expect(result.length, equals(2));
      expect(result['Accept'], equals('application/json'));
    });

    test('ignores malformed headers without a colon', () {
      final result = parseHeaders(['malformed-header']);
      expect(result.isEmpty, isTrue);
    });
  });
}

Run your tests:

dart test

Deploying and Distributing Your CLI

Building a CLI tool is half the work. Getting it into the hands of developers is the other half. There are five distribution paths available, each suited to a different use case.

Mode 1: pub.dev — Public Package Distribution

Publishing to pub.dev makes your tool installable by anyone in the Dart and Flutter community with a single command.

Prepare your package:

Your pubspec.yaml needs to be complete:

name: dart_http
description: A lightweight API request runner for Dart developers.
version: 1.0.0
homepage: https://github.com/yourname/dart_http

environment:
  sdk: '>=3.0.0 <4.0.0'

executables:
  dart_http: dart_http

The executables block is critical. It tells pub.dev which script in bin/ to expose as a runnable command.

You also need:

• README.md — what the tool does, how to install it, usage examples

• CHANGELOG.md — version history

• LICENSE — an open source license (MIT is standard)

Validate before publishing:

dart pub publish --dry-run

This runs all validation checks without actually publishing. Fix any warnings before proceeding.

Publish:

dart pub publish

You will be prompted to authenticate with your pub.dev account. Once published, your tool is available globally:

dart pub global activate dart_http
dart_http get https://api.example.com/users

Mode 2: Local Path Activation

For internal team tools that you don't want to publish publicly, activate directly from a local or cloned repository:

dart pub global activate --source path /path/to/dart_http

Any developer on the team clones the repo and runs this command once. The tool is then available globally in their terminal without needing a pub.dev publish.

This is the right distribution mode for:

• Internal company tooling

• Tools that depend on private packages

• Work-in-progress tools shared within a team before a public release

Mode 3: Compiled Binary via GitHub Releases

Dart can compile to a self-contained native executable — no Dart SDK required on the target machine. This makes your tool accessible to developers outside the Dart ecosystem.

Compile:

# macOS
dart compile exe bin/dart_http.dart -o dist/dart_http-macos

# Linux
dart compile exe bin/dart_http.dart -o dist/dart_http-linux

# Windows
dart compile exe bin/dart_http.dart -o dist/dart_http-windows.exe

The compiled binary is fully self-contained. Copy it to any machine and run it — no Dart installation needed.

Automate with GitHub Actions:

Create .github/workflows/release.yml:

name: Release

on:
  push:
    tags:
      - 'v*'

jobs:
  build:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}

    steps:
      - uses: actions/checkout@v3

      - uses: dart-lang/setup-dart@v1
        with:
          sdk: stable

      - name: Install dependencies
        run: dart pub get

      - name: Compile binary
        run: |
          mkdir -p dist
          dart compile exe bin/dart_http.dart -o dist/dart_http-${{ runner.os }}

      - name: Upload binary to release
        uses: softprops/action-gh-release@v1
        with:
          files: dist/dart_http-${{ runner.os }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Every time you push a version tag (v1.0.0), GitHub Actions compiles binaries for all three platforms and attaches them to the GitHub Release automatically.

Write an install script:

#!/usr/bin/env bash
set -euo pipefail

VERSION="1.0.0"
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
BINARY="dart_http-$OS"
INSTALL_DIR="/usr/local/bin"

curl -L "https://github.com/yourname/dart_http/releases/download/v\(VERSION/\)BINARY" \
  -o "$INSTALL_DIR/dart_http"

chmod +x "$INSTALL_DIR/dart_http"
echo "dart_http installed successfully"

Developers install it with:

curl -fsSL https://raw.githubusercontent.com/yourname/dart_http/main/install.sh | bash

Mode 4: Homebrew Tap

Homebrew is the standard package manager for macOS and is widely used on Linux. A Homebrew tap makes your tool installable with brew install — the most familiar installation pattern for macOS developers.

Create your tap repository:

Create a new GitHub repository named homebrew-tools (the homebrew- prefix is required by Homebrew's naming convention).

Write the formula:

Create Formula/dart_http.rb in that repository:

class DartHttp < Formula
  desc "A lightweight API request runner for the terminal"
  homepage "https://github.com/yourname/dart_http"
  version "1.0.0"

  on_macos do
    url "https://github.com/yourname/dart_http/releases/download/v1.0.0/dart_http-macOS"
    sha256 "YOUR_SHA256_HASH_HERE"
  end

  on_linux do
    url "https://github.com/yourname/dart_http/releases/download/v1.0.0/dart_http-Linux"
    sha256 "YOUR_SHA256_HASH_HERE"
  end

  def install
    bin.install "dart_http-#{OS.mac? ? 'macOS' : 'Linux'}" => "dart_http"
  end

  test do
    system "#{bin}/dart_http", "--help"
  end
end

Generate the SHA256 hash for each binary:

shasum -a 256 dist/dart_http-macOS

Install from the tap:

brew tap yourname/tools
brew install dart_http

When you release a new version, update the url and sha256 values in the formula and push the change. Users run brew upgrade dart_http to update.

Mode 5: Docker

Docker distribution is best suited for CI environments, teams that standardise on containers, or tools with complex dependencies.

Write a Dockerfile:

FROM dart:stable AS build

WORKDIR /app
COPY pubspec.* ./
RUN dart pub get

COPY . .
RUN dart compile exe bin/dart_http.dart -o /app/dart_http

FROM debian:stable-slim
COPY --from=build /app/dart_http /usr/local/bin/dart_http

ENTRYPOINT ["dart_http"]

This uses a multi-stage build: the first stage compiles the binary using the Dart SDK image, and the second stage copies only the binary into a minimal Debian image. The final image has no Dart SDK — just the compiled binary.

Build and run:

docker build -t dart_http .
docker run dart_http get https://jsonplaceholder.typicode.com/users/1

Publish to Docker Hub:

docker tag dart_http yourname/dart_http:1.0.0
docker push yourname/dart_http:1.0.0

Users can then run your tool without installing anything locally:

docker run yourname/dart_http get https://api.example.com/users

Choosing the Right Distribution Mode

For most tools, the practical recommendation is:

• Start with pub.dev if your audience is Dart developers

• Add compiled binary + GitHub Releases once you want broader adoption

• Add a Homebrew tap when macOS developers start asking for it

• Use Docker only when it is already part of your team's workflow

Conclusion

You've gone from understanding what a CLI is to building three progressively complex tools and distributing them across five different channels.

The foundational skills – args, stdin, stdout, stderr, exit codes, file I/O, and process spawning – are the same building blocks that tools like flutter, git, and dart themselves are built on. Everything else is composition.

The three CLIs we built (Hello CLI, dart_todo, and dart_http) each introduced a new layer: raw Dart fundamentals, the args package with JSON persistence, and real-world HTTP interaction. The distribution section ensures that whatever you build next, you have a clear path to getting it in front of the developers who will use it.

Dart is a powerful language for CLI development. Its strong typing, async support, native compilation, and pub.dev ecosystem make it a serious choice for building developer tooling, not just mobile apps.

The next step is building something that solves a real problem for you or your team, and shipping it.

Happy coding!!

But that's only the first 10% of a real payment system.

What happens after the customer pays? You need to record the purchase in your database, send a confirmation email, and grant product access (a GitHub repo invitation, an API key, a license file). You need to notify yourself as the admin. You need to handle refunds two weeks later and send recovery emails when someone abandons checkout.

This is the complete payment lifecycle, and it's where most SaaS applications break.

This article walks you through building the entire flow, from the "Buy" button to the "Welcome" email and everything in between. Every code example comes from a production application processing real payments. You'll see how to design the database schema, create Stripe products, build the checkout flow, process purchases reliably, handle refunds, recover abandoned carts, and send transactional emails.

Here is what you'll learn:

• How to design a database schema that tracks every stage of a purchase

• How to create Stripe products and prices programmatically

• How to build a checkout flow with success/cancel handling

• How to process webhooks securely with signature verification

• How to split post-payment processing into durable, independently retried steps

• How to handle full and partial refunds with automatic access revocation

• How to recover revenue from abandoned checkouts

• How to build transactional email templates with React Email and Resend

• How to test the entire flow locally with Stripe CLI and Inngest

Table of Contents

• Prerequisites

• How to Design the Payment Database Schema

• How to Create Stripe Products and Prices

• How to Build the Checkout Flow

• How to Handle Webhooks Securely

• How to Process Purchases with Durable Background Jobs

• How to Handle Refunds

• How to Recover Abandoned Checkouts

• How to Send Transactional Emails with React Email

• How to Test the Complete Flow Locally

• Conclusion

Prerequisites

To follow along, you should be familiar with:

• TypeScript and Node.js

• SQL databases (the examples use PostgreSQL)

• React (for email templates)

• Basic understanding of webhooks

You don't need prior experience with any of the specific libraries. This handbook explains each one as it appears.

What You Need Installed

Install these packages to run the code examples:

bun add stripe drizzle-orm @neondatabase/serverless inngest resend @react-email/components

You'll also need:

• A Stripe account (test mode is fine)

• A Neon PostgreSQL database (or any PostgreSQL instance)

• A Resend account for sending emails

• The Stripe CLI for local webhook testing

Environment Variables

Set up these environment variables in your .env file:

# Database
DATABASE_URL=postgresql://...

# Stripe
STRIPE_SECRET_KEY=sk_test_...
STRIPE_WEBHOOK_SECRET=whsec_...
STRIPE_PRO_PRICE_ID=price_...

# Email
RESEND_API_KEY=re_...
EMAIL_FROM="Your App <noreply@mail.yourapp.com>"
ADMIN_EMAIL=you@yourapp.com

# App
BETTER_AUTH_URL=http://localhost:3000

How to Design the Payment Database Schema

Before writing any Stripe code, you need a database schema that can track a purchase through every stage of its lifecycle: creation, completion, partial refund, and full refund.

A purchase starts as pending when the user clicks "Buy." After Stripe confirms payment, it transitions to completed. From there, it can move to refunded or partially_refunded. Pending purchases that are never completed expire after 24 hours (abandoned carts).

Here is the schema I use in production, defined with Drizzle ORM. The examples throughout this article grant access to a private GitHub repository because that's what this particular product sells.

Your "grant access" step will be different: upgrading a user to a Pro plan, provisioning API credits, unlocking course content, or activating a subscription. The schema fields and step logic change, but the durable execution pattern is the same.

// src/lib/db/schema.ts
import {
  boolean,
  integer,
  pgEnum,
  pgTable,
  text,
  timestamp,
  varchar,
} from "drizzle-orm/pg-core";

export const purchaseTierEnum = pgEnum("purchase_tier", ["pro"]);
export const purchaseStatusEnum = pgEnum("purchase_status", [
  "completed",
  "partially_refunded",
  "refunded",
]);

export const users = pgTable("users", {
  id: text("id").primaryKey(),
  email: varchar("email", { length: 255 }).notNull().unique(),
  emailVerified: boolean("email_verified").notNull().default(false),
  name: text("name"),
  image: text("image"),
  githubUsername: text("github_username"),
  createdAt: timestamp("created_at").notNull().defaultNow(),
  updatedAt: timestamp("updated_at").notNull().defaultNow(),
});

export const purchases = pgTable("purchases", {
  id: text("id")
    .primaryKey()
    .$defaultFn(() => crypto.randomUUID()),
  userId: text("user_id")
    .notNull()
    .references(() => users.id, { onDelete: "cascade" }),
  stripeCheckoutSessionId: text("stripe_checkout_session_id")
    .notNull()
    .unique(),
  stripeCustomerId: text("stripe_customer_id"),
  stripePaymentIntentId: text("stripe_payment_intent_id"),
  tier: purchaseTierEnum("tier").notNull(),
  status: purchaseStatusEnum("status").notNull().default("completed"),
  githubAccessGranted: boolean("github_access_granted")
    .notNull()
    .default(false),
  githubInvitationId: text("github_invitation_id"),
  amount: integer("amount").notNull(),
  currency: text("currency").notNull().default("usd"),
  purchasedAt: timestamp("purchased_at").notNull().defaultNow(),
  createdAt: timestamp("created_at").notNull().defaultNow(),
  updatedAt: timestamp("updated_at").notNull().defaultNow(),
});

export type Purchase = typeof purchases.$inferSelect;
export type NewPurchase = typeof purchases.$inferInsert;

Let me walk through the design decisions behind this schema.

Why Three Stripe ID Columns?

The purchases table stores three separate Stripe identifiers: stripeCheckoutSessionId, stripeCustomerId, and stripePaymentIntentId.

Each one serves a different purpose.

The checkout session ID is what you receive first. When a customer starts checkout, Stripe creates a session and gives you this ID. You use it to claim the purchase after the customer returns from Stripe's hosted checkout page.

The unique() constraint on this column is your idempotency guard. If someone tries to claim the same session twice, the database rejects the second insert.

The customer ID is Stripe's internal identifier for the buyer. You need this to look up the customer's payment history in Stripe's dashboard and to create future checkout sessions pre-filled with their billing info.

The payment intent ID is what Stripe sends in refund webhook events. When a charge.refunded event fires, it includes the payment intent ID but not the checkout session ID. Without storing this field, you would have no way to match a refund back to a purchase in your database.

Why Track Access State in Your Database

The githubAccessGranted and githubInvitationId fields might look unnecessary. You could check GitHub's API to see if a user has access. But querying an external API every time you need to check a user's access state is slow, rate-limited, and unreliable.

By tracking access state in your own database, you can answer "does this user have access?" with a single indexed query. You also know whether access was ever granted, which is critical for refund processing. If githubAccessGranted is false, you don't need to revoke anything on refund.

Why a Status Enum with Three Values?

The purchaseStatusEnum has three values: completed, partially_refunded, and refunded.

This matters for downstream logic. Your dashboard, analytics, support tools, and email sequences all need to know the exact state of a purchase. A partially refunded customer still has access, but a fully refunded customer doesn't.

If you only tracked "refunded" as a boolean, you would lose the distinction between partial and full refunds. That distinction affects whether you revoke product access.

How to Generate and Run Migrations

After defining your schema, generate a migration file and apply it to your database:

# Generate migration SQL from schema changes
bun run drizzle-kit generate

# Push schema directly (development only)
bun run drizzle-kit push

# Run migrations (production)
bun run drizzle-kit migrate

Drizzle Kit compares your TypeScript schema to the database and generates the SQL needed to bring them in sync. Review the generated migration file before running it in production. Schema changes are one of the few things you can't easily undo.

For development, drizzle-kit push is faster because it applies changes directly without creating migration files. For production, always use drizzle-kit generate followed by drizzle-kit migrate so you have a versioned record of every schema change.

How to Create Stripe Products and Prices

You can create products and prices through the Stripe dashboard, but managing them programmatically is better for reproducibility. Here's a seed script that creates everything you need:

// src/lib/payments/seed.ts
import { stripe } from "./index";

const PRODUCTS = [
  {
    name: "My SaaS Product",
    description: "Full access, one-time purchase",
    features: [
      "Full source code access",
      "Production-ready infrastructure",
      "Lifetime updates",
    ],
    metadata: { tier: "pro" },
    prices: [
      {
        lookupKey: "pro_one_time",
        unitAmount: 19900, // $199.00 in cents
        currency: "usd",
        nickname: "Pro One-Time",
      },
    ],
  },
];

async function main() {
  console.log("Seeding Stripe products and prices...\n");

  for (const config of PRODUCTS) {
    // Create or find product
    const products = await stripe.products.list({ active: true, limit: 100 });
    let product = products.data.find((p) => p.name === config.name);

    if (!product) {
      product = await stripe.products.create({
        name: config.name,
        description: config.description,
        marketing_features: config.features.map((f) => ({ name: f })),
        metadata: config.metadata,
      });
      console.log(`Created product "\({config.name}" (\){product.id})`);
    }

    // Create prices
    for (const priceConfig of config.prices) {
      const existing = await stripe.prices.list({
        lookup_keys: [priceConfig.lookupKey],
        active: true,
        limit: 1,
      });

      if (existing.data[0]) {
        console.log(`Price "${priceConfig.lookupKey}" already exists`);
        continue;
      }

      const price = await stripe.prices.create({
        product: product.id,
        unit_amount: priceConfig.unitAmount,
        currency: priceConfig.currency,
        nickname: priceConfig.nickname,
        lookup_key: priceConfig.lookupKey,
        transfer_lookup_key: true,
      });

      console.log(`Created price "\({priceConfig.lookupKey}" (\){price.id})`);
    }
  }

  console.log("\nDone! Add the price ID to your .env as STRIPE_PRO_PRICE_ID");
}

main().catch(console.error);

Run this with bun run src/lib/payments/seed.ts.

A few things worth noting.

• Use lookup_key instead of hardcoding price IDs: Price IDs are different between test and live mode. Lookup keys let you reference prices by name (pro_one_time) rather than by Stripe's generated ID (price_1P...).

  The transfer_lookup_key: true option ensures that if you create a new price with the same lookup key, it replaces the old one automatically.

• Prices are in cents: Stripe's API expects amounts in the smallest currency unit. For USD, that means 19900 represents $199.00.

  This is a common source of bugs. Always store amounts in cents in your database and convert to dollars only at the display layer.

• The seed script is idempotent: You can run it multiple times safely. It checks for existing products and prices before creating new ones.

How to Set Up the Stripe Client

The Stripe client uses lazy initialization so that importing it doesn't throw if the API key is missing at module load time. This matters in build environments where environment variables aren't set.

// src/lib/payments/index.ts
import Stripe from "stripe";

let stripeClient: Stripe | null = null;

function getStripe(): Stripe {
  if (!stripeClient) {
    const secretKey = process.env.STRIPE_SECRET_KEY;
    if (!secretKey) {
      throw new Error("STRIPE_SECRET_KEY is not set");
    }
    stripeClient = new Stripe(secretKey);
  }
  return stripeClient;
}

export const stripe = new Proxy({} as Stripe, {
  get(_, prop) {
    return Reflect.get(getStripe(), prop);
  },
});

The Proxy wrapper is the key pattern here. Code across your application imports stripe and calls methods like stripe.checkout.sessions.create(...). The proxy intercepts every property access and forwards it to the lazily initialized client.

This means the Stripe SDK only initializes when you actually use it, not when the module is imported.

How to Build the Checkout Flow

The checkout flow has three parts: creating the session, redirecting the customer, and handling the return.

How to Create a Checkout Session

Here's the function that creates a Stripe Checkout session for a one-time payment:

// src/lib/payments/index.ts
export async function createOneTimeCheckoutSession(params: {
  priceId: string;
  successUrl: string;
  cancelUrl: string;
  metadata: Record<string, string>;
  customerEmail?: string;
  couponId?: string;
}) {
  const client = getStripe();

  const session = await client.checkout.sessions.create({
    mode: "payment",
    line_items: [{ price: params.priceId, quantity: 1 }],
    success_url: params.successUrl,
    cancel_url: params.cancelUrl,
    metadata: params.metadata,
    ...(params.customerEmail && {
      customer_email: params.customerEmail,
    }),
    ...(params.couponId
      ? { discounts: [{ coupon: params.couponId }] }
      : { allow_promotion_codes: true }),
  });

  return session;
}

Three details matter here.

• The mode: "payment" setting tells Stripe this is a one-time charge, not a subscription. For subscriptions, you would use mode: "subscription". The mode affects which webhook events Stripe sends after payment.

• The metadata field is how you link the Stripe session back to your application. Pass your internal product tier, user ID, or any other data you need after payment. Stripe stores this metadata and includes it in webhook events and API responses.

• The allow_promotion_codes: true option shows a promo code field on the checkout page. If you have a specific coupon to apply (from a landing page URL parameter, for example), pass it via discounts instead. You can't use both at the same time.

How to Create the Checkout API Endpoint

Here's the API endpoint that creates a checkout session and returns the URL:

// src/server/api.ts
app.post("/api/payments/checkout", async ({ set }) => {
  const priceId = process.env.STRIPE_PRO_PRICE_ID;

  if (!priceId) {
    set.status = 500;
    return { error: "Price not configured" };
  }

  const baseUrl = process.env.BETTER_AUTH_URL ?? "http://localhost:3000";
  const tier = "pro";

  const checkoutSession = await createOneTimeCheckoutSession({
    priceId,
    successUrl: `${baseUrl}/dashboard?purchase=success&session_id={CHECKOUT_SESSION_ID}`,
    cancelUrl: `${baseUrl}/pricing`,
    metadata: { tier },
  });

  return { url: checkoutSession.url };
});

The {CHECKOUT_SESSION_ID} placeholder in the success URL is a Stripe template variable. Stripe replaces it with the actual session ID when redirecting the customer. This lets your frontend know which session just completed.

How to Claim the Purchase After Checkout

When the customer returns to your success URL, your frontend reads the session_id from the URL and sends it to a "claim" endpoint. This endpoint verifies the payment and creates the purchase record.

// src/server/api.ts
app.post(
  "/api/purchases/claim",
  async ({ body, request, set }) => {
    const session = await auth.api.getSession({
      headers: request.headers,
    });

    if (!session) {
      set.status = 401;
      return { error: "Unauthorized" };
    }

    const { sessionId } = body;

    // Check if this session was already claimed
    const existing = await db
      .select()
      .from(purchases)
      .where(eq(purchases.stripeCheckoutSessionId, sessionId))
      .limit(1);

    if (existing[0]) {
      return { success: true, alreadyClaimed: true, tier: existing[0].tier };
    }

    // Retrieve the Stripe checkout session to verify payment
    const stripeSession = await retrieveCheckoutSession(sessionId);

    if (stripeSession.payment_status !== "paid") {
      set.status = 400;
      return { error: "Payment not completed" };
    }

    const tier = (stripeSession.metadata?.tier ?? "pro") as PaymentTier;

    // Create purchase record
    await db.insert(purchases).values({
      userId: session.user.id,
      stripeCheckoutSessionId: sessionId,
      stripeCustomerId:
        typeof stripeSession.customer === "string"
          ? stripeSession.customer
          : stripeSession.customer?.id ?? null,
      stripePaymentIntentId:
        typeof stripeSession.payment_intent === "string"
          ? stripeSession.payment_intent
          : stripeSession.payment_intent?.id ?? null,
      tier,
      status: "completed",
      amount: stripeSession.amount_total ?? 0,
      currency: stripeSession.currency ?? "usd",
    });

    // Trigger background processing
    await inngest.send({
      name: "purchase/completed",
      data: {
        userId: session.user.id,
        tier,
        sessionId,
      },
    });

    return { success: true, tier };
  },
  {
    body: t.Object({
      sessionId: t.String(),
    }),
  }
);

This endpoint does four things, in order.

1. First, it checks if the session was already claimed. The unique() constraint on stripeCheckoutSessionId in the schema prevents duplicate records, but checking first lets you return a clean response without catching a database error.

2. Second, it verifies payment with Stripe. Never trust data from the client. The frontend passes the session ID, but you must call Stripe's API to confirm that payment_status is "paid".

3. Third, it creates the purchase record. Notice how it extracts the customer and payment_intent from the Stripe session. Both fields are returned as either strings or expanded objects depending on your Stripe API settings, so the ternary handles both cases.

4. Fourth, it sends a purchase/completed event to Inngest. This triggers the background processing flow that handles emails, access grants, analytics, and follow-up scheduling. The API endpoint doesn't do any of that work and returns { success: true } immediately.

This separation between recording the purchase and processing it is fundamental. The database insert is fast and reliable. The downstream processing (emails, API calls, analytics) is slow and unreliable.

By splitting them, you ensure the customer sees a success response instantly while the background work happens durably.

How to Handle Webhooks Securely

Your webhook endpoint is the entry point for Stripe events that happen outside your checkout flow: refunds, expired sessions, and disputes.

How to Verify Webhook Signatures

Every webhook from Stripe includes a signature header. You must verify this signature before processing the event. Without verification, anyone could send fake events to your webhook URL.

// src/lib/payments/index.ts
export async function constructWebhookEvent(
  payload: string | Buffer,
  signature: string
) {
  const webhookSecret = process.env.STRIPE_WEBHOOK_SECRET;
  if (!webhookSecret) {
    throw new Error("STRIPE_WEBHOOK_SECRET is not set");
  }
  const client = getStripe();
  return client.webhooks.constructEventAsync(payload, signature, webhookSecret);
}

One critical detail: use constructEventAsync instead of constructEvent. The async version uses the Web Crypto API, which is compatible with modern runtimes like Bun and Cloudflare Workers. The synchronous version depends on Node.js's crypto module, which isn't available everywhere.

Another critical detail: pass the raw request body to signature verification. If your framework parses the body as JSON before you access it, the signature check fails. The signature is computed over the raw bytes of the request, not the parsed JSON.

How to Build the Webhook Endpoint

Here is the production webhook handler. Its only job is to validate the event and route it to the background job system.

// src/server/api.ts
app.post("/api/payments/webhook", async ({ request, set }) => {
  const body = await request.text();
  const sig = request.headers.get("stripe-signature");

  if (!sig) {
    set.status = 400;
    return { error: "Missing signature" };
  }

  try {
    const event = await constructWebhookEvent(body, sig);
    console.log(`[Webhook] Received ${event.type}`);

    if (event.type === "charge.refunded") {
      const charge = event.data.object as {
        id: string;
        payment_intent: string;
        amount: number;
        amount_refunded: number;
        currency: string;
      };
      await inngest.send({
        name: "stripe/charge.refunded",
        data: {
          chargeId: charge.id,
          paymentIntentId: charge.payment_intent,
          amountRefunded: charge.amount_refunded,
          originalAmount: charge.amount,
          currency: charge.currency,
        },
      });
    }

    if (event.type === "checkout.session.expired") {
      const session = event.data.object as {
        id: string;
        customer_email: string | null;
      };
      await inngest.send({
        name: "stripe/checkout.session.expired",
        data: {
          sessionId: session.id,
          customerEmail: session.customer_email,
        },
      });
    }

    return { received: true };
  } catch (error) {
    console.error("[Webhook] Stripe verification failed:", error);
    set.status = 400;
    return { error: "Webhook verification failed" };
  }
});

This is the "thin webhook handler" pattern. Notice what it does not do: it does not query the database, send emails, grant access, or call any external service. It validates the signature, extracts the fields it needs, and sends a typed event to Inngest.

The entire handler completes in milliseconds.

Why does this matter? Stripe expects your webhook to return a 2xx response within about 20 seconds. If your handler tries to do too much work (database queries, email sends, API calls), it risks timing out.

Stripe marks it as failed and retries the entire event. Now you have partial completion and duplicate processing.

The thin handler avoids this entirely. Validate, enqueue, return. All the real work happens asynchronously in durable background functions.

Why Extract Fields Before Enqueueing?

You might notice that the webhook handler extracts specific fields from the Stripe event before sending them to Inngest:

await inngest.send({
  name: "stripe/charge.refunded",
  data: {
    chargeId: charge.id,
    paymentIntentId: charge.payment_intent,
    amountRefunded: charge.amount_refunded,
    originalAmount: charge.amount,
    currency: charge.currency,
  },
});

Why not forward the entire Stripe event? Two reasons.

First, Stripe event objects are large and deeply nested. Your background function only needs five fields. Sending the entire object means your durable function stores a large payload at every checkpoint, and over thousands of runs, this adds up.

Second, extracting fields at the boundary creates a clean contract between your webhook handler and your background functions. If Stripe changes the shape of their event objects in a future API version, you only need to update the extraction logic in the webhook handler. Your background functions keep working because they depend on your own typed data shape, not Stripe's.

How to Set Up Webhooks in Production

For production, you configure webhooks in the Stripe Dashboard:

1. Go to Stripe Dashboard, then Developers, then Webhooks.

2. Add an endpoint pointing to your production URL: https://yourapp.com/api/payments/webhook.

3. Select the events you want to receive: charge.refunded and checkout.session.expired.

4. Copy the signing secret and add it to your production environment variables as STRIPE_WEBHOOK_SECRET.

The production signing secret is different from the one the Stripe CLI generates for local testing. Make sure your environment variables are set correctly for each environment.

Which Webhook Events to Listen For

For a complete payment flow, you need these webhook events configured in Stripe:

For subscription-based billing, you would also listen for customer.subscription.updated, customer.subscription.deleted, and invoice.payment_failed. This article covers one-time payments, so the examples focus on the two events above.

The checkout.session.completed event is notably absent. For one-time payments, you typically process the purchase in the "claim" endpoint (shown in the previous section) rather than in a webhook, because you need the authenticated user's session to link the purchase to their account.

How to Process Purchases with Durable Background Jobs

This is the heart of the payment flow. After the purchase record is created and the purchase/completed event is sent, a durable function takes over and runs the entire post-payment workflow.

Each step in this function is individually checkpointed. If step 5 fails, steps 1 through 4 don't re-run. Step 5 retries on its own, and once it succeeds, steps 6 through 9 continue.

This is what "durable execution" means. It's the difference between a payment system that works in development and one that works in production.

I use Inngest for this. It is an event-driven durable execution platform that provides step-level checkpointing out of the box. You define functions with step.run() blocks, and Inngest handles retry logic, state persistence, and observability.

The Inngest client setup is minimal:

// src/lib/jobs/client.ts
import { Inngest } from "inngest";

export const inngest = new Inngest({
  id: "my-app",
});

Register your functions with the Inngest serve handler so the dev server (and production) can discover them:

import { serve } from "inngest/bun";
import { inngest } from "@/lib/jobs/client";
import { stripeFunctions } from "@/lib/jobs/functions/stripe";

const inngestHandler = serve({
  client: inngest,
  functions: [...stripeFunctions],
});

// Mount on your API
app.all("/api/inngest", async (ctx) => {
  return inngestHandler(ctx.request);
});

Here's the complete purchase function:

// src/lib/jobs/functions/stripe.ts
import { eq } from "drizzle-orm";
import { createElement } from "react";

import { inngest } from "../client";
import { trackServerEvent } from "@/lib/analytics/server";
import { brand } from "@/lib/brand";
import { db, purchases, users } from "@/lib/db";
import {
  sendEmail,
  PurchaseConfirmationEmail,
  AdminPurchaseNotificationEmail,
  RepoAccessGrantedEmail,
} from "@/lib/email";
import { addCollaborator } from "@/lib/github";

export const handlePurchaseCompleted = inngest.createFunction(
  { id: "purchase-completed", triggers: [{ event: "purchase/completed" }] },
  async ({ event, step }) => {
    const { userId, tier, sessionId } = event.data as {
      userId: string;
      tier: string;
      sessionId: string;
    };

    // Step 1: Look up user and purchase details
    const { user, purchase } = await step.run(
      "lookup-user-and-purchase",
      async () => {
        const userResult = await db
          .select({
            id: users.id,
            email: users.email,
            name: users.name,
            githubUsername: users.githubUsername,
          })
          .from(users)
          .where(eq(users.id, userId))
          .limit(1);

        const foundUser = userResult[0];
        if (!foundUser) {
          throw new Error(`User not found: ${userId}`);
        }

        const purchaseResult = await db
          .select({
            amount: purchases.amount,
            currency: purchases.currency,
            stripePaymentIntentId: purchases.stripePaymentIntentId,
          })
          .from(purchases)
          .where(eq(purchases.stripeCheckoutSessionId, sessionId))
          .limit(1);

        const foundPurchase = purchaseResult[0];

        return {
          user: foundUser,
          purchase: foundPurchase ?? {
            amount: 0,
            currency: "usd",
            stripePaymentIntentId: null,
          },
        };
      }
    );

    // Step 2: Track purchase in analytics
    await step.run("track-purchase-to-posthog", async () => {
      try {
        await trackServerEvent(userId, "purchase_completed_server", {
          tier,
          amount_cents: purchase.amount,
          currency: purchase.currency,
          stripe_session_id: sessionId,
          stripe_payment_intent_id: purchase.stripePaymentIntentId,
        });
      } catch (error) {
        console.error(`Failed to track to PostHog:`, error);
      }
    });

    // Step 3: Send purchase confirmation to customer
    await step.run("send-purchase-confirmation", async () => {
      await sendEmail({
        to: user.email,
        subject: `Your ${brand.name} purchase is confirmed!`,
        template: createElement(PurchaseConfirmationEmail, {
          amount: purchase.amount,
          currency: purchase.currency,
          customerEmail: user.email,
        }),
      });
    });

    // Step 4: Send admin notification
    await step.run("send-admin-notification", async () => {
      const adminEmail = process.env.ADMIN_EMAIL;
      if (!adminEmail) return;

      await sendEmail({
        to: adminEmail,
        subject: `New template sale: ${user.email}`,
        template: createElement(AdminPurchaseNotificationEmail, {
          amount: purchase.amount,
          currency: purchase.currency,
          customerEmail: user.email,
          customerName: user.name,
          stripeSessionId: purchase.stripePaymentIntentId ?? sessionId,
        }),
      });
    });

    // Early return if user has no GitHub username
    if (!user.githubUsername) {
      return { success: true, userId, tier, githubAccessGranted: false };
    }

    // Step 5: Grant GitHub repository access
    const collaboratorResult = await step.run(
      "add-github-collaborator",
      async () => {
        return addCollaborator(user.githubUsername!);
      }
    );

    // Step 6: Track GitHub access granted
    await step.run("track-github-access", async () => {
      await trackServerEvent(userId, "github_access_granted", {
        tier,
        github_username: user.githubUsername,
        invitation_status: collaboratorResult.status,
      });
    });

    // Step 7: Update purchase record
    await step.run("update-purchase-record", async () => {
      await db
        .update(purchases)
        .set({
          githubAccessGranted: true,
          githubInvitationId: collaboratorResult.status,
          updatedAt: new Date(),
        })
        .where(eq(purchases.stripeCheckoutSessionId, sessionId));
    });

    // Step 8: Send repo access email
    await step.run("send-repo-access-email", async () => {
      const repoUrl = brand.social.github;
      await sendEmail({
        to: user.email,
        subject: `Your ${brand.name} repository access is ready!`,
        template: createElement(RepoAccessGrantedEmail, { repoUrl }),
      });
    });

    // Step 9: Schedule follow-up email sequence
    await step.run("schedule-follow-up", async () => {
      const purchaseRecord = await db
        .select({ id: purchases.id })
        .from(purchases)
        .where(eq(purchases.stripeCheckoutSessionId, sessionId))
        .limit(1);

      if (purchaseRecord[0]) {
        await inngest.send({
          name: "purchase/follow-up.scheduled",
          data: {
            userId,
            purchaseId: purchaseRecord[0].id,
            tier,
          },
        });
      }
    });

    return { success: true, userId, tier, githubAccessGranted: true };
  }
);

That's a lot of code. Let me break down why each step exists and why it must be separate.

Step 1: Look Up User and Purchase

const { user, purchase } = await step.run(
  "lookup-user-and-purchase",
  async () => {
    // Database queries for user and purchase records
    return { user: foundUser, purchase: foundPurchase };
  }
);

This step queries the database for the user and purchase details. Every subsequent step depends on these values (the user's email, the purchase amount, the user's GitHub username).

Because this is wrapped in step.run(), the return value is cached by Inngest. If a later step fails and the function retries, this step doesn't re-run. The cached values are replayed instead.

If the user doesn't exist in the database, this step throws an error that halts the entire function. There's no point continuing if the user can't be found.

Step 2: Track Analytics

await step.run("track-purchase-to-posthog", async () => {
  try {
    await trackServerEvent(userId, "purchase_completed_server", {
      tier,
      amount_cents: purchase.amount,
      currency: purchase.currency,
    });
  } catch (error) {
    console.error(`Failed to track to PostHog:`, error);
  }
});

Analytics tracking gets its own step because analytics services have their own failure modes. PostHog could be rate-limited or temporarily unreachable. If that happens, you don't want it to block the confirmation email.

Notice the try-catch. A tracking failure logs the error but doesn't halt the function. Analytics data is valuable but not critical to the purchase flow.

Steps 3 and 4: Email Notifications

The customer confirmation and admin notification are separate steps because they are independent operations. If Resend returns a 500 when sending the admin email, the customer should still get their confirmation.

// Step 3: Customer confirmation
await step.run("send-purchase-confirmation", async () => {
  await sendEmail({
    to: user.email,
    subject: `Your ${brand.name} purchase is confirmed!`,
    template: createElement(PurchaseConfirmationEmail, {
      amount: purchase.amount,
      currency: purchase.currency,
      customerEmail: user.email,
    }),
  });
});

// Step 4: Admin notification
await step.run("send-admin-notification", async () => {
  const adminEmail = process.env.ADMIN_EMAIL;
  if (!adminEmail) return;

  await sendEmail({
    to: adminEmail,
    subject: `New template sale: ${user.email}`,
    template: createElement(AdminPurchaseNotificationEmail, {
      // ... admin-specific fields
    }),
  });
});

The admin notification step includes a guard: if ADMIN_EMAIL isn't set, it returns early. This makes the function work in development environments where you haven't configured all environment variables.

Step 5: Grant Product Access

if (!user.githubUsername) {
  return { success: true, userId, tier, githubAccessGranted: false };
}

const collaboratorResult = await step.run(
  "add-github-collaborator",
  async () => {
    return addCollaborator(user.githubUsername!);
  }
);

This is the step most likely to fail. GitHub's API has rate limits, can time out, and the user's GitHub username might be invalid.

By making it its own step, a GitHub API failure doesn't re-trigger the confirmation email (step 3) or the admin notification (step 4). Those are already checkpointed.

Notice the early return before step 5. If the user has no GitHub username linked, the function returns after step 4. The remaining steps only run when there's a GitHub account to grant access to.

Steps 6-7: Track and Update

After granting GitHub access, the function tracks the event in analytics (step 6) and updates the purchase record in the database (step 7).

The database update is intentionally ordered after the GitHub API call. You only set githubAccessGranted: true after the invitation actually succeeded. If you updated the record first and the GitHub step failed, your database would say access was granted when it was not.

Step 8: Send Access Email

await step.run("send-repo-access-email", async () => {
  const repoUrl = brand.social.github;
  await sendEmail({
    to: user.email,
    subject: `Your ${brand.name} repository access is ready!`,
    template: createElement(RepoAccessGrantedEmail, { repoUrl }),
  });
});

This email only sends after the GitHub invitation is confirmed. The ordering is deliberate. You don't tell the customer "your access is ready" if the invitation hasn't been sent.

Step 9: Schedule Follow-Up Sequence

await step.run("schedule-follow-up", async () => {
  const purchaseRecord = await db
    .select({ id: purchases.id })
    .from(purchases)
    .where(eq(purchases.stripeCheckoutSessionId, sessionId))
    .limit(1);

  if (purchaseRecord[0]) {
    await inngest.send({
      name: "purchase/follow-up.scheduled",
      data: {
        userId,
        purchaseId: purchaseRecord[0].id,
        tier,
      },
    });
  }
});

The final step triggers a separate function that handles the follow-up email sequence: day 7 onboarding tips, day 14 feedback request, day 30 testimonial request. This is an event-driven chain: one function completes and triggers another.

The follow-up function uses step.sleep() to wait between emails without consuming compute resources:

export const handlePurchaseFollowUp = inngest.createFunction(
  {
    id: "purchase-follow-up",
    triggers: [{ event: "purchase/follow-up.scheduled" }],
    cancelOn: [
      {
        event: "purchase/follow-up.cancelled",
        match: "data.purchaseId",
      },
    ],
  },
  async ({ event, step }) => {
    await step.sleep("wait-7-days", "7d");
    await step.run("send-day-7-email", async () => {
      // Send onboarding tips
    });

    await step.sleep("wait-14-days", "7d");
    await step.run("send-day-14-email", async () => {
      // Send feedback request
    });
  }
);

The cancelOn option is worth noting. If the purchase is refunded, you send a purchase/follow-up.cancelled event, and the entire follow-up sequence stops. No stale emails to customers who refunded.

The Rule for Step Separation

Any operation that calls an external service or could fail independently should be its own step. A database query is a step because the database can be temporarily unreachable. An email send or API call is a step because those services can return errors or hit rate limits.

If two operations always succeed or fail together, they can share a step. But when in doubt, make it separate. The overhead is negligible, and the reliability gain is significant.

How to Handle Refunds

Refund processing is the most commonly overlooked part of a payment system. You need to handle two cases: full refunds (revoke access) and partial refunds (keep access, update status).

Here's the complete refund handler:

// src/lib/jobs/functions/stripe.ts
export const handleRefund = inngest.createFunction(
  { id: "refund-processed", triggers: [{ event: "stripe/charge.refunded" }] },
  async ({ event, step }) => {
    const data = event.data as {
      chargeId: string;
      paymentIntentId: string;
      amountRefunded: number;
      originalAmount: number;
      currency: string;
    };

    const chargeId = data.chargeId;
    const paymentIntentId = data.paymentIntentId;
    const currency = data.currency;
    const amountRefunded = data.amountRefunded;
    const originalAmount = data.originalAmount;
    const isFullRefund = amountRefunded >= originalAmount;

    // Step 1: Look up the purchase and user
    const { user, purchase } = await step.run(
      "lookup-purchase-by-payment-intent",
      async () => {
        const purchaseResult = await db
          .select({
            id: purchases.id,
            userId: purchases.userId,
            stripePaymentIntentId: purchases.stripePaymentIntentId,
            githubAccessGranted: purchases.githubAccessGranted,
          })
          .from(purchases)
          .where(eq(purchases.stripePaymentIntentId, paymentIntentId))
          .limit(1);

        const foundPurchase = purchaseResult[0];
        if (!foundPurchase) {
          return { user: null, purchase: null };
        }

        const userResult = await db
          .select({
            id: users.id,
            email: users.email,
            name: users.name,
            githubUsername: users.githubUsername,
          })
          .from(users)
          .where(eq(users.id, foundPurchase.userId))
          .limit(1);

        return { user: userResult[0] ?? null, purchase: foundPurchase };
      }
    );

    if (!purchase || !user) {
      return { success: false, reason: "no_matching_purchase" };
    }

    let accessRevoked = false;

    // Step 2: Revoke GitHub access (only for full refunds)
    if (isFullRefund && user.githubUsername && purchase.githubAccessGranted) {
      const revokeResult = await step.run(
        "revoke-github-access",
        async () => {
          return removeCollaborator(user.githubUsername!);
        }
      );
      accessRevoked = revokeResult.success;
    }

    // Step 3: Update purchase status
    await step.run("update-purchase-status", async () => {
      if (isFullRefund) {
        await db
          .update(purchases)
          .set({
            status: "refunded",
            githubAccessGranted: false,
            updatedAt: new Date(),
          })
          .where(eq(purchases.id, purchase.id));
      } else {
        await db
          .update(purchases)
          .set({
            status: "partially_refunded",
            updatedAt: new Date(),
          })
          .where(eq(purchases.id, purchase.id));
      }
    });

    // Step 4: Track refund in analytics
    await step.run("track-refund-event", async () => {
      try {
        await trackServerEvent(user.id, "refund_processed", {
          charge_id: chargeId,
          payment_intent_id: paymentIntentId,
          amount_cents: amountRefunded,
          original_amount_cents: originalAmount,
          currency,
          is_full_refund: isFullRefund,
          github_access_revoked: accessRevoked,
        });
      } catch (error) {
        console.error(`Failed to track to PostHog:`, error);
      }
    });

    // Step 5: Notify customer
    await step.run("send-customer-notification", async () => {
      if (isFullRefund) {
        await sendEmail({
          to: user.email,
          subject: `Your ${brand.name} refund has been processed`,
          template: createElement(AccessRevokedEmail, {
            customerEmail: user.email,
            refundAmount: amountRefunded,
            currency,
          }),
        });
      } else {
        await sendEmail({
          to: user.email,
          subject: `Your ${brand.name} partial refund has been processed`,
          template: createElement(PartialRefundEmail, {
            customerEmail: user.email,
            refundAmount: amountRefunded,
            originalAmount,
            currency,
          }),
        });
      }
    });

    // Step 6: Notify admin
    await step.run("send-admin-notification", async () => {
      const adminEmail = process.env.ADMIN_EMAIL;
      if (!adminEmail) return;

      await sendEmail({
        to: adminEmail,
        subject: `\({isFullRefund ? "Full" : "Partial"} refund processed: \){user.email}`,
        template: createElement(AdminRefundNotificationEmail, {
          customerEmail: user.email,
          customerName: user.name,
          githubUsername: user.githubUsername,
          refundAmount: amountRefunded,
          originalAmount,
          currency,
          stripeChargeId: chargeId,
          accessRevoked,
          isPartialRefund: !isFullRefund,
        }),
      });
    });

    return { success: true, accessRevoked, isFullRefund, userId: user.id };
  }
);

How Full Refunds Differ from Partial Refunds

The function distinguishes between the two with a simple comparison:

const isFullRefund = amountRefunded >= originalAmount;

For a full refund, three things happen:

1. GitHub access is revoked (the removeCollaborator call).

2. The purchase status is set to "refunded".

3. The customer receives an AccessRevokedEmail explaining that their access has been removed.

For a partial refund, the customer keeps access:

1. GitHub access is not revoked.

2. The purchase status is set to "partially_refunded".

3. The customer receives a PartialRefundEmail showing the refunded amount and the original amount.

This distinction matters for your database integrity. Downstream systems (your dashboard, analytics, support tools) need accurate status values. A partially_refunded purchase still represents an active customer.

How Conditional Steps Work

The "revoke GitHub access" step only runs when three conditions are all true: it's a full refund, the user has a GitHub username, and access was previously granted.

if (isFullRefund && user.githubUsername && purchase.githubAccessGranted) {
  const revokeResult = await step.run("revoke-github-access", async () => {
    return removeCollaborator(user.githubUsername!);
  });
  accessRevoked = revokeResult.success;
}

If any of those conditions is false, the step is skipped entirely. Inngest handles this cleanly. The function continues to step 3 (update purchase status) with accessRevoked still set to false.

How to Recover Abandoned Checkouts

When a customer starts checkout but doesn't complete it, Stripe eventually expires the session (after 24 hours by default). You can listen for this event and send a recovery email.

The key insight is that you don't want to send the email immediately. Give the customer an hour to come back on their own.

// src/lib/jobs/functions/stripe.ts
export const handleCheckoutExpired = inngest.createFunction(
  {
    id: "checkout-expired",
    triggers: [{ event: "stripe/checkout.session.expired" }],
  },
  async ({ event, step }) => {
    const { customerEmail, sessionId } = event.data as {
      customerEmail: string | null;
      sessionId: string;
    };

    if (!customerEmail) {
      return { success: false, reason: "no_email" };
    }

    // Wait 1 hour before sending recovery email
    await step.sleep("wait-before-recovery-email", "1h");

    // Send abandoned cart email
    await step.run("send-abandoned-cart-email", async () => {
      const baseUrl =
        process.env.BETTER_AUTH_URL ?? "https://your-app.com";
      const checkoutUrl = `${baseUrl}/pricing`;

      await sendEmail({
        to: customerEmail,
        subject: `Your ${brand.name} checkout is waiting`,
        template: createElement(AbandonedCartEmail, {
          customerEmail,
          checkoutUrl,
        }),
      });
    });

    // Track the recovery attempt
    await step.run("track-abandoned-cart", async () => {
      try {
        await trackServerEvent("anonymous", "abandoned_cart_email_sent", {
          customer_email: customerEmail,
          session_id: sessionId,
        });
      } catch (error) {
        console.error(`Failed to track to PostHog:`, error);
      }
    });

    return { success: true, customerEmail };
  }
);

The step.sleep("wait-before-recovery-email", "1h") line pauses the function for one hour without consuming compute resources. Inngest schedules the function to resume after the delay. No cron jobs, no Redis queues, no setTimeout that gets lost when your server restarts.

There is a guard at the top of the function. If the checkout session has no customer email (the customer closed the page before entering their email), the function returns early. You can't send a recovery email without an address.

You could extend this pattern with a second sleep and follow-up email three days later. You could also check if the customer has since completed a purchase (by querying the database in a step.run()) and skip the email if they have.

Why One Hour Is the Right Delay

Sending the recovery email immediately after checkout expiration feels aggressive. The customer might still be comparing options, waiting for payday, or just distracted. An immediate email says "we noticed you left," which feels surveillance-like.

Waiting 24 hours is too long. The customer has moved on. They have forgotten your product or found an alternative.

One hour is the sweet spot I found through testing. The customer's intent is still fresh, and the email feels helpful rather than pushy.

Your mileage may vary. The delay is configurable: change "1h" to "30m" or "3h" and redeploy.

Why This Is Better Than a Cron Job

Without durable execution, abandoned cart recovery typically works like this: a cron job runs every hour, queries the database for expired sessions that haven't been recovered yet, sends emails to each one, and marks them as recovered.

This approach has several problems. You need a recovered_at column to avoid sending duplicate emails. You need to handle the case where the cron job crashes halfway through the batch, and you need to tune the cron interval carefully.

The step.sleep() approach eliminates all of this. Each expired session gets its own function instance with its own timer. There's no batch processing, no database flag, and no duplicate risk.

How to Send Transactional Emails with React Email

Every email in the payment flow is a React component rendered to HTML and sent via Resend. This gives you type-safe templates with props, component reuse, and the ability to preview emails in your browser during development.

How to Set Up the Email Client

The email client wraps Resend with a simple sendEmail function:

// src/lib/email/index.ts
import { render } from "@react-email/components";
import type { ReactElement } from "react";
import { Resend } from "resend";

import { brand } from "@/lib/brand";

let resendClient: Resend | null = null;

function getResend(): Resend {
  if (!resendClient) {
    const apiKey = process.env.RESEND_API_KEY;
    if (!apiKey) {
      throw new Error("RESEND_API_KEY is not set");
    }
    resendClient = new Resend(apiKey);
  }
  return resendClient;
}

interface SendEmailOptions {
  to: string | string[];
  subject: string;
  template: ReactElement;
  from?: string;
  replyTo?: string;
}

export async function sendEmail({
  to,
  subject,
  template,
  from = process.env.EMAIL_FROM ?? brand.emails.from,
  replyTo,
}: SendEmailOptions) {
  const resend = getResend();
  const html = await render(template);

  return resend.emails.send({
    from,
    to,
    subject,
    html,
    replyTo,
  });
}

The render() function from @react-email/components converts a React element into an HTML string. This HTML is what Resend delivers to the customer's inbox.

The from address defaults to your brand's email configuration. You need a verified domain in Resend for this to work. During development, Resend's free tier lets you send to your own email address without domain verification.

How to Build a Purchase Confirmation Template

Here's the real purchase confirmation email template:

// src/lib/email/emails/purchase-confirmation.tsx
import {
  Body,
  Container,
  Head,
  Heading,
  Hr,
  Html,
  Link,
  Preview,
  Section,
  Text,
} from "@react-email/components";

import { brand } from "@/lib/brand";

interface PurchaseConfirmationEmailProps {
  amount: number;
  currency: string;
  customerEmail: string;
}

const colors = {
  primary: "#d97757",
  background: "#faf9f5",
  foreground: "#30302e",
  muted: "#6b6860",
  border: "#e5e4df",
  card: "#ffffff",
  success: "#16a34a",
  successLight: "#f0fdf4",
};

export default function PurchaseConfirmationEmail({
  amount,
  currency,
  customerEmail,
}: PurchaseConfirmationEmailProps) {
  const formattedAmount = new Intl.NumberFormat("en-US", {
    style: "currency",
    currency: currency.toUpperCase(),
  }).format(amount / 100);

  return (
    <Html>
      <Head />
      <Preview>Your {brand.name} purchase is confirmed!</Preview>
      <Body style={main}>
        <Container style={container}>
          <Section style={header}>
            <Text style={logoText}>{brand.name}</Text>
          </Section>

          <Hr style={divider} />

          <Section style={successBadge}>
            <Text style={successText}>Payment Successful</Text>
          </Section>

          <Heading style={h1}>Thank you for your purchase!</Heading>

          <Text style={text}>
            Your payment has been processed successfully. We are now setting
            up your GitHub repository access. You will receive another email
            shortly with your access link.
          </Text>

          <Section style={detailsBox}>
            <Text style={detailsTitle}>Order Details</Text>

            <Section style={detailRow}>
              <Text style={detailLabel}>Product</Text>
              <Text style={detailValue}>{brand.name}</Text>
            </Section>

            <Section style={detailRow}>
              <Text style={detailLabel}>Amount</Text>
              <Text style={detailValue}>{formattedAmount}</Text>
            </Section>

            <Section style={detailRow}>
              <Text style={detailLabel}>Email</Text>
              <Text style={detailValue}>{customerEmail}</Text>
            </Section>
          </Section>

          <Text style={text}>
            This is a one-time purchase. No recurring charges will be made.
          </Text>

          <Hr style={divider} />

          <Text style={footer}>
            Questions about your purchase? Reply to this email or reach
            out at{" "}
            <Link
              href={`mailto:${brand.emails.support}`}
              style={link}
            >
              {brand.emails.support}
            </Link>
          </Text>
        </Container>
      </Body>
    </Html>
  );
}

PurchaseConfirmationEmail.PreviewProps = {
  amount: 9900,
  currency: "usd",
  customerEmail: "customer@example.com",
} satisfies PurchaseConfirmationEmailProps;

A few things to note about this template.

• Currency formatting happens in the template: The amount prop is in cents (the same format stored in your database and returned by Stripe). The Intl.NumberFormat call converts it to a human-readable string like "$99.00" and keeps currency formatting logic in one place.

• The PreviewProps object is for development. React Email uses these props to render a preview in the browser. The satisfies keyword ensures the preview props match the component's interface.

• All styles are inline objects. Email clients strip <style> tags and ignore most CSS. Inline styles are the only reliable way to style emails across Gmail, Outlook, Apple Mail, and every other client.

How to Build a Repo Access Template

The repo access email is sent after the GitHub invitation succeeds:

// src/lib/email/emails/repo-access-granted.tsx
import {
  Body,
  Button,
  Container,
  Head,
  Heading,
  Hr,
  Html,
  Link,
  Preview,
  Section,
  Text,
} from "@react-email/components";

import { brand } from "@/lib/brand";

interface RepoAccessGrantedEmailProps {
  repoUrl: string;
}

export default function RepoAccessGrantedEmail({
  repoUrl,
}: RepoAccessGrantedEmailProps) {
  return (
    <Html>
      <Head />
      <Preview>Your {brand.name} repository access is ready!</Preview>
      <Body style={main}>
        <Container style={container}>
          <Section style={header}>
            <Text style={logoText}>{brand.name}</Text>
          </Section>

          <Hr style={divider} />

          <Heading style={h1}>You are in!</Heading>

          <Text style={text}>
            Your GitHub repository access has been granted. You now have
            full access to the {brand.name} codebase.
          </Text>

          <Section style={buttonContainer}>
            <Button style={button} href={repoUrl}>
              Open Repository
            </Button>
          </Section>

          <Section style={infoBox}>
            <Text style={infoTitle}>Quick Start</Text>
            <Text style={infoText}>
              <strong>1.</strong> Clone the repository to your machine
            </Text>
            <Text style={infoText}>
              <strong>2.</strong> Run{" "}
              <code style={codeStyle}>bun install</code> to install
              dependencies
            </Text>
            <Text style={infoText}>
              <strong>3.</strong> Follow the README for environment setup
            </Text>
            <Text style={infoText}>
              <strong>4.</strong> Run{" "}
              <code style={codeStyle}>bun dev</code> to start building
            </Text>
          </Section>

          <Hr style={divider} />

          <Text style={footer}>
            Need help? Reply to this email or reach out at{" "}
            <Link
              href={`mailto:${brand.emails.support}`}
              style={link}
            >
              {brand.emails.support}
            </Link>
          </Text>
        </Container>
      </Body>
    </Html>
  );
}

This template includes a <Button> component that links directly to the GitHub repository. The quick start section gives the customer immediate next steps so they aren't left wondering what to do after gaining access.

How to Build an Abandoned Cart Template

The abandoned cart email brings the customer back to your pricing page:

// src/lib/email/emails/abandoned-cart.tsx
import {
  Body,
  Button,
  Container,
  Head,
  Heading,
  Hr,
  Html,
  Preview,
  Section,
  Text,
} from "@react-email/components";

import { brand } from "@/lib/brand";

interface AbandonedCartEmailProps {
  customerEmail: string;
  checkoutUrl: string;
}

export default function AbandonedCartEmail({
  customerEmail,
  checkoutUrl,
}: AbandonedCartEmailProps) {
  return (
    <Html>
      <Head />
      <Preview>Your {brand.name} checkout is waiting for you</Preview>
      <Body style={main}>
        <Container style={container}>
          <Section style={header}>
            <Text style={logoText}>{brand.name}</Text>
          </Section>

          <Hr style={divider} />

          <Heading style={h1}>You left something behind</Heading>

          <Text style={text}>
            We noticed you started a checkout but did not complete your
            purchase. No worries. Your cart is still waiting for you.
          </Text>

          <Text style={text}>
            {brand.name} gives you everything you need to ship your
            startup this weekend: authentication, payments, email,
            background jobs, and more. All wired together and ready
            to go.
          </Text>

          <Section style={buttonContainer}>
            <Button style={button} href={checkoutUrl}>
              Complete Your Purchase
            </Button>
          </Section>

          <Text style={textSmall}>
            If you ran into any issues during checkout or have questions
            about {brand.name}, just reply to this email. I read every
            message personally.
          </Text>

          <Hr style={divider} />

          <Text style={footer}>
            This email was sent to {customerEmail} because you started
            a checkout on {brand.name}. If this was not you, you can
            safely ignore this email.
          </Text>
        </Container>
      </Body>
    </Html>
  );
}

The tone matters here. "You left something behind" is friendly, not pushy. The email explains the product's value briefly, includes a single clear call to action, and the footer explains why they received the email.

How Templates Integrate with Durable Steps

Every email template is invoked via createElement inside a step.run() block:

await step.run("send-purchase-confirmation", async () => {
  await sendEmail({
    to: user.email,
    subject: `Your ${brand.name} purchase is confirmed!`,
    template: createElement(PurchaseConfirmationEmail, {
      amount: purchase.amount,
      currency: purchase.currency,
      customerEmail: user.email,
    }),
  });
});

The createElement call creates a React element from the template component with the given props. The sendEmail function renders it to HTML via React Email's render() and sends it through Resend.

Because this is inside a step.run(), the email send is checkpointed. If Resend is down and the step fails, it retries on its own without re-running previous steps. The customer never gets a duplicate email.

How to Test the Complete Flow Locally

Testing the complete payment lifecycle locally requires three things running simultaneously: your application, the Stripe CLI forwarding webhook events, and the Inngest dev server processing background jobs.

Step 1: Start the Stripe CLI

Install the Stripe CLI and log in:

# macOS
brew install stripe/stripe-cli/stripe

# Authenticate
stripe login

Forward webhook events to your local server:

stripe listen --forward-to localhost:3000/api/payments/webhook

The CLI prints a webhook signing secret starting with whsec_. Copy this to your .env as STRIPE_WEBHOOK_SECRET.

Step 2: Start the Inngest Dev Server

The Inngest dev server gives you real-time visibility into every function execution, every step, and every retry:

npx inngest-cli@latest dev -u http://localhost:3000/api/inngest

Open http://localhost:8288 in your browser. This is the Inngest dashboard where you'll watch your durable functions execute step by step.

Step 3: Start Your Application

bun run dev

Your application should now be running on http://localhost:3000.

Step 4: Test the Purchase Flow

1. Go to your pricing page and click the checkout button.

2. Use Stripe's test card number 4242 4242 4242 4242 with any future expiration date and any CVC.

3. Complete the checkout. Stripe redirects you to your success URL.

4. Your frontend calls the /api/purchases/claim endpoint with the session ID.

5. Watch the Inngest dashboard. You should see the purchase-completed function trigger and each step execute in sequence.

In the Inngest dashboard, you will see:

• Step 1: "lookup-user-and-purchase" completes with the user and purchase data.

• Step 2: "track-purchase-to-posthog" completes (or logs a warning if PostHog isn't configured).

• Step 3: "send-purchase-confirmation" completes. Check your email.

• Step 4: "send-admin-notification" completes (if ADMIN_EMAIL is set).

• Steps 5-9: Run if the user has a GitHub username linked.

Step 5: Test a Refund

Trigger a refund through the Stripe CLI:

stripe trigger charge.refunded

Or go to the Stripe dashboard, find the test payment, and issue a refund manually. The Stripe CLI will forward the charge.refunded webhook to your local server.

In the Inngest dashboard, you'll see the refund-processed function trigger with its own set of steps: lookup, conditional access revocation, status update, analytics tracking, and email notifications.

Step 6: Test Abandoned Cart Recovery

Trigger a checkout expiration:

stripe trigger checkout.session.expired

The checkout-expired function will appear in the Inngest dashboard. You'll see the 1-hour sleep step. In the dev server, you can fast-forward through sleeps by clicking the "Skip" button in the dashboard. This lets you test the delayed email without actually waiting an hour.

How to Simulate Step Failures

To test the retry behavior, temporarily throw an error in one of your steps:

const collaboratorResult = await step.run(
  "add-github-collaborator",
  async () => {
    throw new Error("Simulated GitHub API failure");
  }
);

In the Inngest dashboard, you'll see:

• Steps 1 through 4 succeed and their results are cached.

• Step 5 fails and is retried with exponential backoff.

• Steps 6 through 9 remain pending.

Remove the thrown error, and on the next retry, step 5 succeeds. Steps 6 through 9 execute, while steps 1 through 4 aren't re-executed. This is the checkpointing behavior that makes durable execution reliable.

Conclusion

Building a complete SaaS payment flow is more than integrating Stripe Checkout. It's the entire lifecycle from "Buy" button to "Welcome" email, including the parts that happen when things go wrong.

Here's what you built in this tutorial:

• A database schema that tracks purchases through every state: completed, partially refunded, and fully refunded.

• A Stripe product and price seed script that creates your catalog programmatically.

• A checkout flow with session creation, payment verification, and idempotent purchase claiming.

• A thin webhook handler that validates signatures and routes events to background jobs.

• A 9-step durable purchase function where each step is independently checkpointed and retried.

• A refund handler that distinguishes between full and partial refunds, revoking access only when appropriate.

• An abandoned cart recovery flow that waits an hour before sending a friendly recovery email.

• Three transactional email templates built with React Email: purchase confirmation, repo access granted, and abandoned cart.

• A local testing setup with Stripe CLI, Inngest dev server, and step-by-step observability.

The most important pattern is the separation between receiving and processing. Your API endpoints and webhook handlers should be thin: validate, record, enqueue, return. All the complex multi-step work happens in durable background functions where failures are isolated and retried at the step level.

This pattern scales. Add a new step to the purchase flow, and it gets the same checkpointing and retry behavior. Add a new webhook event, and you route it to a new durable function.

Your requirements may differ. You might sell subscriptions instead of one-time purchases, or provision API keys instead of GitHub access. The specific steps change, but the architecture stays the same.

If you want to start with all of these patterns already wired together in a production-ready codebase, Eden Stack includes the complete payment flow described in this article, along with 30+ additional production-tested patterns for authentication, email, analytics, background jobs, and more.

Magnus Rødseth builds AI-native applications and is the creator of Eden Stack, a production-ready starter kit with 30+ Claude skills encoding production patterns for AI-native SaaS development.

And honestly? It was a fair point. You'd write the contract — the interface, the abstract class, the protocol — and then write the implementation. Two files where one would do. That's more surface area, more indirection, and more to maintain.

The Ruby and Rails communities built an entire philosophy around this: convention over configuration, less ceremony, fewer keystrokes. If the framework could infer your intent, why spell it out?

Then AI happened.

I was recently chatting with a CEO about what current-generation software engineers get wrong, and he put it cleanly:

"Abstract interfaces were challenging a few months ago just because it required twice as much code. But with AI, lines of code are free. The reason we still need such constructs is because at some point a human still needs to look at the code. Interfaces reduce the cognitive load."

That framing stuck with me. The cost of writing code has collapsed. The cost of reading it hasn't moved. And that asymmetry changes everything about how you should think about abstraction.

Here's what I mean.

Table of Contents

• Your Brain Is the Bottleneck

• The Greats Already Knew This

• The Economics Have Flipped

• The Data Backs It Up

• The Contrarian Case (And Why It Actually Agrees)

• What This Means for You

• References

Your Brain Is the Bottleneck

This isn't a vibes argument. There's actual neuroscience behind why interfaces help.

In 1988, educational psychologist John Sweller introduced Cognitive Load Theory. A 2022 ACM review covers how it's been applied to computing education since.

The short version: your brain juggles three types of load when processing information. Intrinsic load is the inherent difficulty of the problem itself. Extraneous load is the noise — poorly organized information, unnecessary details, bad naming. Germane load is the good stuff — the mental effort you spend building useful mental models.

Here's the kicker: your working memory can only hold a handful of chunks of information at a time — cognitive scientists typically estimate somewhere between 2 and 6. Not 2 to 6 files, or 2 to 6 classes — 2 to 6 things.

Felienne Hermans explores this in The Programmer's Brain (2021), arguing that design patterns act as chunking aids. When you recognize a Strategy pattern, your brain collapses an entire class hierarchy into a single cognitive unit. The word "Strategy" replaces five classes and their relationships. That's not hand-waving about clean code — that's how human memory actually works.

And we can literally see it on brain scans. In 2021, a team led by Norman Peitek and Janet Siegmund published an fMRI study on program comprehension that won the ACM SIGSOFT Distinguished Paper Award at ICSE.

They put developers in brain scanners and watched what happened when they read code. The finding: semantic-level comprehension — understanding what code does — required measurably less neural activation than bottom-up syntactic parsing — tracing how it does it.

An interface lets you comprehend at the semantic level. UserRepository.findById(id) tells you everything you need to know without opening the implementation. Your brain doesn't need to hold the SQL query, the connection pool logic, the error handling, and the result mapping in working memory simultaneously. The interface compresses all of that into one chunk.

That's not elegance. That's neuroscience.

The Greats Already Knew This

The case for abstraction isn't new. The people who built the foundations of computer science were making this argument before most of us were born.

Dijkstra said it with precision:

"The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise."

Abstraction isn't about hiding things from people who can't handle complexity. It's about creating a level of discourse where you can reason clearly.

David Parnas formalized information hiding in his 1972 ACM paper: "Every module is characterized by its knowledge of a design decision which it hides from all others." He proved that decomposing systems by design decisions (rather than processing steps) produced modules that were both more flexible and easier to understand. Comprehensibility wasn't a bonus — it was the design criterion.

Tony Hoare argued that abstraction is the most powerful tool available to the human intellect — a way to manage complexity by focusing on what matters and ignoring what doesn't. Martin Fowler brought it down to earth:

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand."

And then there's John Ousterhout, whose book A Philosophy of Software Design (2018) makes the connection to cognitive load explicit. His central argument: more lines of code can actually be simpler if they reduce cognitive load.

His concept of deep modules — simple interfaces hiding complex implementations — is essentially the argument that interfaces are worth their weight in code. The Unix file system API (open, close, read, write, lseek) is five functions hiding an enormous amount of complexity. That's a deep module. That's the goal.

The Gang of Four put it first in their book for a reason. Page one: "Program to an interface, not an implementation."

None of this is controversial. But it's easy to forget when your AI tool just generated 200 lines of perfectly functional inline code in three seconds.

The Economics Have Flipped

Here's where the CEO's insight becomes an economic argument.

The historical case against interfaces was always about writing cost. Interfaces meant more code to write, more files to create, more boilerplate to maintain. The entire dynamic typing movement — Python, Ruby, JavaScript — was partly a reaction to the ceremony that languages like Java imposed. Convention over configuration. Don't Repeat Yourself. Less is more.

But ask yourself: what exactly is the cost of writing boilerplate now?

GitHub's 2022 controlled study found that developers using Copilot completed tasks 55% faster. The boilerplate that used to justify skipping interfaces — the extra file, the type definitions, the method signatures — takes seconds to generate. The writing cost of an interface has effectively collapsed to zero.

But again, the reading cost hasn't budged.

Robert C. Martin argued in Clean Code (2008) that developers spend far more time reading code than writing it — an observation he framed as a ratio of 10 to 1.

You can quibble with the exact number (it's anecdotal), but the direction is consistent across studies. A large-scale field study tracking 78 professional developers across 3,148 working hours found they spend roughly 58% of their time on program comprehension alone. New developer onboarding averages six weeks — most of which is spent understanding existing systems, not producing new ones.

Addy Osmani named this asymmetry perfectly. In a March 2026 piece, he described comprehension debt:

"When a developer on your team writes code, the human review process has always been a bottleneck — but a productive and educational one. Reading their PR forces comprehension. AI-generated code breaks that feedback loop. The volume is too high."

The output looks clean, passes linting, follows conventions — precisely the signals that historically triggered merge confidence. But comprehension debt is distinct from technical debt because it accumulates invisibly — your velocity metrics, your DORA scores, your PR counts all look fine while your team's actual understanding of the codebase quietly erodes.

So here's the math: AI reduced the cost of writing abstractions to near zero. The cost of not having them — in human reading time, onboarding friction, and comprehension debt — hasn't changed at all. The break-even point for "is this interface worth it?" just shifted massively in favor of "yes."

The Data Backs It Up

This isn't theoretical. We have data on what happens when AI generates code without good abstractions.

GitClear analyzed 211 million changed lines of code between 2020 and 2024. Their findings: code churn — lines reverted or updated within two weeks — doubled compared to the pre-AI baseline. Copy-pasted code blocks rose from 8.3% to 12.3%. And refactoring-associated changes dropped from 25% to under 10%.

AI-generated code, as they put it, "resembles an itinerant contributor, prone to violate the DRY-ness of the repos visited."

The METR study (2025) found something even more striking. Experienced open-source developers predicted AI would make them 24% faster. They perceived being 20% faster while using it. They were actually 19% slower. The perception gap is the story — you feel productive while generating code that creates more work downstream.

And then there's a study from Anthropic (yes, the company that makes Claude — full disclosure). They observed 52 software engineers learning a new library. The AI-assisted group completed tasks at the same speed, but scored 17% lower on comprehension quizzes afterward — 50% versus 67%. The biggest declines were in debugging ability. You can ship code you don't understand. You can't debug code you don't understand.

Kent Beck put it bluntly: "The value of 90% of my skills just dropped to $0. The leverage for the remaining 10% went up 1000x." What that remaining 10% is, he leaves deliberately open — but it's hard to read that and not think about system design.

The Contrarian Case (And Why It Actually Agrees)

I'd be dishonest if I didn't address the people who argue against abstraction. And some of them are very smart.

Casey Muratori's "Clean Code, Horrible Performance" demonstrated that polymorphism and virtual dispatch can make code 10 to 15 times slower than straightforward procedural alternatives.

His benchmark is real. If you're writing a game engine or a high-frequency trading system, abstract interfaces on your hot path will cost you.

Dan Abramov wrote "Goodbye, Clean Code" after watching a premature abstraction make his codebase harder to modify:

"My code traded the ability to change requirements for reduced duplication, and it was not a good trade."

Sandi Metz put it more sharply: "Duplication is far cheaper than the wrong abstraction."

And Rich Hickey, in his talk "Simple Made Easy", draws the critical distinction: simple (not intertwined) is not the same as easy (familiar). Wrong abstractions complect — they braid concerns together rather than separating them.

Here's the thing: none of these are arguments against abstraction. They're arguments against bad abstraction.

Muratori's performance argument applies to hot paths in performance-critical systems — not to your REST API's service layer. Abramov and Metz argue against premature abstraction — pulling patterns out before you understand the domain. And Hickey's entire talk is a case for the right abstractions, the ones that genuinely decompose rather than complect.

The irony is that in an AI-assisted world, these arguments are easier to address. You can generate the explicit, unabstracted version first. Let it stabilize. Watch the patterns emerge. Then extract the abstraction — with AI handling the mechanical refactoring. The cost of the "duplicate first, abstract later" approach just dropped to near zero.

What This Means for You

If you're writing code with AI tools — and at this point, most of us are — the temptation is to let the AI produce whatever it produces and move on. It works. It passes the tests. Ship it.

But "it works" is table stakes. The harder question is: can the next person who opens this code understand it in under five minutes? Can you understand it in six months?

Interfaces aren't about making code prettier or satisfying some abstract (pun intended) design principle. They're compression algorithms for human cognition. They let your brain operate at the semantic level instead of the syntactic level. And now that AI has eliminated the only real cost of creating them — the boilerplate — there's no economic argument left for skipping them.

The rules haven't changed. The excuse has just expired.

References

Academic Papers

• Duran, R., Zavgorodniaia, A., & Sorva, J. (2022). "Cognitive Load Theory in Computing Education Research: A Review." ACM Transactions on Computing Education, 22(4), Article 40.

• Parnas, D.L. (1972). "On the Criteria To Be Used in Decomposing Systems into Modules." Communications of the ACM, 15(12), 1053–1058.

• Peitek, N., Apel, S., Parnin, C., Brechmann, A., & Siegmund, J. (2021). "Program Comprehension and Code Complexity Metrics: An fMRI Study." ICSE 2021. ACM SIGSOFT Distinguished Paper Award.

• Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590.

• Shen, J.H. & Tamkin, A. (2026). "How AI Impacts Skill Formation." arXiv:2601.20245.

• Xia, X., Bao, L., Lo, D., Xing, Z., Hassan, A.E., & Li, S. (2018). "Measuring Program Comprehension: A Large-Scale Field Study with Professionals." IEEE Transactions on Software Engineering, 44(10), 951–976.

• METR. (2025). "Measuring the Impact of Early 2025 AI on Experienced Open Source Developer Productivity." metr.org.

Talks and Blog Posts

• Hickey, R. (2011). "Simple Made Easy." Strange Loop Conference.

• Beck, K. (2023). "90% of My Skills Are Now Worth $0." Tidy First? Substack.

• Osmani, A. (2026). "Comprehension Debt: The Hidden Cost of AI-Generated Code." addyosmani.com.

• Muratori, C. (2023). "Clean Code, Horrible Performance." Computer Enhance.

• Abramov, D. (2020). "Goodbye, Clean Code." overreacted.io.

• Metz, S. (2016). "The Wrong Abstraction." sandimetz.com.

• GitClear. (2025). "AI Assistant Code Quality in 2025." gitclear.com.

The webhook handler is where most payment integrations silently break. Your server crashes halfway through granting access. Your email service is down when you try to send the confirmation. Your database times out during a write.

Stripe retries the entire webhook, but your handler already sent the confirmation email before it crashed. Now the customer gets two emails and no access.

This article shows you how to fix this. You'll learn how to build webhook handlers that survive failures by splitting your post-payment logic into durable, independently retried steps. The pattern works for any multi-step webhook processing, not just Stripe.

Here's what you'll learn:

• Why Stripe webhooks fail silently in production

• How a naïve inline handler breaks under real-world conditions

• The pattern: webhook receives, validates, and enqueues (nothing more)

• How to build a durable purchase flow with individually checkpointed steps

• How to handle refunds and abandoned checkouts with the same pattern

• How to test webhook handlers locally

Prerequisites

To follow along, you should be familiar with:

• Node.js and TypeScript

• Basic Stripe integration (checkout sessions, webhooks)

• SQL databases (the examples use PostgreSQL with Drizzle ORM)

• npm or any Node.js package manager

You don't need prior experience with Inngest or durable execution. This article explains both from scratch.

What You Need to Install

If you want to run the code examples, install these packages:

npm install inngest stripe drizzle-orm @react-email/components resend

You'll also need the Stripe CLI for local webhook testing. Install it via Homebrew on macOS (brew install stripe/stripe-cli/stripe) or follow the instructions in Stripe's documentation for other platforms.

Table of Contents

• Why Stripe Webhooks Fail Silently

• The Naïve Approach (and Why It Breaks)

• The Pattern: Webhook to Event to Durable Function

• How to Set Up the Webhook Endpoint

• How to Build a Durable Purchase Flow

• How to Handle Refunds with the Same Pattern

• How to Recover Abandoned Checkouts

• How to Test Webhook Handlers Locally

• Conclusion

Why Stripe Webhooks Fail Silently

The happy path is easy. A customer pays, Stripe sends a checkout.session.completed event to your server, and your handler processes it. In development, this works every time.

Production is different: Your webhook handler typically needs to do several things after a successful payment. It looks up the user in the database, records the purchase, sends a confirmation email, notifies the admin, grants access to the product (maybe via a GitHub invitation or an API key), and schedules follow-up emails. That's five or six operations involving three or four external services.

Here are the failure modes that will eventually hit your webhook handler:

1. Your server crashes mid-processing

The database write succeeded, but the email never sent. Stripe retries the webhook, and your handler runs again.

Now you have a duplicate database entry or a unique constraint error that kills the retry.

2. An external service is temporarily down

Your email provider returns a 500. Your GitHub API call gets rate-limited. Your analytics service times out.

The webhook handler throws, and Stripe retries the entire thing. But the steps that already succeeded (the database write, the first email) run again.

3. The handler times out

Stripe expects a 2xx response within about 20 seconds. If your handler does too much work, Stripe marks it as failed and retries. Your handler may have partially completed before the timeout.

4. Partial completion with no rollback

This is the worst failure mode. Steps 1 through 3 succeed. Step 4 fails. Stripe retries, and steps 1 through 3 run again.

The customer gets two confirmation emails. The database gets a duplicate record. But step 4 still fails because the underlying issue (a rate limit, a service outage) hasn't been resolved.

5. Race conditions on retry

Stripe can deliver the same event more than once even without a failure on your end. Network glitches, load balancer timeouts, and Stripe's own retry logic mean your handler must be prepared for duplicate deliveries. If your handler isn't idempotent at every step, duplicates compound the partial-completion problem.

Stripe's retry behavior is well-designed. It uses exponential backoff and retries up to dozens of times over several days. But Stripe retries the entire webhook delivery.

It has no way to know that your handler completed steps 1 through 3 and only needs to retry step 4. That distinction is your responsibility.

The core problem is that your webhook handler does too many things in a single request. Every external call is a potential failure point, and you have no checkpointing between them. When one fails, you lose track of which ones already succeeded.

The Naïve Approach (and Why It Breaks)

Here's what a typical webhook handler looks like. I've seen hundreds of variations of this pattern across codebases, tutorials, and Stack Overflow answers:

app.post("/api/payments/webhook", async (req, res) => {
  const event = stripe.webhooks.constructEvent(
    req.body,
    req.headers["stripe-signature"],
    process.env.STRIPE_WEBHOOK_SECRET
  );

  if (event.type === "checkout.session.completed") {
    const session = event.data.object;

    // Step 1: Look up the user
    const user = await db.users.findOne({ id: session.metadata.userId });

    // Step 2: Record the purchase
    await db.purchases.insert({
      userId: user.id,
      stripeSessionId: session.id,
      amount: session.amount_total,
      status: "completed",
    });

    // Step 3: Send confirmation email
    await sendEmail({
      to: user.email,
      subject: "Purchase confirmed!",
      template: "purchase-confirmation",
    });

    // Step 4: Grant product access (GitHub repo invitation)
    await addCollaborator(user.githubUsername);

    // Step 5: Send access email
    await sendEmail({
      to: user.email,
      subject: "Your repository access is ready!",
      template: "repo-access",
    });

    // Step 6: Track analytics
    await analytics.track(user.id, "purchase_completed", {
      amount: session.amount_total,
    });
  }

  res.json({ received: true });
});

This looks clean. It reads top-to-bottom. Every tutorial teaches it this way.

Now walk through what happens when step 4 fails. Maybe GitHub's API is rate-limited and the addCollaborator call throws an error. Your handler returns a 500 to Stripe.

Here is the state after the failure:

• The user exists in the database (step 1 was just a lookup, no problem).

• A purchase record was created (step 2 succeeded).

• The confirmation email was sent (step 3 succeeded).

• GitHub access was not granted (step 4 failed).

• The access email was not sent (step 5 never ran).

• Analytics were not tracked (step 6 never ran).

Stripe retries the webhook. Your handler runs again from the top:

• Step 1: Looks up the user again. Fine.

• Step 2: Tries to insert another purchase record. If you have a unique constraint on stripeSessionId, this throws. If you don't, you now have a duplicate.

• Step 3: Sends the confirmation email again. The customer gets a second "Purchase confirmed!" email.

• Step 4: Tries GitHub access again. Maybe it works this time, maybe not.

• Steps 5-6: May or may not run depending on step 4.

You can patch this with idempotency checks: "if purchase already exists, skip step 2." But now your handler is full of conditional logic for every step. And you still have the duplicate email problem, because there's no way to check "did I already send this email?" without building your own tracking system.

This approach doesn't scale. Every new step adds another failure mode, another idempotency check, and another edge case.

The Pattern: Webhook to Event to Durable Function

The fix is a separation of concerns. Your webhook handler should do exactly one thing: validate the incoming event and enqueue it for processing. Nothing else.

All the actual work (database writes, emails, API calls, analytics) moves into a durable background function where each step is individually checkpointed, retried, and tracked.

Here's the flow:

Stripe webhook
    |
    v
Webhook endpoint (validate signature, extract event, enqueue)
    |
    v
Background job system (receives event)
    |
    v
Durable function
    |-- Step 1: Look up user and purchase (checkpointed)
    |-- Step 2: Track analytics (checkpointed)
    |-- Step 3: Send confirmation email (checkpointed)
    |-- Step 4: Send admin notification (checkpointed)
    |-- Step 5: Grant GitHub access (checkpointed)
    |-- Step 6: Track GitHub access (checkpointed)
    |-- Step 7: Update purchase record (checkpointed)
    |-- Step 8: Send repo access email (checkpointed)
    |-- Step 9: Schedule follow-up sequence (checkpointed)

Each step wrapped in step.run() is a durable checkpoint. If step 5 fails:

• Steps 1 through 4 do not re-run. Their results are cached.

• Step 5 retries independently, with its own retry counter.

• Once step 5 succeeds, steps 6 through 9 continue.

This is what "durable execution" means. The function's progress survives failures. You get step-level retries instead of function-level retries. No duplicate emails. No duplicate database writes. No partial completion.

I use Inngest for this. It's an event-driven durable execution platform that provides step-level checkpointing out of the box. You define functions with step.run() blocks, and Inngest handles retry logic, state persistence, and observability. No Redis, no worker processes, no custom retry code.

Other tools can achieve similar results (Temporal, for example), but Inngest's developer experience with TypeScript is what sold me. You write normal async functions. The step.run() wrapper is the only addition.

How to Set Up the Webhook Endpoint

Your webhook endpoint should be minimal. Validate the signature, extract the event data, send it to your background job system, and return a 200 immediately.

Here's the real webhook endpoint from my production codebase:

import { constructWebhookEvent } from "@/lib/payments";
import { inngest } from "@/lib/jobs";

app.post("/api/payments/webhook", async ({ request, set }) => {
  const body = await request.text();
  const sig = request.headers.get("stripe-signature");

  if (!sig) {
    set.status = 400;
    return { error: "Missing signature" };
  }

  try {
    const event = await constructWebhookEvent(body, sig);
    console.log(`[Webhook] Received ${event.type}`);

    if (event.type === "charge.refunded") {
      const charge = event.data.object;
      await inngest.send({
        name: "stripe/charge.refunded",
        data: {
          chargeId: charge.id,
          paymentIntentId: charge.payment_intent,
          amountRefunded: charge.amount_refunded,
          originalAmount: charge.amount,
          currency: charge.currency,
        },
      });
    }

    if (event.type === "checkout.session.expired") {
      const session = event.data.object;
      await inngest.send({
        name: "stripe/checkout.session.expired",
        data: {
          sessionId: session.id,
          customerEmail: session.customer_email,
        },
      });
    }

    return { received: true };
  } catch (error) {
    console.error("[Webhook] Stripe verification failed:", error);
    set.status = 400;
    return { error: "Webhook verification failed" };
  }
});

Notice what this handler does not do: it does not look up users, write to the database, send emails, or call external APIs. It validates the Stripe signature, extracts the relevant fields, and sends a typed event to Inngest. The entire handler completes in milliseconds.

The constructWebhookEvent function wraps Stripe's signature verification:

import Stripe from "stripe";

export async function constructWebhookEvent(
  payload: string | Buffer,
  signature: string
) {
  const webhookSecret = process.env.STRIPE_WEBHOOK_SECRET;
  if (!webhookSecret) {
    throw new Error("STRIPE_WEBHOOK_SECRET is not set");
  }
  const client = new Stripe(process.env.STRIPE_SECRET_KEY);
  return client.webhooks.constructEventAsync(payload, signature, webhookSecret);
}

One critical detail: you must pass the raw request body (as a string or buffer) to Stripe's signature verification. If your framework parses the body as JSON before you can access the raw string, the signature check will fail. This is the number one cause of "webhook signature verification failed" errors.

The Inngest client setup is minimal:

import { Inngest } from "inngest";

export const inngest = new Inngest({
  id: "my-app",
});

For the purchase flow specifically, a different endpoint sends the event (the "claim" route that the frontend calls after the customer returns from Stripe checkout). But the principle is identical: validate, enqueue, return.

// After verifying payment status with Stripe
await inngest.send({
  name: "purchase/completed",
  data: {
    userId: session.user.id,
    tier,
    sessionId,
  },
});

How to Build a Durable Purchase Flow

This is the core of the article. The handlePurchaseCompleted function processes a purchase after payment using 9 individually checkpointed steps. Every step is real production code.

The example below grants access to a private GitHub repository because that's what this particular product sells.

Your product's "grant access" step will be different: upgrading a user to a Pro membership, provisioning API credits, unlocking a course, or activating a subscription. The durable step pattern is the same regardless of what you're delivering.

If step 5 fails (for example, the email provider is down), Inngest retries only step 5. Steps 1 through 4 are already checkpointed and don't re-execute. Steps 6 through 9 wait until step 5 succeeds.

import { eq } from "drizzle-orm";
import { createElement } from "react";

import { inngest } from "@/lib/jobs/client";
import { trackServerEvent } from "@/lib/analytics/server";
import { brand } from "@/lib/brand";
import { db, purchases, users } from "@/lib/db";
import {
  sendEmail,
  PurchaseConfirmationEmail,
  AdminPurchaseNotificationEmail,
  RepoAccessGrantedEmail,
} from "@/lib/email";
import { addCollaborator } from "@/lib/github";

export const handlePurchaseCompleted = inngest.createFunction(
  { id: "purchase-completed", triggers: [{ event: "purchase/completed" }] },
  async ({ event, step }) => {
    const { userId, tier, sessionId } = event.data;

    // Step 1: Look up user and purchase details
    const { user, purchase } = await step.run(
      "lookup-user-and-purchase",
      async () => {
        const userResult = await db
          .select({
            id: users.id,
            email: users.email,
            name: users.name,
            githubUsername: users.githubUsername,
          })
          .from(users)
          .where(eq(users.id, userId))
          .limit(1);

        const foundUser = userResult[0];
        if (!foundUser) {
          throw new Error(`User not found: ${userId}`);
        }

        const purchaseResult = await db
          .select({
            amount: purchases.amount,
            currency: purchases.currency,
            stripePaymentIntentId: purchases.stripePaymentIntentId,
          })
          .from(purchases)
          .where(eq(purchases.stripeCheckoutSessionId, sessionId))
          .limit(1);

        const foundPurchase = purchaseResult[0];

        return {
          user: foundUser,
          purchase: foundPurchase ?? {
            amount: 0,
            currency: "usd",
            stripePaymentIntentId: null,
          },
        };
      }
    );

    // Step 2: Track purchase completion in analytics
    await step.run("track-purchase-to-posthog", async () => {
      await trackServerEvent(userId, "purchase_completed_server", {
        tier,
        amount_cents: purchase.amount,
        currency: purchase.currency,
        stripe_session_id: sessionId,
      });
    });

    // Step 3: Send purchase confirmation to customer
    await step.run("send-purchase-confirmation", async () => {
      await sendEmail({
        to: user.email,
        subject: `Your purchase is confirmed!`,
        template: createElement(PurchaseConfirmationEmail, {
          amount: purchase.amount,
          currency: purchase.currency,
          customerEmail: user.email,
        }),
      });
    });

    // Step 4: Send admin notification
    await step.run("send-admin-notification", async () => {
      const adminEmail = process.env.ADMIN_EMAIL;
      if (!adminEmail) return;

      await sendEmail({
        to: adminEmail,
        subject: `New sale: ${user.email}`,
        template: createElement(AdminPurchaseNotificationEmail, {
          amount: purchase.amount,
          currency: purchase.currency,
          customerEmail: user.email,
          customerName: user.name,
          stripeSessionId: purchase.stripePaymentIntentId ?? sessionId,
        }),
      });
    });

    // Early return if user has no GitHub username
    if (!user.githubUsername) {
      return { success: true, userId, tier, githubAccessGranted: false };
    }

    // Step 5: Grant GitHub repository access
    const collaboratorResult = await step.run(
      "add-github-collaborator",
      async () => {
        return addCollaborator(user.githubUsername!);
      }
    );

    // Step 6: Track GitHub access granted
    await step.run("track-github-access", async () => {
      await trackServerEvent(userId, "github_access_granted", {
        tier,
        github_username: user.githubUsername,
        invitation_status: collaboratorResult.status,
      });
    });

    // Step 7: Update purchase record
    await step.run("update-purchase-record", async () => {
      await db
        .update(purchases)
        .set({
          githubAccessGranted: true,
          githubInvitationId: collaboratorResult.status,
          updatedAt: new Date(),
        })
        .where(eq(purchases.stripeCheckoutSessionId, sessionId));
    });

    // Step 8: Send repo access email
    await step.run("send-repo-access-email", async () => {
      await sendEmail({
        to: user.email,
        subject: `Your repository access is ready!`,
        template: createElement(RepoAccessGrantedEmail, {
          repoUrl: "https://github.com/your-org/your-repo",
        }),
      });
    });

    // Step 9: Schedule follow-up email sequence
    await step.run("schedule-follow-up", async () => {
      const purchaseRecord = await db
        .select({ id: purchases.id })
        .from(purchases)
        .where(eq(purchases.stripeCheckoutSessionId, sessionId))
        .limit(1);

      if (purchaseRecord[0]) {
        await inngest.send({
          name: "purchase/follow-up.scheduled",
          data: {
            userId,
            purchaseId: purchaseRecord[0].id,
            tier,
          },
        });
      }
    });

    return { success: true, userId, tier, githubAccessGranted: true };
  }
);

That's a lot of code. Let me walk through each step and explain why it's a separate checkpoint.

Step 1: Look Up User and Purchase

const { user, purchase } = await step.run(
  "lookup-user-and-purchase",
  async () => {
    // ... database queries ...
    return { user: foundUser, purchase: foundPurchase };
  }
);

This step queries the database for the user and purchase records. If the database is temporarily unreachable, this step retries on its own.

The return value (user and purchase) is cached by Inngest. Every subsequent step can use user.email, user.githubUsername, and purchase.amount without re-querying the database.

If this step fails permanently (the user doesn't exist), it throws an error that halts the entire function. This is intentional. There's no point continuing if you can't find the user.

Step 2: Track Analytics

await step.run("track-purchase-to-posthog", async () => {
  await trackServerEvent(userId, "purchase_completed_server", {
    tier,
    amount_cents: purchase.amount,
  });
});

Analytics tracking is a separate step because analytics services have their own failure modes (rate limits, outages, network timeouts). If PostHog is down, you don't want it to block the confirmation email.

In the production code, this step wraps the call in a try-catch so that a tracking failure doesn't halt the entire function. The analytics event is "nice to have," not critical.

Step 3: Send Purchase Confirmation Email

await step.run("send-purchase-confirmation", async () => {
  await sendEmail({
    to: user.email,
    subject: `Your purchase is confirmed!`,
    template: createElement(PurchaseConfirmationEmail, {
      amount: purchase.amount,
      currency: purchase.currency,
      customerEmail: user.email,
    }),
  });
});

This is the customer-facing confirmation. It's a separate step from the admin notification (step 4) because they're independent operations. If the admin email fails, the customer should still get their confirmation.

The sendEmail function uses Resend under the hood. If Resend returns a 500, this step retries. Because step 2 (analytics) already completed and is checkpointed, it won't re-run.

Step 4: Send Admin Notification

await step.run("send-admin-notification", async () => {
  const adminEmail = process.env.ADMIN_EMAIL;
  if (!adminEmail) return;

  await sendEmail({
    to: adminEmail,
    subject: `New sale: ${user.email}`,
    template: createElement(AdminPurchaseNotificationEmail, { /* ... */ }),
  });
});

Admin notifications are completely independent from customer-facing operations. Separating them means a failure in one doesn't affect the other.

Step 5: Grant GitHub Access

const collaboratorResult = await step.run(
  "add-github-collaborator",
  async () => {
    return addCollaborator(user.githubUsername!);
  }
);

This is the step most likely to fail. GitHub's API has rate limits: it can time out, and the user's GitHub username might be invalid.

By making this its own step, a GitHub API failure doesn't trigger re-sends of the confirmation email (step 3) or the admin notification (step 4). Those steps are already checkpointed.

Notice the early return before this step: if the user has no GitHub username, the function returns early after step 4. The remaining steps only run when there's a GitHub account to grant access to.

Step 6: Track GitHub Access

await step.run("track-github-access", async () => {
  await trackServerEvent(userId, "github_access_granted", {
    tier,
    github_username: user.githubUsername,
    invitation_status: collaboratorResult.status,
  });
});

This uses the collaboratorResult from step 5. Because step.run() caches return values, collaboratorResult.status is available here even if the function was interrupted and resumed between steps 5 and 6.

Step 7: Update Purchase Record

await step.run("update-purchase-record", async () => {
  await db
    .update(purchases)
    .set({
      githubAccessGranted: true,
      githubInvitationId: collaboratorResult.status,
      updatedAt: new Date(),
    })
    .where(eq(purchases.stripeCheckoutSessionId, sessionId));
});

The database update happens after GitHub access is confirmed. You only mark githubAccessGranted: true after the collaborator invitation actually succeeded.

If you updated the record before granting access and the GitHub step failed, your database would say access was granted when it was not.

Step 8: Send Repo Access Email

await step.run("send-repo-access-email", async () => {
  await sendEmail({
    to: user.email,
    subject: `Your repository access is ready!`,
    template: createElement(RepoAccessGrantedEmail, {
      repoUrl: "https://github.com/your-org/your-repo",
    }),
  });
});

This email only sends after the GitHub invitation is confirmed (step 5) and the database is updated (step 7). The ordering matters. You don't want to tell the customer "your access is ready" if the invitation hasn't been sent.

Step 9: Schedule Follow-Up Sequence

await step.run("schedule-follow-up", async () => {
  const purchaseRecord = await db
    .select({ id: purchases.id })
    .from(purchases)
    .where(eq(purchases.stripeCheckoutSessionId, sessionId))
    .limit(1);

  if (purchaseRecord[0]) {
    await inngest.send({
      name: "purchase/follow-up.scheduled",
      data: {
        userId,
        purchaseId: purchaseRecord[0].id,
        tier,
      },
    });
  }
});

The final step triggers a separate Inngest function that handles the follow-up email sequence (day 7 onboarding tips, day 14 feedback request, day 30 testimonial request). This is an event-driven chain: one function completes and triggers another.

The follow-up function uses step.sleep() to wait between emails:

export const handlePurchaseFollowUp = inngest.createFunction(
  {
    id: "purchase-follow-up",
    triggers: [{ event: "purchase/follow-up.scheduled" }],
    cancelOn: [
      {
        event: "purchase/follow-up.cancelled",
        match: "data.purchaseId",
      },
    ],
  },
  async ({ event, step }) => {
    const { userId, purchaseId } = event.data;

    await step.sleep("wait-7-days", "7d");

    await step.run("send-day-7-email", async () => {
      // Check eligibility (user exists, not unsubscribed, not refunded)
      // Send onboarding tips email
    });

    await step.sleep("wait-14-days", "7d");

    await step.run("send-day-14-email", async () => {
      // Send feedback request email
    });

    await step.sleep("wait-30-days", "16d");

    await step.run("send-day-30-email", async () => {
      // Send testimonial request email
    });
  }
);

Notice the cancelOn option. If the purchase is refunded, you can send a purchase/follow-up.cancelled event, and the entire follow-up sequence stops. No stale emails sent to customers who asked for a refund.

Why Each Step Must Be Separate

The rule is simple: any operation that calls an external service or could fail independently should be its own step.

A database query is a step because the database can be temporarily unreachable. An email send is a step because the email provider can return a 500. A GitHub API call is a step because it can be rate-limited.

If two operations always succeed or fail together (they share a single external call), they can be in the same step. But when in doubt, make it a separate step. The overhead is negligible, and the reliability gain is significant.

How to Handle Refunds with the Same Pattern

The refund flow follows the exact same durable step pattern. This function lives in the same file as handlePurchaseCompleted, so it shares the same imports (plus removeCollaborator from @/lib/github and the refund-specific email templates). Here's the handleRefund function:

export const handleRefund = inngest.createFunction(
  { id: "refund-processed", triggers: [{ event: "stripe/charge.refunded" }] },
  async ({ event, step }) => {
    const {
      chargeId,
      paymentIntentId,
      amountRefunded,
      originalAmount,
      currency,
    } = event.data;

    const isFullRefund = amountRefunded >= originalAmount;

    // Step 1: Look up the purchase and user
    const { user, purchase } = await step.run(
      "lookup-purchase-by-payment-intent",
      async () => {
        const purchaseResult = await db
          .select({
            id: purchases.id,
            userId: purchases.userId,
            stripePaymentIntentId: purchases.stripePaymentIntentId,
            githubAccessGranted: purchases.githubAccessGranted,
          })
          .from(purchases)
          .where(eq(purchases.stripePaymentIntentId, paymentIntentId))
          .limit(1);

        const foundPurchase = purchaseResult[0];
        if (!foundPurchase) {
          return { user: null, purchase: null };
        }

        const userResult = await db
          .select({
            id: users.id,
            email: users.email,
            name: users.name,
            githubUsername: users.githubUsername,
          })
          .from(users)
          .where(eq(users.id, foundPurchase.userId))
          .limit(1);

        return { user: userResult[0] ?? null, purchase: foundPurchase };
      }
    );

    if (!purchase || !user) {
      return { success: false, reason: "no_matching_purchase" };
    }

    let accessRevoked = false;

    // Step 2: Revoke GitHub access (only for full refunds)
    if (isFullRefund && user.githubUsername && purchase.githubAccessGranted) {
      const revokeResult = await step.run(
        "revoke-github-access",
        async () => {
          return removeCollaborator(user.githubUsername!);
        }
      );
      accessRevoked = revokeResult.success;
    }

    // Step 3: Update purchase status
    await step.run("update-purchase-status", async () => {
      if (isFullRefund) {
        await db
          .update(purchases)
          .set({
            status: "refunded",
            githubAccessGranted: false,
            updatedAt: new Date(),
          })
          .where(eq(purchases.id, purchase.id));
      } else {
        await db
          .update(purchases)
          .set({
            status: "partially_refunded",
            updatedAt: new Date(),
          })
          .where(eq(purchases.id, purchase.id));
      }
    });

    // Step 4: Track refund in analytics
    await step.run("track-refund-event", async () => {
      await trackServerEvent(user.id, "refund_processed", {
        charge_id: chargeId,
        amount_cents: amountRefunded,
        original_amount_cents: originalAmount,
        currency,
        is_full_refund: isFullRefund,
        github_access_revoked: accessRevoked,
      });
    });

    // Step 5: Notify customer
    await step.run("send-customer-notification", async () => {
      if (isFullRefund) {
        await sendEmail({
          to: user.email,
          subject: "Your refund has been processed",
          template: createElement(AccessRevokedEmail, {
            customerEmail: user.email,
            refundAmount: amountRefunded,
            currency,
          }),
        });
      } else {
        await sendEmail({
          to: user.email,
          subject: "Your partial refund has been processed",
          template: createElement(PartialRefundEmail, {
            customerEmail: user.email,
            refundAmount: amountRefunded,
            originalAmount,
            currency,
          }),
        });
      }
    });

    // Step 6: Notify admin
    await step.run("send-admin-notification", async () => {
      const adminEmail = process.env.ADMIN_EMAIL;
      if (!adminEmail) return;

      await sendEmail({
        to: adminEmail,
        subject: `\({isFullRefund ? "Full" : "Partial"} refund: \){user.email}`,
        template: createElement(AdminRefundNotificationEmail, {
          customerEmail: user.email,
          customerName: user.name,
          githubUsername: user.githubUsername,
          refundAmount: amountRefunded,
          originalAmount,
          currency,
          stripeChargeId: chargeId,
          accessRevoked,
          isPartialRefund: !isFullRefund,
        }),
      });
    });

    return { success: true, accessRevoked, isFullRefund, userId: user.id };
  }
);

Three things are worth calling out in the refund flow.

1. Partial versus full refunds: The function distinguishes between the two using a simple comparison: amountRefunded >= originalAmount. For a partial refund, the customer keeps access but the purchase status changes to partially_refunded. For a full refund, GitHub access is revoked and the status becomes refunded.

   This matters for your database integrity. Downstream systems (your dashboard, your analytics, your support tools) need accurate status values.

2. Conditional step execution: The "revoke GitHub access" step only runs if three conditions are true: it's a full refund, the user has a GitHub username, and access was previously granted. Inngest handles this cleanly by skipping steps that don't need to run.

   This is more readable than deeply nested if-else blocks in a monolithic handler.

3. Separate notifications for customers and admins: The customer gets a different email depending on whether the refund is full or partial. The admin always gets a detailed notification including the charge ID, the customer's GitHub username, and whether access was revoked.

These are separate steps because a failure in the admin notification shouldn't block the customer notification. The customer's email is the higher priority.

How to Recover Abandoned Checkouts

Abandoned cart recovery is where the step.sleep() method shines. When a Stripe checkout session expires, you want to send a recovery email. But not immediately.

You want to wait an hour or so, giving the customer time to return on their own.

export const handleCheckoutExpired = inngest.createFunction(
  {
    id: "checkout-expired",
    triggers: [{ event: "stripe/checkout.session.expired" }],
  },
  async ({ event, step }) => {
    const { customerEmail, sessionId } = event.data;

    if (!customerEmail) {
      return { success: false, reason: "no_email" };
    }

    // Wait 1 hour before sending recovery email
    await step.sleep("wait-before-recovery-email", "1h");

    // Send abandoned cart email
    await step.run("send-abandoned-cart-email", async () => {
      const checkoutUrl = `https://yoursite.com/pricing`;

      await sendEmail({
        to: customerEmail,
        subject: "Your checkout is waiting",
        template: createElement(AbandonedCartEmail, {
          customerEmail,
          checkoutUrl,
        }),
      });
    });

    // Track the event
    await step.run("track-abandoned-cart", async () => {
      await trackServerEvent("anonymous", "abandoned_cart_email_sent", {
        customer_email: customerEmail,
        session_id: sessionId,
      });
    });

    return { success: true, customerEmail };
  }
);

The step.sleep("wait-before-recovery-email", "1h") line is the key. This pauses the function for one hour without consuming any compute resources.

Inngest handles the scheduling internally. After one hour, the function resumes and sends the email.

Without durable execution, you would need a cron job that queries a database for expired sessions, or a delayed job queue with Redis, or a setTimeout that gets lost when your server restarts. The step.sleep() approach is simpler, more readable, and more reliable.

There's also a guard at the top of the function. If Stripe doesn't have a customer email for the session (the customer closed the checkout before entering their email), the function returns early. There's no point scheduling a recovery email with no address to send it to.

This pattern scales to more complex recovery flows. You could add a second step.sleep() and send a follow-up recovery email three days later if the customer still hasn't purchased. You could check if the customer has since completed a purchase (by querying the database in a step.run()) and skip the email if they have.

Each additional step is one more step.run() or step.sleep() call. The function reads like a script describing your business logic, not a tangle of cron jobs and database flags.

How to Test Webhook Handlers Locally

Local testing is one of the biggest pain points with Stripe webhooks. You need Stripe to send events to your local machine, and you need your background job system running to process them. Here's the setup.

How to Forward Stripe Events Locally

Install the Stripe CLI and forward webhook events to your local server:

stripe listen --forward-to localhost:3000/api/payments/webhook

The CLI prints a webhook signing secret (starting with whsec_). Set this as your STRIPE_WEBHOOK_SECRET environment variable for local development.

You can trigger test events directly:

stripe trigger checkout.session.completed
stripe trigger charge.refunded
stripe trigger checkout.session.expired

How to Run the Inngest Dev Server

Inngest provides a local dev server that shows you every function execution, every step, and every retry in real time:

npx inngest-cli@latest dev -u http://localhost:3000/api/inngest

The -u flag tells the Inngest dev server where your application is running so it can discover your functions. Open http://localhost:8288 in your browser to see the Inngest dashboard.

How to Watch Step Execution

The Inngest dev dashboard is where the durable execution pattern really clicks. When you trigger a Stripe event, you can see:

1. The event arriving in the "Events" tab.

2. The function triggering in the "Runs" tab.

3. Each step executing one by one, with its input, output, and duration.

4. If a step fails, you see the error and the retry attempt.

This visibility is something you don't get with inline webhook handlers. When a customer reports "I paid but didn't get access," you can look up the function run in the Inngest dashboard and see exactly which step failed and why. That kind of observability is invaluable in production.

How to Simulate Failures

To test the retry behavior, you can intentionally make a step fail. For example, temporarily throw an error in the "add-github-collaborator" step:

const collaboratorResult = await step.run(
  "add-github-collaborator",
  async () => {
    throw new Error("Simulated GitHub API failure");
  }
);

In the Inngest dashboard, you'll see:

• Steps 1 through 4 succeed and their results are cached.

• Step 5 fails and is retried according to the retry policy.

• Steps 6 through 9 remain pending until step 5 succeeds.

Remove the thrown error, and on the next retry, step 5 succeeds. Steps 6 through 9 then execute in sequence, while steps 1 through 4 aren't re-executed. This is the checkpoint behavior in action.

Conclusion

The pattern for reliable Stripe webhooks comes down to one principle: separate receiving from processing.

Your webhook endpoint validates the Stripe signature and sends a typed event to a background job system. That's all it does. The processing happens in a durable function where each step is individually checkpointed and retried.

Here's what this gives you:

• No duplicate emails: A step that already succeeded doesn't re-run.

• No partial state: If step 5 fails, steps 1 through 4 are preserved and step 5 retries independently.

• Full observability: You can see exactly which step failed and why, for every function run.

• Built-in delayed execution: step.sleep() handles recovery emails and follow-up sequences without cron jobs.

• Composable workflows: One function can trigger another via events, creating chains like purchase completion leading to a 30-day follow-up sequence.

This pattern isn't limited to Stripe. Any multi-step webhook processing benefits from durable execution: GitHub webhooks that trigger CI pipelines, Resend webhooks that track email delivery, or calendar webhooks that sync across services.

The principle is the same: Validate. Enqueue. Process durably.

I've used this pattern in production for Eden Stack, where the purchase flow handles everything from payment confirmation to GitHub repository access grants to multi-week email sequences. The 9-step purchase function has processed every payment without a single missed step or duplicate email.

If you're building a SaaS with Stripe, start with the webhook endpoint pattern from this article. Keep the endpoint thin and move the processing into durable steps. You'll save yourself from the 3 AM debugging session when a customer says "I paid but nothing happened."

If you want the complete Stripe webhook and Inngest integration pre-built with purchase flows, refund handling, and follow-up email sequences ready to go, Eden Stack includes everything from this article alongside 30+ additional production-tested patterns.

Magnus Rodseth builds AI-native applications and is the creator of Eden Stack, a production-ready starter kit with 30+ Claude skills encoding production patterns for AI-native SaaS development.

You scroll a little more, and someone is claiming that a new AI tool released last week has already made entry-level developers obsolete. You go to YouTube, and a thumbnail screams that all technologies are dead, all developer jobs are dead, and at the same time, a solo founder claims that they've built a million-dollar full-stack app in five minutes using AI agents.

At some point, you start feeling overwhelmed. You start to question and doubt the nights you've spent learning something, building something. You wonder whether the effort you're putting into mastering a programming language or framework still makes sense. You start asking yourself an extremely uncomfortable question: "Is my career still safe?"

This concern is valid. Instead of dismissing the concern with a lot of motivational talk or toxic positivity, let's do a reality check. The industry is fundamentally changing. Hiring patterns are shifting. Expectations for both junior and senior developers are rising exponentially. And yes, AI is the main catalyst accelerating all these changes.

But there is a massive misunderstanding around what's going on. The narrative that "AI is replacing developers" lacks a lot of details. It has created unnecessary fear because it fails to specify what's actually happening.

Not many devs are coming up to explain these details because a good portion of us are still observing, and some are steering the fear to their individual benefits.

Well, here's my take: AI isn't replacing all software engineers. It's replacing a specific kind of work. The low-level, average, routine execution work is getting replaced with AI much faster than anyone could imagine. As a result, it's forcing us to think of what it means to be a software engineer in today's market.

This article is about that thought process. It's a deep dive into the changing landscape of software development, the shift from effort-based to impact-based engineering, and a practical, actionable roadmap to enable you to remain relevant in the era of AI-assisted coding.

Table of Contents

1. The End of the Tutorial-Driven Era

2. Let's Decode the "AI is Taking Jobs" Myth

3. Applying a Clean Architecture

4. A Practical, AI-Era Engineering Roadmap

   • Step 1: Strengthen Your Fundamentals

   • Step 2: Build Real (Uncomfortable) Systems

   • Step 3: Master the Art of Debugging

   • Step 4: Use AI as a Tool, Not as a Crutch

   • Step 5: Establishing a Strong Proof of Work

5. The Must-Needed Mindset Shift

6. If You've Read This Far...

The End of the Tutorial-Driven Era

Let's step back for a moment and look at how most of us learned to develop software over the last decade or so.

Between 2010 and 2023, the industry was filled with tutorial-driven developers. We learned to build software by following step-by-step instructions.

Applications like TODO Apps, Weather dashboards, or clones of YouTube or Spotify were in high demand among developers. These projects gave us confidence. They helped us memorise syntax, learn how to use libraries, and figure out how to write a basic frontend and backend.

For a long time, this was enough. The goal was simple: "Can I build this full-stack application that works?"

If you could write code, connect to a few APIs, and build a working interface, companies were willing to hire you. They viewed junior developers as an investment. The expectation was that you should be trainable: you would come in, write standard boilerplate code, and learn the complexities of the system architecture on the job. The industry had the budget and patience for that learning curve.

But while memorizing the syntax and completing Udemy courses, the tooling was quietly evolving. Today, AI has taken that to a different extreme.

A significant portion of what we used to learn manually can now be generated, assisted, and suggested by AI in seconds.

• Need a basic Express server setup with rate limiting and CORS integrated? Can be generated.

• Need a responsive navigation bar written in React? Can be assisted.

• Need a standard SQL query to fetch company data? Can be suggested.

If a machine can do something exponentially faster, cheaper, and reasonably well, that specific task stops being the differentiator in the job market. So, when people say AI is replacing junior developers, what they mean is that AI has automated the execution of these surface-level tasks.

But does it mean developers are no longer needed? No, it means the value of our work has moved up the stack. Building a TODO app, a Weather dashboard, or website clones is no longer a portfolio item. They're just your warm-up exercises.

Let's Decode the "AI is Taking Jobs" Myth

Traditionally, software engineers were given requirements: they wrote code, and they ensured it worked. The value of a software engineer was tied to their work execution. Even in interviews, the emphasis was on effort and memory:

• Can you write a linked list from scratch?

• Can you check if this text is a palindrome?

• Can you find the duplicates in this array of numbers?

If you were a developer who put in long hours analyzing problem statements, manually debugging critical issues, and hand-crafting thousands of lines of source code, you were seen as a dedicated, high-valued employee.

Today, the effort alone is no longer a metric for success.

If you spend hours writing regular expressions or standard authentication flows that an AI agent can scaffold within two minutes, the industry doesn't reward you for your six hours of hard work. The industry asks: "What value did you add beyond what the machine generated?"

This is an uncomfortable truth, but accepting it could be the turning point in your career. Once you accept that AI can write code, your mindset shifts. You start accepting that you no longer have to worry about your execution speed, and you need to focus on System Composition and Abstract Thinking.

If you're a front-end developer today, your job is no longer limited to translating a Figma design into pixel-perfect React components. An AI coding assistant can do 80% of that in a few constructive prompts. Your job role expectations as a front-end developer are now shifted to:

• When that UI connects to the backend, and 10K users log in concurrently, how does the system behave?

• Suppose a customer has an SLA (Service Level Agreement) stating that the dashboard must render with all data in 1.2 seconds on a slow 4G network, in 500 ms on a fast 4G network, and in 12 ms on a 5G network. How do you architect your Next.js application to meet that?

• Are you leveraging server-side rendering, static generation, or edge caching correctly?

• How does the application behave for users depending on screen readers?

Source code is no longer the primary output. It should be the byproduct of your thinking and reasoning. You need to anticipate edge cases, and most importantly, you need to take ownership.

AI can write an API, but AI can't sit in a meeting with a furious client and explain why the production database went down. AI cann't own the consequences of a system failure. That accountability belongs entirely to you.

Applying a Clean Architecture

Suppose you ask an LLM to build a complex application, say, an e-commerce product dashboard with sorting, filtering, and pagination. It will gladly generate the code that you'll be able to run and render on the browser. Bur AI has a very peculiar tendency in that it loves to build monoliths.

The AI will likely output a massive 1000+ line React component. The state management, UI rendering, data fetching, and business logic will be clubbed together in a single file. So it'll technically work in the browser, but it will be a nightmare to test, maintain, and scale.

This is where the human software engineers come in. A modern engineer understands clean code principles and design patterns. Instead of accepting the monolith AI output blindly, the engineer thinks in terms of LEGO-block compositions of React components.

A capable engineer looks into the requirements and thinks, " We shouldn't put everything in a single file. Let's use the Compound Components Pattern here to make the UI flexible. Let's use the Slot Pattern to create holes in our layout so consumers of this component can pass in their own custom elements without breaking the underlying logic."

You apply abstract thinking. You ask architectural questions:

• How are we managing side effects vs. the data fetching?

• Can we swap out the payment provider later with a very small code change?

• What happens if the network drops while the user is filtering?

AI provides us with the bare metal raw materials. We need to provide the engineering discipline on top of it to make it production-ready.

A Practical, AI-Era Engineering Roadmap

Now, it's time to think about how to bridge the gap between a tutorial-driven developer and a modern, impact-driven engineer. Here is a practical stage-by-stage roadmap for you.

Step 1: Strengthen Your Fundamentals

You can't use AI effectively if you don't understand the code it generates. In the past, a surface-level knowledge of a framework would have been enough for you to execute your tasks. You might have gotten away without knowing the "under the hood" aspects of it.

Today, AI abstracts the frameworks. If something breaks underneath, you're multiple layers away from the actual problem. Having a strong fundamental knowledge will help you to battle this situation, and you'll enjoy working with AI even more.

You must go deep into the fundamentals of Computer Science & Web Technologies:

• How does the internet work? Understand Networking basics.

• Don't just learn to write JavaScript promises. Learn about the event loop. Understand the call stack, the microtask queue, and how memory allocation works.

• When a React application has a memory leak, AI will struggle to find it if it spans multiple files. You need to know how to use Chrome DevTools memory profilers.

• Instead of focusing on random algorithmic puzzles, focus on applied abstract thinking. If you're building a real-time collaborative document editor, how do you manage the data structure for concurrent edits? This is how DSA is tested in this era of technical interviews.

Step 2: Build Real (Uncomfortable) Systems

Stop building TODO apps. Stop building basic CRUD applications that only work in an ideal, localhost environment. Learn to build systems to handle failures.

Instead of building a generic e-commerce clone, build an Automated E-book Delivery and Waitlist system. For example,

• The stack: Tanstack Start for the front end, NestJS for the API, Supabase for the database, Razorpay for payment processing, Firebase for social logins, and Resend for email delivery.

• The challenge: Don't be satisfied with just making the happy path work. What happens if the Razorpay webhook fails to reach your server after a user pays? How do you implement a retry mechanism? How do you secure your Supabase database with RLS (Row Level Security) so users can only download the book they paid for? How do you prevent duplicate sign-ups on your waitlist?

When you build systems like this, you naturally run into complex real-world problems. Solving these, you'll build the exact engineering muscles that companies are now desperate to hire.

Step 3: Master the Art of Debugging

When the system breaks in production, panic starts. The developers who can stay calm, isolate assumptions, trace problems, and fix them are invaluable.

AI is great at explaining isolated error messages, but it can't easily debug a distributed system where a frontend state mismatch is caused by a race condition in a backend microservice. That's on you to burn the midnight oil and get it done.

As a software developer at any level:

• Learn how to implement structured logging in your code.

• Learn how to read a stack trace systematically.

• Practice fixing performance bottlenecks without causing regressions in other parts of the application.

• Understand Web Vitals (LCP, CLS, INP, and so on.) and how to profile a slow rendering page.

Step 4: Use AI as a Tool, Not as a Crutch

First of all, stop blind copy-pasting AI responses. Treat AI like an incredibly fast, highly confident, but slightly carefree junior developer.

• Use it for boilerplate: Need an ExpressJS setup? Zustand store set up? Generate it.

• Use it for research: Learning a new thing like Rust, Go, or Cybersecurity? Prompt the AI to generate a 30-day learning roadmap tailored to your existing programming language knowledge.

• Use it for content: Want to write a READ ME file? Want to brainstorm a DRAFT idea? AI can be your companion.

• Use it for scaffolding: Need to write unit tests for a utility function? Let AI scaffold the test suites.

Note, every time you copy code from an LLM without understanding it, you're creating tech debt unknowingly. Your job is to make the AI's response as optimal as possible for production.

If you prompt an AI to write a complex data aggregation logic, and it outputs 72 lines of reducer function, don't just copy-paste it. Read it line-by-line, and ask yourself: Is this optimal? What's the Big O time complexity of this code? Can I make it more readable?

Step 5: Establishing a Strong Proof of Work

A résumé listing your skills or a certificate from a bootcamp aren't very strong proof of work achievements today.

Strong proof of work looks like:

• A GitHub repository featuring a complex real-world application with a beautifully written README explaining the architectural choices.

• Meaningful contributions to the open-source projects where your code had to pass serious reviews from senior maintainers.

• Writing deep tech articles or LinkedIn posts explaining how you solved a difficult rendering bug or why you chose a specific database schema for a project.

• Participating in a hackathon to build something that is either trendy, or has potential to go viral, or can bring revenue, or a combination of all of these.

Don't just code in silos. Build in public. Explain your thought process socially. When you articulate your engineering thoughts and decisions publicly, it separates you from millions of developers who are just relying on the response from ChatGPT or any other AI tools.

The diagram below captures all five steps visually for you to connect them and revisit at any point in time.

You can download this tech frame and many others from here.

The Must-Needed Mindset Shift

"It all begins and ends in your mind. What you give power to, has power over you" - by Leon Brown

If you're currently looking for a job, you need to immediately stop asking people, "Will I get a Job?" It's the wrong question. You can't be sure you'll get a job if you don't have a convincing reason why a company should hire you.

Instead, look at the job descriptions. Look at the companies you admire. Then ask yourself: "Why should they hire me in today's circumstances?"

If you don't have a convincing answer yet, that's perfectly fine! That's your baseline, and you've identified your skill gap. Your mission now is to bridge that gap.

We've entered a phase where the definition of a software engineer is sharper and more demanding than ever before. The bar is higher, but the expectations are clearer. If you refuse to adapt and insist on staying at the level of simple execution, the path forward will likely be incredibly difficult. You'll compete with AI tools that never sleep and developers who are utilizing those tools to do the work of three people.

But if you embrace the shift and move toward abstract thinking, deep fundamentals, system architecture, and true accountability, the opportunities are limitless. You're no longer competing with everyone. Your competition will be with a small set of developers willing to take up the challenge of evolving.

The software engineering of the future (read: "today") is not about typing code syntax into an editor. It's about understanding what to build, why to build it, how it impacts the business, how to design it to last, and how to use AI as a tool to accelerate things exponentially.

If You've Read This Far...

Thank You!

I'm a Full Stack Software Engineer with more than two decades of experience in building products and people. At present, I'm pushing my startup, CreoWis Technologies, and teaching/mentoring developers on my YouTube channel, tapaScript.

I'm thrilled to publish my 50th article on the freeCodeCamp platform, and it makes me exceptionally proud to give back my knowledge to the developer community. If you want to connect with me,

• Follow on LinkedIn and X

• Subscribe to my YouTube Channel

• Catch up with my React Clean Code Rules Book

See you soon with my next article. Until then, please take care of yourself and keep learning.

Errors like that could be fine for little tasks, but it's a disaster for production systems. The gap between "this worked in my ChatGPT conversation" and "this runs reliably in production" is massive. It's not closed by better prompts. It's closed by engineering.

This article is about that engineering. You'll learn the architecture patterns, failure modes, and implementation strategies that separate AI experiments from AI products.

What You'll Learn

In this tutorial, you'll learn how to:

• Understand why AI systems fail differently from traditional software

• Identify and prevent the three critical failure modes in production AI

• Implement the validator sandwich pattern for consistent outputs

• Build observable pipelines with proper monitoring and alerting

• Control costs at scale with rate limiting and circuit breakers

• Design a complete production-ready AI architecture

Prerequisites

To get the most from this tutorial, you should have:

• Basic understanding of any programming language

• Familiarity with REST APIs and asynchronous programming

• Experience with at least one LLM API (OpenAI, Anthropic, or similar)

• Node.js installed locally (optional, for running code examples)

You don't need to be an expert in any of these. Intermediate knowledge is sufficient.

Table of Contents

1. What Makes AI Systems Fundamentally Different

2. Failure Mode #1: Inconsistent Outputs

3. Failure Mode #2: Silent Failures

4. Failure Mode #3: Uncontrolled Costs

5. How to Build a Complete Production Architecture

6. Conclusion

What Makes AI Systems Fundamentally Different

Traditional software is deterministic. You write if (urgency > 8) { return 'high' } and it does exactly that, every single time. Same input, same output. Forever. You can write unit tests that cover every path. You can predict every failure mode.

AI systems, on the other hand, are probabilistic. You ask an large language model (LLM) to classify urgency and sometimes it says "high," sometimes "urgent," sometimes it gives you a 1–10 score, sometimes it writes a paragraph explaining its reasoning. Same input, different outputs, depending on temperature settings, model version, context window, and factors you can't fully control.

Here's what that looks like in practice:

The engineering challenge is: how do you build reliability on top of inherent unpredictability?

The answer is not "use a better model." The model is maybe 20% of the solution. The remaining 80% is the system you build around it.

Failure Mode #1: Inconsistent Outputs

The Problem

You ask the AI to extract a customer email from a support ticket. Sometimes you get the email back. Sometimes you get just the name. Sometimes you get a phone number. The format changes every time. Same prompt, different outputs.

Prompt: "Extract the customer email from this support ticket"

Output on Monday:    "john@example.com"
Output on Tuesday:   "Customer email: john@example.com (verified)"
Output on Wednesday:   "John Doe"
Output on Thursday: {
                       "customer_info": {
                         "email": "john@example.com"
                       }
                     }

All three outputs contain correct information, but you can't parse them programmatically. You can't route tickets, trigger workflow systems, or integrate with other code because your response data lacks consistency.

The Solution: The Validator Sandwich Pattern

The validator sandwich pattern (also called the guardrails pattern) ensures the AI system doesn't generate or process the wrong data by sandwiching your AI between two layers of deterministic code.

Essentially, you have three layers:

1. The top bun: Input guardrails (deterministic)

2. The meat: The LLM (probabilistic)

3. The bottom bun: Output guardrails (deterministic)

Let's break down each layer.

The Top Bun: Input Guardrails

Before anything touches the AI, validate it. Reject garbage immediately, fail fast and cheaply. Here's a basic example with deterministic code that checks the data being received:

function validateTicketInput(raw): TicketInput {
  // Type checks
  if (!raw.email || typeof raw.email !== "string") {
    throw new ValidationError("Missing or invalid email");
  }

  // Format checks
  if (!isValidEmail(raw.email)) {
    throw new ValidationError(`Invalid email format: ${raw.email}`);
  }

  // Range checks
  if (!raw.body || raw.body.length < 10) {
    throw new ValidationError("Ticket body too short to classify");
  }

  if (raw.body.length > 10000) {
    throw new ValidationError("Ticket body exceeds max length");
  }

  // Return typed, validated input
  return {
    email: raw.email.toLowerCase().trim(),
    subject: raw.subject?.trim() || "No subject",
    body: raw.body.trim(),
    timestamp: new Date(raw.timestamp),
  };
}

This runs before the LLM is ever called. It's fast, cheap, and deterministic. It catches easy failures immediately.

The Meat: Structured Outputs from the LLM

Stop asking the AI for free text. Force it into a schema. Most modern APIs support this directly.

So what does "free text" mean? When you prompt an LLM without constraints, it returns unstructured natural language. The model decides the format. Sometimes it's a sentence, sometimes a paragraph, sometimes it adds extra context you didn't ask for. This makes programmatic parsing nearly impossible.

Forcing it into a schema, on the other hand, means that you explicitly tell the model: "Respond only with JSON matching this exact structure", for example. Modern LLM APIs have built-in features to enforce this. Instead of hoping the AI formats its response correctly, you make it structurally impossible for it to return anything else.

Here's the difference in practice:

Without schema enforcement (free text):

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{
    role: "user",
    content: "Classify this support ticket as bug, billing, or feature request: " + ticketText
  }]
});

// Response could be:
// "This appears to be a billing issue"
// "billing"
// "Category: Billing (confidence: high)"
// { "type": "billing" }  <- if you're lucky

With schema enforcement:

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{
    role: "user",
    content: "Classify this support ticket: " + ticketText
  }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "ticket_classification",
      strict: true,
      schema: {
        type: "object",
        properties: {
          category: {
            type: "string",
            enum: ["bug", "billing", "feature", "other"]
          },
          confidence: {
            type: "number",
            minimum: 0,
            maximum: 1
          },
          priority: {
            type: "integer",
            minimum: 1,
            maximum: 5
          }
        },
        required: ["category", "confidence", "priority"],
        additionalProperties: false
      }
    }
  }
});

// Response is GUARANTEED to be:
// { "category": "billing", "confidence": 0.89, "priority": 2 }

The response_format parameter forces the model to output valid JSON matching your schema. If it can't, the API will retry internally until it does. You get predictable, parseable data every single time.

The key difference: you're making the AI conform to your format instead of hoping it does the right thing.

The Bottom Bun: Output Guardrails

This is the most critical layer. LLMs will hallucinate. This layer catches those hallucinations before they break your database or confuse your users.

Guardrails are validation checks that run after the LLM responds. Think of them as safety barriers on a highway: they don't prevent the car from moving, but they can stop it from going off the road.

In AI systems, guardrails verify that:

1. The output matches your expected schema

2. The data types are correct

3. The values fall within acceptable ranges

4. The business logic makes sense

Alright, now you have a structured response. Now you'll want to validate it aggressively before you use it:

function validateClassification(raw): Classification {
  const required = ["category", "confidence", "priority", "reasoning"];
  for (const field of required) {
    if (raw[field] === undefined || raw[field] === null) {
      throw new ValidationError(`Missing required field: ${field}`);
    }
  }

  if (!["bug", "billing", "feature", "other"].includes(raw.category)) {
    throw new ValidationError(`Invalid category: ${raw.category}`);
  }

  if (typeof raw.confidence !== "number" || 
      raw.confidence < 0 || raw.confidence > 1) {
    throw new ValidationError(`Invalid confidence: ${raw.confidence}`);
  }

  if (!Number.isInteger(raw.priority) || 
      raw.priority < 1 || raw.priority > 5) {
    throw new ValidationError(`Invalid priority: ${raw.priority}`);
  }

  if (raw.category === "billing" && raw.priority > 3) {
    logger.warn("Suspicious: billing classified as low priority", raw);
  }

  return raw as Classification;
}

Validating aggressively means checking everything, not just schema compliance. You're validating:

• Schema compliance: Does the JSON have the right fields?

• Type safety: Is "confidence" actually a number, not a string?

• Range validity: Is confidence between 0 and 1, not -5 or 999?

• Business logic: Does the combination of fields make sense for your domain?

• Confidence thresholds: Is the AI actually confident in this answer?

If any validation fails, you don't silently accept bad data. You have three options:

1. Retry with a clearer prompt: Ask the model to try again with stricter instructions

2. Escalate to human review: Log the failure and route to a review queue

3. Use a fallback: Return a safe default value that requires human attention

The Deterministic Rule

Here's a rule to follow religiously:

If it can be solved with an if-statement, don't use AI.

Email format validation? Use regex. Date parsing? Use a date library. Checking if a string contains a keyword? Use a string method. Math? Use actual math.

AI is expensive and probabilistic. Traditional code is free, instant, and deterministic. Use AI for genuinely ambiguous tasks, extracting meaning from unstructured text, generating content, and reasoning about complex inputs. Let deterministic code handle everything else.

Failure Mode #2: Silent Failures

The Problem

Model hallucinations are quite common in AI workflows, ranging from degraded accuracy to outdated training data to misclassification issues. This is the scariest failure mode because you don't know it's happening.

Consider accuracy drift. You trained your model on 2024 data. It's now mid-2026. Your vendors changed their invoice formats. Your classification accuracy has drifted from 95% down to 71%. You won't know until you do a quarterly audit. And by then, thousands of records have been processed incorrectly.

The principle is simple: you cannot fix what you cannot see.

The Solution: Observable Pipelines

Every production AI system needs observability baked in from day one. Here's how this plays out in a production system:

In the diagram above:

1. Input arrives: A user request comes in (support ticket, document, query). You log: request ID, timestamp, user ID, input hash (for deduplication).

2. LLM Processing: The request goes to your AI model. You log which model was called, how long it took (latency), how many tokens used, what it cost, and critically, the confidence score.

3. Confidence Gate: This is where you make a routing decision:

   • High confidence (>0.8): Auto-process and execute the action

   • Medium confidence (0.6-0.8): Send to human review queue

   • Low confidence (<0.6): Immediate escalation + alert

4. Monitoring Dashboard: All this data flows into your observability tools, where you track trends over time.

With monitoring, you can detect issues in your system and address them as soon as possible. Monitoring doesn't just catch problems. It gives you data to diagnose and fix them in hours instead of months.

What you're measuring and why:

The goal is not to remove humans from the loop, it's to only involve humans when the system is genuinely uncertain.

Failure Mode #3: Uncontrolled Costs

The Problem

You test your workflow with 10 tickets. It works great and costs 50 cents. You deploy to production. 1,000 requests hit your API. Your bill: $500 for the day.

Or you write a retry loop incorrectly. It creates infinite API calls. Your bill: $5,000 for the day.

Or you're using the most expensive model for everything, including simple tasks that a cheaper model could handle.

The reality: "works for 10 requests" ≠ "works for 10,000 requests." Scale changes everything.

The Solution: Gated Pipelines with Circuit Breakers

To move from a fragile prototype to a robust production system, you must abandon the naive approach of directly connecting user inputs to LLM APIs. Instead, implement a gated pipeline.

Think of this architecture as a series of blast doors. A request must successfully pass through each gate before it earns the right to cost you money. If any gate closes, the request is rejected cheaply and quickly, protecting your budget and your upstream dependencies.

From the diagram above, these gates are:

1. The rate limiter

2. The cache check

3. The request queue

4. The circuit breaker

Let's examine each one.

Gate 1: Rate limiting

The first line of defence stops abuse before it enters your system. In standard web development, rate limiting is about protecting the server CPU. In AI development, it's about protecting your wallet.

Gate 2: Cache check

The cheapest LLM API call is the one you never have to make. Many AI requests are repeated or highly similar. Cache aggressively.

Gate 3: Request queue

LLM APIs are not like standard REST APIs; requests often take 10–30 seconds to complete. If 500 users hit "submit" simultaneously, your server cannot open 500 simultaneous connections without crashing or hitting provider concurrency limits. A request queue solves this by batching requests and processing them at a controlled rate.

Gate 4: Circuit breaker

Retry logic is necessary for transient network blips, but it is destructive during a real outage. If an LLM provider is experiencing downtime and returning 500 errors, a naive retry loop will frantically hammer their API, wasting your money on failed requests.

How to implement a gated pipeline

Here's an example implementation showing all four gates working together:

Step 1: Rate Limiter (using Redis)

import { RateLimiterRedis } from "rate-limiter-flexible";
import Redis from "ioredis";

const redis = new Redis({
  host: process.env.REDIS_HOST,
  port: 6379
});

// Rate limiting per user
const userLimiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: "rl:user",
  points: 100,        
  duration: 3600,     
  blockDuration: 60   
});

// Rate limiting globally 
const globalLimiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: "rl:global",
  points: 1000,       
  duration: 3600      
});

Step 2: Cache Layer

import { createHash } from "crypto";

class AICache {
  private redis: Redis;
  private ttl: number = 3600; 

  hashInput(input: string): string {
    return createHash("sha256").update(input).digest("hex");
  }

  async get(input: string): Promise {
    const key = `ai:cache:${this.hashInput(input)}`;
    const cached = await this.redis.get(key);
    
    if (cached) {
      // Cache hit - free!
      await metrics.increment("ai.cache.hits");
      return JSON.parse(cached);
    }
    
    await metrics.increment("ai.cache.misses");
    return null;
  }

  async set(input: string, result: T): Promise {
    const key = `ai:cache:${this.hashInput(input)}`;
    await this.redis.setex(key, this.ttl, JSON.stringify(result));
  }
}

Step 3: Request Queue

import Queue from "bull";

const aiQueue = new Queue("ai-requests", {
  redis: {
    host: process.env.REDIS_HOST,
    port: 6379
  }
});

aiQueue.process(5, async (job) => {
  // Only 5 simultaneous LLM calls max
  const { ticket } = job.data;
  return await callLLM(ticket);
});

async function enqueueRequest(ticket: Ticket) {
  const job = await aiQueue.add(
    { ticket },
    {
      attempts: 3,
      backoff: {
        type: "exponential",
        delay: 2000
      }
    }
  );
  
  return job.finished(); 
}

Step 4: Circuit Breaker

enum CircuitState {
  CLOSED,   
  OPEN,     
  HALF_OPEN 
}

class CircuitBreaker {
  private state = CircuitState.CLOSED;
  private failures = 0;
  private lastFailureTime?: Date;
  private successesInHalfOpen = 0;

  private readonly failureThreshold = 3;
  private readonly openDurationMs = 5 * 60 * 1000; 
  private readonly halfOpenSuccesses = 2;

  async execute(
    fn: () => Promise,
    fallback?: () => T
  ): Promise {
    if (this.state === CircuitState.OPEN) {
      const elapsed = Date.now() - (this.lastFailureTime?.getTime() || 0);
      
      if (elapsed < this.openDurationMs) {
        // Still in open state - use fallback or throw
        if (fallback) {
          logger.warn("Circuit OPEN - using fallback");
          return fallback();
        }
        throw new Error("Circuit breaker OPEN - service unavailable");
      }
      
      // Transition to half-open
      this.state = CircuitState.HALF_OPEN;
      logger.info("Circuit transitioning to HALF_OPEN");
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    if (this.state === CircuitState.HALF_OPEN) {
      this.successesInHalfOpen++;
      
      if (this.successesInHalfOpen >= this.halfOpenSuccesses) {
        // Service recovered - close circuit
        this.state = CircuitState.CLOSED;
        this.failures = 0;
        this.successesInHalfOpen = 0;
        logger.info("Circuit CLOSED - service recovered");
      }
    } else {
      this.failures = 0;
    }
  }

  private onFailure() {
    this.failures++;
    this.lastFailureTime = new Date();

    if (this.state === CircuitState.HALF_OPEN) {
      // Failed during test - back to open
      this.state = CircuitState.OPEN;
      this.successesInHalfOpen = 0;
      logger.error("Circuit reopened during HALF_OPEN test");
    } else if (this.failures >= this.failureThreshold) {
      // Too many failures - open circuit
      this.state = CircuitState.OPEN;
      logger.error(`Circuit OPEN after ${this.failures} failures`);
    }
  }
}

Step 5: Putting it all together

const cache = new AICache();
const circuitBreaker = new CircuitBreaker();

async function processWithGatedPipeline(ticket: Ticket) {
  try {
    await userLimiter.consume(ticket.userId);
    await globalLimiter.consume("global");
  } catch (error) {
    throw new Error("Rate limit exceeded. Please try again later.");
  }

  const cacheKey = ticket.body;
  const cached = await cache.get(cacheKey);
  if (cached) {
    logger.info("Cache hit - returning cached result");
    return cached;
  }

  const queuedResult = await enqueueRequest(ticket);

  const result = await circuitBreaker.execute(
    async () => {
      const classification = await callLLM(ticket);
      await cache.set(cacheKey, classification);
      return classification;
    },
    () => ({
      category: "other",
      confidence: 0,
      requiresHumanReview: true,
      reason: "service_unavailable"
    })
  );

  return result;
}

What this achieves:

• Rate limiting: Prevents abuse and runaway costs

• Caching: 30-40% cost reduction on repeated queries

• Queueing: Prevents server overload during traffic spikes

• Circuit breaker: Fails fast during outages instead of wasting money on retries

Each gate is cheap to operate. Together, they protect your system from the most common production failures.

How to Build a Complete Production Architecture

When you combine all three failure mode solutions-consistent outputs, observability, and cost control, you get a complete production architecture.

When you solve for all three major failure modes, inconsistent outputs, silent failures, and uncontrolled costs. You graduate from a simple script to a true enterprise-grade system. This architecture doesn't just generate text; it actively protects itself, manages resources, and learns from its mistakes.

The Complete Workflow Implementation

Here's how all the pieces we've covered fit together in a single workflow. This brings together the validation functions from Failure Mode #1, the observability from Failure Mode #2, and the gated pipeline from Failure Mode #3:

class TicketWorkflow {
  async processTicket(rawInput: unknown): Promise<TicketResult> {
    const requestId = generateId();
    const startTime = Date.now();

    try {
      // LAYER 1: Input validation + rate limiting + cache
      const ticket = validateTicketInput(rawInput);
      await rateLimiter.consume(ticket.userId);
      
      const cached = await cache.get(ticket.body);
      if (cached) return { ...cached, source: "cache" };

      // LAYER 2: AI processing with circuit breaker protection
      const classification = await circuitBreaker.execute(() => 
        classifyTicket(ticket)
      );

      // LAYER 3: Output validation + confidence routing
      const validated = validateClassification(classification);
      
      let action: string;
      if (validated.confidence >= 0.8) {
        await sendToAgent(ticket, validated);
        action = "auto_assigned";
      } else {
        await sendToReviewQueue(ticket, validated);
        action = "needs_review";
      }

      // LAYER 4: Log everything for observability
      await logger.log({
        requestId,
        userId: ticket.userId,
        confidence: validated.confidence,
        action,
        latencyMs: Date.now() - startTime,
        cost: calculateCost(classification.tokensUsed)
      });

      await cache.set(ticket.body, validated);
      return { classification: validated, action };

    } catch (error) {
      await logger.logError(requestId, error);
      throw error;
    }
  }
}

What each layer does:

Layer 1 (Input) protects your system from bad data and abuse:

• Validates the ticket has required fields (email, subject, body)

• Checks rate limits (prevents one user from overwhelming the system)

• Returns cached results if we've seen this exact ticket before

Layer 2 (Orchestration) is where the AI does its work:

• Calls the LLM with structured output requirements

• Wrapped in a circuit breaker (fails fast if the API is down)

• Uses the cheapest model that works (Haiku for classification)

Layer 3 (Validation) ensures the output is safe to use:

• Validates the response matches our schema

• Routes based on confidence (high confidence → auto-assign, low → human review)

• Never blindly trusts AI output

Layer 4 (Observability) tracks everything:

• Logs every request with latency, cost, and confidence scores

• Sends metrics to your monitoring dashboard

• Alerts on anomalies (confidence dropping, costs spiking)

This architecture takes you from "it worked in my ChatGPT demo" to "it runs reliably at 10,000 tickets per day." The code is more complex than a simple API call, but the complexity is intentional. It's what makes the system production-ready.

Conclusion: Engineering Over Prompting

The teams winning with AI right now aren't winning because they have better models. They're winning because they've built better systems around imperfect models.

Any company can call the OpenAI API. The ones that pull ahead are the ones who wrap that API call in validation, observability, cost controls, and thoughtful architecture — the ones who treat AI as a component in an assembly line, not a creative partner in a conversation.

The three things every production AI system needs:

1. Structure: Validators, schemas, deterministic layers that enforce consistency and eliminate unpredictability at the edges.

2. Visibility: Logging, monitoring, and alerting so you catch problems in hours, not months. Observable pipelines that let you see exactly what the system is doing and why.

3. Control: Rate limits, caching, circuit breakers, and cost gates so scale doesn't turn your experiment into a budget emergency.

Reliable AI workflows aren't about better prompts. They're about better architecture around unreliable components.

If you found this helpful, you can connect with me on LinkedIn or subscribe to my newsletter. You can also visit my website.

This is the fundamental problem with single-pass AI code generation: the same context that created the code is the one evaluating it. There's no adversarial pressure. No second opinion. No fresh eyes.

What if you could structure the work so that separate agents generate and critique each other in iterative loops, the way a generator and discriminator improve each other in a GAN? The code that reaches you has already survived an argument between agents who disagreed about whether it was good enough.

This article walks through why that pattern works, how to build it, and when it is (and is not) worth the extra tokens. The concrete example is an open source project called Claude Forge, but the ideas are framework-agnostic. Anything that supports subagent spawning with fresh context windows can implement this pattern.

Table of Contents

• The Single-Pass Problem

• What the Ecosystem Is Solving

• The GAN Pattern Applied to Code

• Why Rhetorical Questions Outperform Direct Instructions

• Feedback as Filesystem

• The Zero-Context Engineer

• Phase-0: Immutable Conventions

• Convergence Design: Knowing When to Stop

• Ground Truth Documents and the Pipeline

• What the Adversarial Loop Actually Catches

• Honest Trade-offs

• When to Use This (And When Not To)

• Getting Started

Prerequisites

• Familiarity with Claude Code or a similar AI coding agent

• A working installation of Claude Code (for the hands-on sections)

• Basic understanding of how LLM context windows work

• Git installed and configured

No machine learning background is required. The GAN concepts are explained from first principles where they appear.

The Single-Pass Problem

The AI generates code in one pass. If it hallucinates a file path, misunderstands the architecture, or writes tests that don't actually test anything, you catch it during review. Or worse, you don't.

This isn't a hypothetical. Anyone who has used AI coding agents at scale has seen placeholder tests like expect(true).toBe(true), phantom dependencies where Phase 2 assumes a model that Phase 1 never creates, and instructions so ambiguous that two valid interpretations exist. These aren't rare edge cases. They're the predictable failure mode of single-pass generation.

The problem compounds with task complexity. A simple utility function generates fine in one pass. An auth middleware with token refresh, error handling, rate limiting, and logging across multiple files? The agent starts cutting corners, because the entire generation happened inside one context window that is simultaneously tracking the plan, the code, the tests, and the growing weight of its own prior reasoning.

What the Ecosystem Is Solving

There is a growing ecosystem of frameworks tackling different aspects of this problem. They each bring real contributions worth understanding.

Superpowers focuses on development methodology. It uses subagent-driven development, TDD enforcement, and multi-stage review. The framework generates a design spec, then an implementation plan, then dispatches subagents to execute. Review subagents check the output, and if they find issues, the implementer revises and gets re-reviewed until approved.

Get Shit Done (GSD) focuses on context engineering. Its key insight is fighting context window degradation through fresh 200k subagent contexts, parallel wave execution, and XML-structured plans. A JavaScript CLI handles the deterministic work (tracking progress, dependency ordering, context budgets) so the LLM never wastes tokens on bookkeeping it would do unreliably anyway.

Both frameworks share a crucial design decision: fresh context windows. When an agent has been reasoning for 100k tokens, its attention degrades. By spawning subagents with clean 200k contexts, these frameworks sidestep the "context rot" problem that plagues long-running agent sessions.

Where these frameworks diverge is in how they handle quality assurance. GSD relies on mechanical verification: lint, test, type-check, and auto-fix retries if the checks fail. There is no agent reading another agent's code to assess whether it matches the spec's intent. The "review" is whether npm run test passes.

Superpowers does have agent-to-agent review with iterative loops. But the review is enforced by in-context instructions, which means the agent can (and frequently does) rationalize skipping the review step to save tokens.

This is a known issue in the project. When review enforcement lives inside the same prompt that the model is also using to make efficiency decisions, the model sometimes decides that review is not worth the cost.

The adversarial GAN pattern addresses this differently. Instead of asking an agent to review its own work or trusting in-context instructions to enforce review, it structures the pipeline so that review is architecturally mandatory. The reviewer is a separate agent that cannot be skipped, because the orchestrator will not advance the pipeline without the reviewer's signal. The reviewer cannot modify source code, only feedback.md. The generator cannot approve its own output. Role separation is enforced by the system, not suggested by the prompt.

The GAN Pattern Applied to Code

In machine learning, GANs pit two networks against each other: a generator creates content, a discriminator evaluates it, and the feedback loop between them drives both to improve. The generator gets better at producing realistic output. The discriminator gets better at finding flaws. The adversarial tension is what produces quality.

Applied to software development, this creates two stacked feedback loops:

Each role runs as a separate agent with its own fresh context window. The Plan Reviewer has never seen the Planner's reasoning process. It only sees the output. The Code Reviewer has never seen the Implementer's struggles. It only sees the code.

This separation fundamentally changes what the reviewer can catch. When a reviewer shares context with the generator, it inherits the generator's blind spots. When a reviewer starts fresh, it reads the plan the way an actual engineer would: with no assumptions about what the author "meant" versus what they wrote.

The adversarial Plan Reviewer doesn't just verify structure. It actively tries to break the plan:

• Deadlock search: Is there a task ordering that would deadlock the implementer? (Task 3 needs the output of Task 5.)

• False positive verification: Could any verification checklist pass even with a wrong implementation?

• Ambiguity search: Are there instructions that could be interpreted two valid ways?

• Missing context: Could the implementer get stuck because a task assumes knowledge not provided?

This is where the GAN analogy is most literal. The discriminator isn't checking if the plan looks good. It's trying to find failure modes.

Why Rhetorical Questions Outperform Direct Instructions

When a reviewer finds an issue, there are two ways to communicate it.

Direct instruction:

Fix line 45: the error handler returns 500 instead of 401 for invalid tokens.

Rhetorical question:

Consider: The test test_invalid_token_rejection expects a 401 status code.
Are you returning the correct HTTP status in your error handling?

Think about: In src/auth/middleware.js:45, what happens when the token is
invalid? Is the error properly caught?

Reflect: Look at how other middleware handles auth errors. Are you following
the same pattern?

The direct instruction produces a mechanical edit. The agent changes line 45 and moves on. The rhetorical question produces a deeper investigation. The agent re-examines the surrounding code, considers the pattern used elsewhere, and is more likely to find the root cause rather than just patching the symptom.

This maps to how the underlying models work. When given an explicit instruction, the model follows it literally. When guided to reason about a problem, it activates a broader search through its understanding of the codebase. The fix addresses related issues that a mechanical edit would miss.

Reviewer prompts structured around "Consider," "Think about," and "Reflect" prefixes consistently produce better fixes than "Fix" or "Change" directives. The implementer agent receives these as feedback in feedback.md and addresses them in the next iteration of the GAN loop.

Feedback as Filesystem

Most agent orchestration systems rely on some form of message passing: API calls, databases, queue systems, in-memory state. These all work, but they introduce infrastructure dependencies and make the agent conversation opaque after the fact.

An alternative: use the filesystem as the message bus and git as the orchestration layer.

All agent communication flows through feedback.md, a structured markdown file with two sections:

## Active Feedback (OPEN)

### FB-001: Auth middleware missing rate limiting
- **Status:** OPEN
- **Source:** Plan Reviewer
- **Phase:** 1
- **Detail:** The plan specifies JWT validation but does not address rate
  limiting for failed auth attempts. Consider: what happens if an attacker
  brute-forces tokens?

## Resolved Feedback

### FB-000: Missing error codes in API spec
- **Status:** RESOLVED
- **Resolution:** Added error code table to Phase-0 conventions

This design has several properties that matter in practice:

Full audit trail: Every piece of feedback, every resolution, every signal is committed to git alongside the code it produced. When you want to understand why the auth middleware was designed a certain way, the conversation that shaped it is right there in the commit history.

State recovery: If a pipeline gets interrupted (token limits, network issues, you need to step away), resuming is trivial. The orchestrator re-reads feedback.md and git log, determines what stage the pipeline reached, and picks up where it left off. No cloud infrastructure, no database, no queue. Just files.

Transparency: You can read the agent conversation in your editor. You can see exactly what the reviewer flagged, exactly how the implementer responded, and whether the resolution actually addressed the concern.

Agents communicate through structured signals routed by the orchestrator:

• PLAN_COMPLETE / REVISION_REQUIRED / PLAN_APPROVED (plan GAN loop)

• IMPLEMENTATION_COMPLETE / CHANGES_REQUESTED / PHASE_APPROVED (code GAN loop)

• GO / NO-GO (final gate)

• VERIFIED / UNVERIFIED (post-remediation verification)

Each signal marks a state transition. The orchestrator reads the signal, determines the next agent to invoke, and passes it the relevant context. The orchestrator itself is a Claude Code session, but the agents it spawns are fresh subagents with clean context windows.

The Zero-Context Engineer

One of the most effective constraints in the system is the "zero-context engineer" framing. The Planner writes every plan as if it will be executed by an engineer who:

• Is skilled but has zero context on the codebase

• Is unfamiliar with the toolset and problem domain

• Will follow instructions precisely

• Will not infer missing details. If it's not in the plan, it won't happen.

This constraint forces explicit instructions. No "add the usual auth middleware." Instead: which library, which pattern, which error codes, which files to create, which existing files to modify, and how to verify the result.

The Plan Reviewer then simulates this zero-context experience: "If I knew nothing about this codebase, could I follow these instructions and produce a working result?"

This framing catches a class of failures that are invisible to someone with context. The author of the plan knows what they meant. The zero-context reviewer only knows what is written. The gap between intention and specification is where bugs live.

Phase-0: Immutable Conventions

Every pipeline run starts with a Phase-0 document that defines immutable rules: tech stack, testing strategy, deployment approach, shared patterns, commit format. Every subsequent phase inherits from Phase-0. Every reviewer checks against it.

This solves a common multi-agent problem: drift. Without a shared source of truth, Agent A might decide to use Jest while Agent B sets up Vitest. Agent C might use a different error handling pattern than Agent D. Phase-0 prevents this by establishing conventions before any code is written.

The conventions aren't suggestions. They're constraints that every agent in the pipeline must respect, and every reviewer must verify against.

Convergence Design: Knowing When to Stop

An adversarial loop without exit conditions is just two agents arguing forever. The convergence design has three mechanisms:

Iteration caps: Each GAN loop (plan review, code review) runs a maximum of 3 iterations. If the planner and reviewer cannot converge in 3 rounds, the issue requires human judgment, not more machine cycles.

Signal protocol: The structured signals (PLAN_APPROVED, GO, NO-GO) are explicit state transitions, not suggestions. When the final reviewer issues NO-GO, the pipeline rolls back the phase. There is no "let's try one more time." The rollback is automatic.

Token budget: Each phase targets roughly 50k tokens with a 75k hard ceiling. This prevents any single phase from consuming the entire context budget and ensures the orchestrator retains enough headroom to manage the pipeline.

These caps exist because adversarial loops have a cost curve. The first iteration catches major issues. The second iteration catches subtle issues. The third iteration catches edge cases. A fourth iteration almost never catches anything the previous three missed, but it costs just as many tokens. Three iterations hit the sweet spot between thoroughness and efficiency.

Ground Truth Documents and the Pipeline

The adversarial pipeline doesn't start from a vague prompt. Every workflow begins with an intake skill that produces a structured ground truth document. The pipeline then runs from that document, not from the original user request.

Brainstorm: Turning Ideas into Specs

The /brainstorm skill is the feature creation workflow. Given a feature idea, it first explores the codebase to understand the existing architecture, tech stack, and patterns. Then it asks 5-15 clarifying questions designed to front-load high-impact decisions:

The codebase uses DynamoDB for storage. For this feature's data, should we:

A) Add tables to the existing DynamoDB setup
B) Use a different storage approach (e.g., S3 for documents)
C) Both - DynamoDB for metadata, S3 for content

These aren't generic questions. They're grounded in what the skill found during codebase exploration. The skill identifies the real decision points for this specific project and surfaces them before any planning or code generation begins.

The output is brainstorm.md, a structured design spec. Not a conversation transcript, but a distilled set of decisions that the Planner agent can consume cold. This document becomes the single source of truth for the entire pipeline run.

Repository Evaluation, Health, and Documentation Audits

The same ground-truth-document pattern applies to the audit workflows:

• /repo-eval spawns three evaluator agents in parallel (the Pragmatist, the Oncall Engineer, the Team Lead), each scoring the codebase from a different lens across 12 pillars. The output is eval.md.

• /repo-health runs a technical debt auditor across four vectors (architectural, structural, operational, hygiene). The output is health-audit.md.

• /doc-health runs six detection phases comparing documentation against actual code. The output is doc-audit.md.

• /audit runs any combination of the above. It asks scoping questions once, then spawns up to 5 agents in parallel (3 evaluators + health auditor + doc auditor). All intake documents land in one directory.

Each of these intake skills produces a read-only assessment. The agents doing the evaluation never modify the codebase. They only write their findings into the intake document.

The Pipeline Runs from Ground Truth

The /pipeline skill reads whatever intake documents exist and runs the adversarial GAN loop from them. For a feature, it reads brainstorm.md. For an audit, it reads whichever combination of eval.md, health-audit.md, and doc-audit.md are present.

When multiple intake documents exist (from a combined audit), the Planner reads all findings together and consolidates overlapping concerns into a single unified plan. Phases are tagged by implementer type and ordered:

1. [HYGIENIST] phases first, subtractive cleanup (deleting dead code, simplifying over-abstractions)

2. [IMPLEMENTER] phases next, structural fixes on clean code

3. [FORTIFIER] phases next, locking in the clean state (linting, CI checks, git hooks)

4. [DOC-ENGINEER] phases last, documentation reflecting final code

The ordering matters. You don't want the implementer building on top of dead code that the hygienist would have removed. You don't want the doc-engineer documenting an API that the fortifier is about to add validation to.

This separation between intake and pipeline is deliberate. The intake skills are exploratory and interactive. They ask questions, explore the codebase, and produce a document. The pipeline is autonomous. It reads the document and runs through the adversarial loops with minimal human intervention, stopping only at explicit decision points.

What the Adversarial Loop Actually Catches

In practice, the adversarial loops catch issues that single-pass generation consistently misses.

Plan Review catches:

• Hallucinated file paths (the Planner says "modify" a file that doesn't exist)

• Phantom dependencies (Phase 2 assumes a model that Phase 1 never creates)

• Test strategies that require live cloud resources instead of mocks

• Ambiguous instructions that a zero-context engineer could misinterpret

• Deadlocks in task ordering (Task 3 needs the output of Task 5)

Code Review catches:

• Placeholder tests (expect(true).toBe(true))

• Deviations from Phase-0 architecture conventions

• Missing error path coverage (only happy paths tested)

• Hardcoded secrets and input validation gaps

Verification catches:

• Remediation targets that weren't actually addressed

• Regressions introduced during fixes

• Partial fixes where the symptom changed but the root cause remains

An earlier design re-ran the full evaluator or auditor agents after remediation, 3-5 agents re-scanning the entire codebase. This was token-expensive and redundant since the per-phase reviewers had already verified each fix. The current design uses a single verification agent with a targeted scope: read the original intake document findings and check each specific file:line location. One agent, targeted scope, a fraction of the tokens. Evaluator and auditor agents run exactly once (during intake) and never again.

Honest Trade-offs

This pipeline is not free. There are some trade-offs you'll want to consider and be aware of:

Token Cost

Multiple agents reviewing each other's work uses significantly more tokens than a single-pass approach. The adversarial loops can triple the total token usage for a feature. On a subscription plan, this means hitting session limits faster. On API billing, this means real money.

Time

A feature that takes one agent 10 minutes might take the pipeline 30-45 minutes with review loops. Multi-agent frameworks in general are slower than single-pass. The adversarial loops add time on top of the orchestration overhead that any multi-agent system carries.

Orchestrator Context Pressure

The orchestrator accumulates agent result summaries across phases. Long pipelines with many phases may hit context compression, which degrades the orchestrator's ability to route effectively.

Not Fire-and-Forget

Despite the automation, complex features benefit from human checkpoints. The pipeline stops and asks for judgment at key moments. If you skip those checkpoints, you may end up with technically correct code that misses the actual requirement.

Diminishing Returns on Simple Tasks

For a quick script, a utility function, or a prototype, the adversarial overhead is pure waste. Single-pass generation is faster, cheaper, and sufficient.

The trade-off is worth it for features where correctness matters more than speed: anything touching auth, payments, data integrity, or infrastructure. When the cost of a bug in production exceeds the cost of the extra tokens to prevent it, the math works. For everything else, single-pass is fine.

When to Use This (And When Not To)

Use adversarial multi-agent patterns when:

• The feature touches authentication, authorization, or session management

• The code handles payments or financial transactions

• Data integrity is critical (migrations, schema changes, ETL pipelines)

• Infrastructure changes could affect production (IaC, CI/CD modifications)

• The codebase is unfamiliar to the agents (large legacy systems)

Use single-pass generation when:

• Prototyping or exploring an idea

• Writing utility scripts or one-off tools

• Making small, well-scoped changes to familiar code

• Speed matters more than thoroughness

• You will review the output carefully yourself anyway

Getting Started

Claude Forge is built entirely from Claude Code custom skills. No external tooling, no CI integration required. Install by copying the skills directory into your project:

git clone https://github.com/hatmanstack/claude-forge.git
cp -r claude-forge/.claude/skills/ /path/to/your-project/.claude/skills/

Then in your project:

# Feature development
/brainstorm I want to add webhook support for payment events
/pipeline 2026-03-12-payment-webhooks

# Full audit (health + eval + docs), one command
/audit all
/pipeline 2026-03-16-audit-remediation

# Individual audits
/repo-eval
/repo-health
/doc-health

The pipeline handles the orchestration. You'll see progress reports between stages, and it will stop and ask when something needs human judgment.

Wrapping Up

The adversarial pattern (separate generator and discriminator with isolated context windows, structured feedback as the communication channel, iteration caps for convergence) can be implemented in any agent system that supports subagent spawning with fresh contexts. The specific implementation uses Claude Code skills, but the pattern is the contribution, not the tooling.

Sometimes the best code comes from the argument, not the agreement.

For those unfamiliar, a PIP at Google is a two-month plan to show improvement – a final chance to prove yourself. You’re given a project and a strict deadline. Deliver successfully, or you’re fired. There are no extensions, no middle ground.

Scary thoughts about providing for my family’s finances raced through my mind. But my deeper fear was this: what story would I carry about myself if I tried to persevere and failed?

If I got fired, I would need to face job interviews. And I knew the question would come: “Tell me about a project you worked on at Google that you’re most proud of.” The honest answer was that I didn’t have one. I hadn’t yet excelled at a project, hadn’t gone deep enough into any system to truly own it. I imagined myself sitting in an interview, with a blank face, with nothing to say.

That dreadful image became my motivation. I wanted a project I could truly own, something I could explain inside and out, regardless of how the PIP ended. I’m also not the type of person who simply backs down when things get tough. I needed to prove to myself that I could rise up. I was gonna give the project everything I had, week after week after week. That singular commitment became the start of my transformation into a more focused, disciplined engineer.

In this guide, you’ll learn how to turn professional setbacks into catalysts for growth. While examining my journey on Google’s Performance Improvement Plan, I’ll show you how to face underperformance head-on, rebuild your confidence, and come out stronger than before. You’ll see how focus, discipline, and gratitude can turn the lowest points of your career into launch ramps for acceleration.

Here’s what I’ll cover:

1. The Backstory

2. The PIP Begins

3. The Project

4. Fatherhood

5. Letting Go

6. What’s Next

7. Closing

The Backstory

To understand how I landed in that chair across from my manager, you need to know where I came from. Before Google, I’d worked at Meta. I was hired there as an IC3 (entry-level engineer) and promoted after a year to IC4 (mid-level). But that promotion didn't come from technical excellence. It came from my connecting our engineering projects to the business’ needs.

I had worked on a payments system used by large enterprises. By sitting with the operations staff and customer service reps, I spotted inefficiencies and built small features that saved time, reduced errors, and allowed the team to scale. Those changes had a big impact, and they earned me a promotion. But, in hindsight, my success had come from soft skills such as teamwork and business awareness. I hadn’t actually developed the technical knowledge expected of an IC4.

On my team at Google, technical mastery was the main thing we were measured on, while business awareness was a side point. On top of that, I had recently immigrated from the United States to Israel and needed to learn the local language of Hebrew and the local culture. It was a lot all at once: new country, new language, new company, and strong engineering expectations.

The gap between my technical skills and those of my peers eventually led to my being placed on a Performance Improvement Plan.

The PIP Begins

When the PIP started, I increased my working hours to 60 hours per week. I cut out almost everything else in my life – news, side projects, YouTube – and focused only on my work. When you’re falling behind, cutting distractions isn’t punishment. Rather, it’s how you create the quiet needed to actually improve.

It was brutal. Despite all the intense hours, I was slower than my coworkers. They shipped code confidently while I second-guessed myself. They reviewed my work and pointed out ways to improve it.

Some nights I walked away from my desk ready to cry. I was exhausted, and even after pouring in all those hours, I still wasn’t keeping pace. I felt defeated.

But I kept grinding away, day after day, week after week. Ignoring every side project and distraction forced me to confront the real issue: my lack of depth in the systems I was working on.

The Project

For confidentiality reasons, I won’t describe the actual project I was assigned. But here’s a similar example.

Imagine Google had a small music game built into Google Search. My task was to add a line of text above the game’s start button telling players how many people had reached the next level. The goal was to encourage more people to keep playing.

The text addition would run as an experiment. We’d launch it to a small percentage of users, measure the impact, and then either shut it down or roll it out to everyone.

The problem? At that time, Google Search didn’t even store how many players completed each level. So before I could add the text, I had to design and run a new data pipeline to track how many players completed each level of the game.

At first, I got lost. Just before my PIP began, our broader organization had been through a reorg, and my small team of five engineers was newly assigned to focus on games within Search. None of us had touched the gaming code before, and the particular game I was assigned hadn't had any meaningful updates in several years.

I spent days combing through design docs from 3-7 years earlier, only to find that the original authors had long since moved on. Each time I reached out, I’d get referred to someone else. Eventually, I found the current owner of the data storage systems for gaming. She had recently inherited them and hadn’t built the systems herself, but she helped me understand their current state.

With that clarity, I was finally making headway. But I realized I needed to rethink my priorities. Data pipeline code could be reviewed and deployed relatively quickly, while code changes to Google Search required slower, more comprehensive quality assurance checks. If I wanted to have any chance of meeting the PIP deadline, I had to shift focus to the Search-side work first.

As I dug deeper, another issue surfaced: the people who were listed as the engineering owners of the game hadn’t touched it in years and didn’t want to be involved anymore. I learned that our team would be running many more experiments on that code. Therefore, after consulting with my manager, I became the game’s code owner, the person ultimately responsible for all engineering decisions.

Taking ownership didn’t mean I suddenly moved fast or flawlessly. Some of my choices slowed me down. I aimed for near-perfect data accuracy when “mostly accurate” would have been enough for an experiment. I also wasted days digging through outdated documentation instead of simply reaching out to the people behind it.

After several days reading a four-year-old doc, I finally messaged the author. They immediately redirected me to someone else, who then forwarded me again. The third person turned out to be the current owner, and within minutes, they shared with me their private notes which clarified a ton.

But those mistakes were part of the learning curve. Each week, I dove deeper into the engineering tasks, internalized more of the systems, and made more progress than the week before.

By the time the final two weeks of the PIP arrived, I was operating at a whole new level. While the first month had felt like drowning, the last two weeks had felt like flying. I was excitedly diving into the code, unblocking myself, and helping teammates navigate the codebase.

That turnaround, from tears of frustration to the thrill of ownership, was exhilarating. For the first time at Google, I was independently driving my project forward. And I loved it.

When the PIP deadline arrived, though, I hadn’t yet delivered the full project. I was just a few hours of engineering work away from getting a working end-to-end flow with hardcoded data, but the actual data collection and experiment launch would have required about nine more days of engineering work.

On a PIP, “almost there” isn’t good enough. I was called in for a hearing with my director and HR, where I was given an opportunity to explain my case.

I didn’t walk into that final meeting empty-handed. I brought a detailed handoff plan listing the current state of the project, the remaining steps, and every contact and document another engineer would need to continue. I also brought a plan for improving collaboration amongst our various gaming engineers by creating a doc that would function as a centralized directory of all gaming systems, their owners, and their design docs. I offered to maintain this directory as a side effort, building it up naturally through my ongoing engineering work and conversations with past owners of the systems.

The hearing was an hour. I walked my director and HR through what I had shipped, my handoff plan, and my roadmap for unblocking future projects. I left the meeting proud of all that I’d learned over the previous two months. “Whatever will be will be,” I told myself.

A few days later, HR and my director called me back with their decision. Their feedback was straightforward: I had shown steady improvement, but I hadn’t delivered the final project on time, and therefore I hadn’t met expectations for my level.

Their feedback didn’t mention my handoff plan, my roadmap, nor my becoming the game’s code owner. That’s because a PIP isn’t a coaching program, it’s an evaluation. It doesn’t measure acceleration, it measures completion. It’s binary: You either deliver within the two months, or you don’t. And I hadn’t.

Upon hearing their decision, I thanked my director and the HR representative for having given me a final chance. I told them that the PIP had succeeded: it had built within me an internal engine of ownership over my engineering career. The fact that I would no longer be at Google was irrelevant. There would be no break in my internal transformation.

Fatherhood

The official decision closed one chapter. But the habits I’d built during the PIP of focus, ownership, and accountability began reshaping more than just my work. They changed how I saw myself as a father and husband.

Before the PIP, I’d take my toddler to the playground after work. During it, I was often too drained for that. I’d sit him in front of the TV while I caught up on writing code or reading documentation. Date nights with my wife slipped away too. For a while, I wondered: What kind of father and husband does that make me?

One night, I was listening to financial coach Dave Ramsey, a religious Christian who often brings faith into his talk show. He spoke about a father’s responsibility to provide for his family. It reframed how I saw my long hours. Had I made more disciplined decisions and strengthened my engineering skills months earlier, I never would have been placed on the PIP. The newer, more focused, harder-working version of me wasn’t the problem, it was the solution.

So as my son sat in front of the TV, I reminded myself: An earlier version of me had made decisions which resulted in my now being less available for my family. The new me, the disciplined me, hadn’t made that choice. My current unavailability was a course correction that needed to happen for me to become the type of father and husband I wanted to become.

Letting Go

When I was let go, I felt a little lost. One of my biggest worries was financial. Not only did I lose a high-paying job, but I was also frustrated that I wouldn’t receive the yearly bonus Google gives its employees. I had plans for how I would use it, and letting go of that expectation was difficult.

I spoke with my Rabbi about being let go. He told me: In Judaism, we believe that everything happens for a reason. If I lost the job, then God wanted me to lose it. He encouraged me to view my overall experience at Google in positive terms, and to focus on appreciation to God for having a plan for me. It made logical sense, but I was still frustrated about the loss of the bonus income.

The inner peace came later, when I realized something simple: I hadn’t earned that yearly bonus. My performance before the PIP hadn’t justified it. It made sense, in fact it felt right, that I didn’t receive it.

With that acceptance came space for gratitude, especially toward my former coworkers and managers. During those final two months, they reviewed my work, pointed out ways to improve it, answered my questions, and patiently explained how Google’s internal engineering systems worked. I will always be grateful for how much they taught me.

That gratitude extended to my manager as well. Several weeks after being let go, I visited the office one last time to say goodbye to my team. My manager explained that the decision to let me go had been a difficult one. He told me he liked me as a person and recognized how much I had improved during the PIP. But keeping me on would have required certainty that I was already operating at the expected engineering level, and that was something he wasn’t sure of. I understood his position. If I had been in his shoes, I would have made the same decision.

Because of the Performance Improvement Plan I had gained growth, humility, and clarity. Letting go was about moving forward with gratitude for what had gone right.

What’s Next

The PIP had given me something invaluable: structure and accountability. During those eight weeks, I lived by a project plan timeline, and when my time at Google ended, I didn’t let that habit go. The very first thing I did afterward was set up a new timeline, this time for my job hunt. Tasks were ordered by priority, with time estimates and due dates, so that my search itself became a disciplined project. My wife, or anyone else, could hold me accountable just as my manager once had.

As an example, the below table is a snippet from my job hunt timeline:

The key to creating my job hunting timeline was being clear on my priorities regarding what type of engineering position and what type of company I’d prefer to work for.

At Google, a company with tens of thousands of engineers, I used coding frameworks and technologies that were custom-built for Google engineers and used by no one outside of Google. I felt isolated from the greater world and to engineers outside of the company. So I want my next engineering role to be at a smaller company, where I’ll use popular open-source technologies and software used by engineers throughout the world.

To prepare myself for my next engineering role, I'm now laser-focused on upskilling my technical knowledge. I’ve been interviewing engineers at local startups about the technologies they use and then sharing the lessons publicly on LinkedIn. Each 1:1 interview and write-up helps close the skills gap that led to my firing at Google.

By following my written timeline and by knowing my end goal, I’ve been able to sustain long-term momentum in my job hunt.

Closing

Whatever will be will be. I’m grateful for the PIP experience, because it caused me to claw my way out of underperformance. It stripped away distractions, forced me to confront my engineering weaknesses head-on, and gave me the discipline to close the gap.

The momentum I built during those eight weeks never stopped. There was no break between week eight and week nine, just continuous acceleration. Week eight was about my PIP project, and week nine was about my job hunt. The external goals changed, but the internal engine kept running.

While my momentum softened the blow of getting fired, it didn’t erase the emotions that came with it. Sharing my story of being fired for underperformance has felt awkward and vulnerable, but also has given me a feeling of pride. Pride at who I’ve become. And pride in my giving back to the greater community, by enabling others facing similar struggles to learn from my story.

The eight weeks of the PIP were my launch ramp, and my acceleration continued long after the official PIP was over. To quote someone I know, “Like the mythical Phoenix, I believe in rising from the ashes, no matter how daunting the obstacle.” The PIP was my ashes, but it was also my fire.

Yet, a critical paradox lies at the heart of their software engineer training: most university programs still prepare them for “middle-layer” duties – wiring together pre-built libraries, cloud services, and hardware they rarely see or touch, treating the physical world as a distant abstraction.

This narrow educational focus has tangible consequences. It can blunt creativity and problem-solving skills, leaving graduates ill-prepared to design the complete, resilient solutions that society urgently needs.

This disconnect is reflected in surprising employment statistics, where computer science graduates can face higher unemployment rates than those in some non-technical fields. More importantly, it creates a generation of specialists who understand software in isolation but may lack the holistic perspective to build systems that are secure, robust, and seamlessly integrated with the physical world.

This handbook argues for a necessary evolution: a new, end-to-end engineering education that fuses software, hardware, robotics, mechanics, and cybersecurity into a single, coherent toolkit. It provides a blueprint for educators, industry leaders, and aspiring engineers to build a new generation of creators who can think across disciplines, solve complex problems from concept to deployment, and drive meaningful, sustainable progress. The moment demands not just programmers, but true system architects.

By the end of this handbook, you’ll be able to:

1. Articulate why traditional "middle integration" software education is no longer sufficient for today's technological challenges.

2. Define the core principles of End-to-End Engineering and how it integrates software with hardware, robotics, and mechanics.

3. Analyze the economic, societal, and demographic forces that demand a new, more versatile type of engineer.

4. Incorporate cybersecurity and ethical design as foundational pillars of system development, not as afterthoughts.

5. Develop a framework for overseeing and validating AI-driven systems to ensure they are reliable and secure.

6. Outline a practical, year-by-year curriculum for implementing an end-to-end engineering program.

7. Identify the benefits of this holistic approach for graduates, industry, and society as a whole.

8. Formulate strategies for overcoming common challenges in implementation, from faculty training to infrastructure investment.

Table of Contents

1. Inspiration for this Handbook

2. Why End‑to‑End Engineering Matters

3. Understanding End-to-End vs. Middle Integration in Engineering

4. Economic Challenges and Opportunities for Software Engineers

5. The Role of Institutions in Cultivating End‑to‑End Engineers

6. Proposed Reforms: Designing End-to-End Programs

7. Benefits for Graduates and Society

8. Overcoming Challenges in Implementation

9. Conclusion: A Path Forward for Engineering Education

10. Further Resources

Inspiration for this Handbook

The current educational landscape

In our complex and rapidly evolving digital world, the role of higher education as a foundation for innovation and societal progress is more crucial than ever. The rigorous systems established by universities are essential for cultivating the expertise that drives our economies forward.

At the same time, the current educational landscape presents significant opportunities for growth and adaptation. The financial model for higher education is a subject of ongoing discussion, as substantial investments from grants and endowments exist alongside rising levels of student debt. This is causing many to wonder how to best align resources with student outcomes and evolving industry needs.

This dynamic is contributing to a noticeable shift in how learners approach higher education. University degrees are no longer always seen as the exclusive pathway to a skilled career – and this trend is reflected in enrollment data across the globe.

In regions from the United States to Canada and Armenia and beyond, a significant number of university positions that were once highly competitive now remain unfilled. In response, many prospective students are diversifying their educational portfolios, pursuing industry-recognized certifications from technology leaders like Google, AWS, and Microsoft, or engaging in self-directed learning.

This suggests a broader re-evaluation of educational return on investment, as the traditional assumption of a guaranteed path from a degree to employment comes under greater scrutiny.

Evolving educational systems

Established institutions, by their nature, often take a measured approach to curricular change. This can sometimes create a gap between traditional programs and the fast-paced innovation occurring in the technology sector, where open-source knowledge and new learning platforms are becoming increasingly prevalent.

We should consider diverse global strategies in this conversation. For example, China’s model of offering extensive scholarships to international students highlights an approach focused on attracting global talent. Likewise, its emergence as a leading contributor to open-source projects and academic research demonstrates a powerful commitment to widespread knowledge sharing.

The ultimate goal of any educational system is to equip graduates with durable and relevant skills. A student’s education can be viewed as their professional operating system. A strong foundation provides the essential hardware, while a modern, integrated curriculum installs the powerful, adaptable software needed to solve complex problems and create value.

This presents a compelling opportunity for a strategic evolution in higher education. By fostering greater collaboration between academia and industry and thoughtfully integrating new hands-on learning models, we can enhance the impact and accessibility of our educational systems. The path forward lies in building a more responsive, inclusive, and sustainable framework that empowers the next generation of innovators to meet the challenges of the future.

You can download a free copy of the ebook version of this handbook here.

And you can listen to it as a podcast here:

Why End‑to‑End Engineering Matters

Employment data tell a cautionary tale. Computer science graduates currently face about 6.1% unemployment, while computer engineering majors experience a 7.5% rate – higher than fields like art history (3%) or journalism (4.4%). This mismatch stems from curricula that prize isolated coding skills over the interdisciplinary fluency modern industry expects.

Big-tech titans such as Apple, Amazon, Alphabet, Meta, Microsoft, Nvidia, and Tesla push the frontier of AI and automation, but they also expose society to new vulnerabilities – from misinformation cascades to brittle supply-chain software. And there are valid criticisms of universities – such as outdated approaches that reinforce these vulnerabilities in various ways. For example, many university programs focus courses on stitching together third-party APIs or cloud SDKs, leading students to depend on vendor ecosystems rather than building foundational technologies themselves. But, these institutions remain invaluable assets for any country.

MIT is still MIT, and Stanford continues to produce some of the world's best engineers, driving innovation through cutting-edge programs. Universities overall generate a massive workforce that transforms fields, along with groundbreaking research papers that advance global knowledge.

But many universities are being left behind due to insufficient investment in the education system and systemic inefficiencies, which are causing huge troubles for the entire world. For instance, nations need to keep pace with aging populations, where rising old-age dependency ratios – projected to increase significantly by 2055 – mean fewer workers supporting more retirees. This will potentially requiring two people to effectively pay for one non-worker through higher taxes and social security burdens.

This is evident in aging societies like Japan, Denmark, and Finland, where top personal income tax rates exceed 55%, and citizens face mounting fiscal pressures to fund pensions and healthcare.

Security is another critical concern: even nuclear agencies are being hacked, as seen in the July 2025 breach of the U.S. National Nuclear Security Administration (NNSA) by Chinese state-sponsored hackers exploiting Microsoft SharePoint vulnerabilities.

These issues highlight the urgent need for universities to foster resilient, skilled talent that can safeguard economies and societies. What this likely means is a shift away from traditional models – like over-relying on international student tuition and exorbitant fees – toward hands-on, open-source styles that democratize learning.

For example, organizations like freeCodeCamp, alongside tech giants such as Google, Microsoft, and Amazon, are open-sourcing vast engineering content that rivals entire university curricula, all without massive endowments or campus infrastructures.

Google's AI tools, like NotebookLM for generating educational content, OpenAI's agents for interactive learning, and productivity boosters such as Cursor (despite its limitations in studies showing 19% slower task completion due to bugs) are unlocking doors previously locked by institutional barriers.

These innovations allow single engineers to achieve more, as industry can no longer afford inefficiencies. This has been made clear by companies rapidly adopting alternatives to traditional systems, swapping locked gates for open pathways to boost output and adaptability.

In the context of educational institutions, end-to-end curricula offer a different path. By combining rigorous software foundations with hardware prototyping, robotics labs, mechanical design, and embedded security, universities can graduate engineers who understand an entire system’s life cycle – from concept sketches and circuit diagrams to secure deployment in the field.

Such breadth does more than widen a résumé. It also empowers graduates to spot hidden failure points, slash integration overhead, and create novel products that are both robust and ethically sound. The payoff is twofold. First, students gain adaptability: a graduate who can write control firmware, machine-learning inference code, and penetration tests is far harder to automate or outsource.

Second, industry gains innovators who can push technology forward without leaning exclusively on closed-source toolchains. This reduces systemic risk and diversifies the ecosystem.

This handbook sets out the full case for such a transformation. We will examine the economic and societal forces demanding new skills, survey pioneering institutions already leading the charge, and map a practical blueprint for universities ready to pivot.

The goal is simple: equip tomorrow’s engineers to build end-to-end solutions that drive progress responsibly – and ensure they share equitably in the value they create.

Understanding End-to-End vs. Middle Integration in Engineering

The Scope of Traditional Software Engineering

Traditional software engineering education focuses on intermediary roles, where engineers develop software to bridge users and systems – such as connecting databases to applications, devices to networks, or algorithms to outputs.

This "middle" integration approach often involves working with pre-existing hardware, such as laptops from manufacturers like Dell or Apple, and leveraging APIs or cloud services provided by leading tech companies.

While it’s effective in specific contexts, this focus can lead to inefficiencies, as engineers dedicate significant time to managing integrations rather than creating innovative solutions. Also, reliance on third-party tools can introduce complexities, including compatibility issues or security vulnerabilities, which require ongoing maintenance and can limit creative problem-solving.

For example, engineers working with cloud platforms may spend considerable effort resolving version conflicts or debugging third-party APIs, diverting resources from developing new features. This dynamic can also expose systems to risks, as external tools may contain outdated libraries or vulnerabilities that require constant updates.

The 2020 SolarWinds hack, which compromised organizations through a supply chain attack, illustrates the challenges of fragmented development, where reliance on external components can introduce unforeseen risks.

The Vision of End-to-End Engineering

End-to-end engineering education adopts a holistic approach, training students to oversee every stage of system development, from ideation to deployment. This encompasses software development, hardware prototyping, mechanical engineering for physical systems like robotics, and cybersecurity to ensure system integrity.

For instance, an end-to-end engineer might design a robotic arm’s software, optimize its mechanical components for precision and durability, and embed security protocols to protect against cyber threats. This comprehensive skill set helps engineers create integrated, resilient systems that minimize reliance on external tools and enhance system reliability.

The benefits of this approach are multifaceted. Robotics training equips engineers to address physical constraints, such as sensor accuracy, motor efficiency, or material strength, fostering innovation in fields like autonomous vehicles, industrial automation, and medical robotics.

Mechanical engineering bridges the digital and physical realms, enabling engineers to design systems that interact seamlessly with the real world.

Cybersecurity integration is critical in an era of increasing connectivity, as devices like robots and IoT systems face growing risks of cyber threats. For example, industrial robots designed with embedded security can prevent disruptions like the Stuxnet attack, which targeted control systems, ensuring operational continuity and safety.

Addressing Curriculum Gaps

Current software engineering curricula, typically spanning 120-130 credits over four years, cover foundational topics such as mathematics (calculus, linear algebra), programming languages (Python, Java, C++), data structures, and software design principles. While these are essential, programs often include courses like introductory chemistry or unrelated electives that may not align with modern industry needs, consuming valuable time and resources.

Meanwhile, key interdisciplinary skills – robotics, mechanical engineering, and cybersecurity – are often underrepresented, leaving graduates less prepared for real-world challenges where software must integrate with hardware under security constraints.

This curriculum gap can impact graduates’ economic outcomes. At companies like Meta, engineers earn competitive salaries ($210,000 to $3.67 million annually, including bonuses and stock), yet the broader distribution of corporate profits, such as Meta’s $39 billion in 2023, tends to favor executives and shareholders.

Similarly, Vivaro, an online casino platform based in Armenia, has leveraged the country’s relatively low labor costs and favorable government relations to achieve rapid growth with minimal regulatory oversight, highlighting how companies can benefit from localized economic advantages.

This dynamic underscores how reliance on integration-focused roles can limit engineers’ ability to capture the full value of their work, as companies maximize profits through strategic labor and regulatory practices.

End-to-end education addresses this by equipping engineers with versatile skills to innovate independently, pursue entrepreneurial ventures, or lead multidisciplinary projects, enabling them to contribute meaningfully and share more equitably in the value they create.

Pioneering Models for the Future

Institutions like MIT are leading the way with programs that integrate computer science, electrical engineering, robotics, and cybersecurity.

MIT’s Department of Electrical Engineering and Computer Science (EECS) offers courses like "Robotics: Science and Systems," where students design complete robotic solutions, blending software, hardware, and security. These programs produce graduates who excel in diverse roles, from developing secure autonomous systems to founding innovative startups.

Similarly, Stanford’s AI and Robotics track combines software development with mechanical engineering and cybersecurity, preparing students for complex challenges like secure drone navigation.

By adopting such models, educational institutions can better prepare students for a rapidly evolving industry, ensuring they are equipped to navigate and contribute to a technology-driven world.

Economic Challenges and Opportunities for Software Engineers

Today’s software engineers face a complex landscape of economic and societal pressures that are fundamentally reshaping their roles. Much of the work has shifted from pure invention to integration, often centering on stitching together proprietary clouds and third-party APIs.

This moves engineering effort toward upkeep – resolving version conflicts, debugging vendor libraries, and managing deployment pipelines – rather than creating foundational technology. This dynamic not only suppresses an engineer's individual earning potential, as disproportionate profits flow to leadership and investors, but also leaves businesses vulnerable to vendor lock-in and supply-chain shocks.

Dual Nature of AI

Compounding this challenge is the dual nature of modern artificial intelligence. While AI tools promise to accelerate code generation, their practical application reveals significant limitations and challenges. Real-world studies, such as the METR study, show that developers often overestimate AI's productivity benefits and can face slowdowns of nearly 20% due to the time spent fixing flawed or inefficient code.

This highlights that human oversight remains indispensable, especially when AI outputs must interface with custom hardware or meet strict safety standards.

The opportunity lies with engineers who understand the full system – electronics, mechanics, and secure architecture – and can effectively validate and harden AI-driven solutions.

Societal Challenges

Simultaneously, society is placing new and urgent demands on the engineering profession. An aging global population and declining birth rates are tightening the economic noose, with fewer workers supporting more retirees. This demographic headwind necessitates greater automation in manufacturing, food production, and healthcare.

The engineers who can deliver these solutions – by designing robotic arms for harvesting, smart greenhouses for urban farming, or humanoid helpers for elder care – will be at the forefront of tackling this challenge and opening new economic frontiers.

Beyond this, in a world flooded by misinformation and clickbait, engineers have an ethical duty to build systems that prioritize truth and transparency, embedding features like content-verification protocols and secure data handling to foster a trustworthy digital environment.

Changing Demands

These evolving demands expose a critical disconnect in traditional education. Costly four-year degrees too often leave graduates with narrow skill sets and surprisingly high unemployment rates (6-7.5%) that rival non-technical fields. This mismatch arises from curricula that prioritize isolated foundational skills or include unrelated electives over the practical, interdisciplinary training modern industry requires.

The path forward is through a more streamlined and relevant education that acts as a catalyst for resilience. By replacing less applicable courses with accelerated, hands-on projects, institutions can transform learners from passive code-integrators into formidable innovators.

Globally, leading institutions are already recognizing this need. In Nordic countries like Sweden and Finland, programs that integrate sustainability, ethics, and interdisciplinary skills are producing graduates who excel at innovation.

By adopting similar approaches – offering real-world modules in robotics prototyping, embedded security, and end-to-end system integration – we can empower engineers to meet today's complex demands and build the resilient, automated, and trustworthy systems our world urgently needs.

The Role of Institutions in Cultivating End‑to‑End Engineers

As technology shifts ever faster – reintegrating software with custom hardware, AI-driven automation, and secure connected systems – traditional universities risk obsolescence unless they reinvent themselves. Beyond breaking down academic silos, forward‑looking institutions will need to embrace four key strategies:

1. Embrace Agility Through Continuous Curriculum Evolution

Modular, Stackable Credentials
Universities should offer micro‑certificates in robotics prototyping, embedded security, or systems integration alongside full degrees. Students and professionals can assemble just the modules they need, when they need them – mirroring the on‑demand model of platforms like Coursera or Google’s own AI toolkits.

Real‑Time Industry Feedback Loops
They should also have rolling curriculum reviews with employer advisory boards. If a new sensor technology or cloud‑native inference engine emerges, courses can pivot within months, not years, ensuring graduates never learn outdated tools.

2. Partner with EdTech Leaders – Don’t Compete Alone

Leverage Existing Toolchains
Rather than ignoring Google’s free AI labs or Microsoft’s cloud credits, universities can integrate them directly into their coursework. Assignments could require deploying a hardware‑accelerated model on Google Coral or securing an Azure‑hosted IoT network.

Co‑Create Open Educational Resources
Institutions could also collaborate on open‑source textbooks, interactive labs, and tutorial videos – both to amplify institutional reach and to demonstrate that the university is part of, not apart from, today’s creator economy.

3. Prioritize User‑Centric Design in Education

Student and Employer Needs First
Schools should also treat their “customers” (students and hiring companies) as co‑designers. Conduct regular surveys and job‑task analyses: What exact blend of Linux kernel debugging, CAD design, and cryptographic key management does the next‑gen engineer need? Then build courses to match.

Flexible Delivery Modalities
They could also combine in‑person maker‑space workshops with online simulators (for example, Gazebo robotics, virtual FPGA labs) so that learners worldwide can participate – reducing geographic and economic barriers.

4. Cultivate an Ecosystem of Lifelong Learning

Alumni‑for‑Credit Programs
Universities could offer discounted, advanced modules for graduates to return and upskill as hardware standards or threat landscapes evolve. This continuous‑learning pathway turns one‑off degrees into multi‑decade partnerships.

Innovation Incubators and Industry Challenges
They could also host hackathons, sponsored capstone projects, and startup incubators right on campus. When students design and pitch end‑to‑end solutions for real companies – say, a secure medical‑robotics prototype – they graduate not just with a diploma, but with market‑tested experience and potential investors.

5. Staying Relevant – and Un-gatekeeping

With Google, Apple, and a legion of online platforms freely distributing cutting‑edge AI, robotics toolkits, and interactive tutorials, any institution that clings to century‑old lecture halls and fixed curricula looks increasingly like a barrier, not a gateway. To avoid that fate:

Shift from “Seat Time” to “Skill Proof”: Replace rigid credit hours with outcomes‑based assessments – portfolios, live demos, and secure system audits prove mastery far better than final exams.

Align incentives around impact, not enrollment: Reward faculty for evolving courses, publishing open resources, and mentoring student startups rather than gatekeeping admissions or ballooning class sizes.

By viewing themselves not as ivory‑tower knowledge guardians but as agile partners in an ever‑changing tech ecosystem, educational institutions can remain indispensable. They’ll graduate engineers who wield software and hardware with equal fluency, who adapt on the fly, and who drive innovation – and who never fear being “left behind” by the next big Google toolkit.

Proposed Reforms: Designing End-to-End Programs

Curriculum Transformation

To implement end-to-end engineering education, institutions should redesign curricula to prioritize interdisciplinary skills across a structured timeline like the following:

1. Year 1: Core Foundations – Focus on mathematics (calculus, linear algebra, probability) and programming (Python, C++, JavaScript), introducing systems thinking, basic robotics concepts, and an overview of cybersecurity principles. This foundational year ensures students build a strong technical base while gaining exposure to interdisciplinary applications.

2. Year 2: Software and Hardware Integration – Combine software development with mechanical engineering, emphasizing hands-on projects like robot prototyping. Courses might include designing simple robotic systems, such as a sensor-based navigation device, to connect digital and physical systems and introduce students to hardware constraints.

3. Year 3: Cybersecurity and Ethics – Teach cybersecurity principles, such as encryption and secure system design, alongside AI ethics to promote responsible technology development. Projects could involve securing IoT devices or analyzing AI-generated code for vulnerabilities, preparing students for real-world challenges.

4. Year 4: Capstone Projects – Require students to design and deploy real-world systems, such as secure IoT devices, autonomous robots, or energy-efficient automation systems, integrating all learned disciplines. These projects should involve collaboration with industry partners or research labs to ensure practical relevance.

This structure prioritizes practical, relevant skills, replacing less applicable courses with interdisciplinary modules that align with industry needs.

Faculty and Resources

Recruiting faculty with expertise in robotics, mechanical engineering, and cybersecurity is essential for delivering a robust curriculum. Institutions can support collaboration through training programs, workshops, and incentives like joint research grants. For example, faculty from computer science and mechanical engineering could co-teach courses on robotic system design, fostering an interdisciplinary approach.

Investments in infrastructure, such as robotics labs, 3D printing facilities, and cybersecurity simulation environments, are necessary but can be costly. Institutions can implement phased rollouts, starting with virtual simulations or open-source tools to reduce initial expenses.

Grants from organizations like the National Science Foundation (NSF) or partnerships with industry can offset costs, ensuring long-term sustainability. For instance, virtual robotics platforms like Gazebo allow students to simulate robot designs before building physical prototypes, making training more accessible.

Industry Collaboration

Partnerships with industry provide hands-on experience, ensuring students gain practical skills aligned with market needs. These collaborations should prioritize ethical practices, focusing on projects that address societal challenges, such as sustainable technology, secure systems, or healthcare robotics.

For example, joint labs with companies developing energy-efficient automation systems can enhance learning while fostering responsible development. Institutions must ensure partnerships emphasize student development and societal benefit, avoiding scenarios where corporate priorities overshadow educational goals.

Accessible and Flexible Pathways

To make end-to-end education accessible, institutions can offer accelerated programs, such as three-year degrees or modular bootcamps, incorporating AI tools to enhance efficiency.

For example, once they’ve learned key programming concepts, students could use AI-assisted coding platforms to prototype systems, learning to validate outputs for accuracy and security. Online platforms can broaden access, enabling diverse populations to benefit from comprehensive training. Partnerships with community colleges and vocational programs can create pathways for underrepresented groups, fostering an inclusive engineering workforce.

Continuous Curriculum Evolution

To remain relevant, institutions must continuously evolve their curricula to reflect emerging technologies and industry trends. This includes incorporating advancements in AI, such as generative models or reinforcement learning, and addressing new cybersecurity threats, like quantum computing risks. Regular feedback from alumni, industry partners, and students can ensure curricula stay aligned with real-world needs, preparing graduates for long-term success.

Benefits for Graduates and Society

Enhancing Graduate Outcomes

End-to-end education prepares graduates for a competitive market, reducing unemployment risks and enabling higher earnings. With skills in AI oversight, robotics, and hardware design, graduates can pursue roles in high-demand fields like healthcare robotics, secure IoT systems, or autonomous vehicle development, commanding 10-20% higher salaries due to their interdisciplinary expertise.

For example, engineers trained in robotics and cybersecurity can design secure medical robots, addressing the growing demand for healthcare automation.

By launching startups or freelancing, end-to-end engineers can innovate independently, bypassing traditional corporate structures and sharing more directly in the value they create.

Societal Contributions

Society benefits significantly from resilient, secure systems designed by end-to-end engineers. Secure robots and IoT devices protect critical infrastructure, such as manufacturing plants, hospitals, or transportation networks, from cyber threats.

For example, a secure robotic system in a hospital could ensure reliable operation of surgical robots, improving patient outcomes. Training in AI ethics ensures graduates prioritize societal good, mitigating risks like misinformation by designing platforms with robust content verification.

Accessible, accelerated programs promote equity, fostering diverse talent pools and countering job polarization, where AI enhances 25% of roles but automates others. By making education more inclusive, institutions can reduce disparities, ensuring underrepresented groups have access to high-demand careers in engineering.

Sustainability and Global Impact

Sustainability is a key benefit of end-to-end education. Engineers trained in holistic design can create energy-efficient systems, such as optimized robots for logistics or manufacturing, aligning with global environmental goals.

For instance, a robotic system designed to minimize energy consumption in a warehouse could reduce carbon emissions, contributing to sustainability efforts. Institutions adopting this model produce leaders who drive innovation and inclusive growth, addressing global challenges like climate change and digital equity.

Ethical Technology Development

End-to-end education fosters ethical awareness, equipping graduates to combat societal challenges like misinformation and system vulnerabilities. By integrating AI ethics and cybersecurity, graduates can design technologies that prioritize public good, ensuring platforms and systems are trustworthy and resilient. This approach aligns with the growing demand for ethical technology, as emphasized by many in the field who believe in the importance of critical thinking and responsibility in engineering.

Overcoming Challenges in Implementation

Faculty Engagement and Training

Transitioning to end-to-end programs may face resistance from faculty accustomed to traditional, siloed teaching. Institutions can address this through training workshops, collaborative research opportunities, and incentives like joint research grants.

For example, as mentioned above, faculty from computer science and mechanical engineering could co-develop courses on robotic system design, fostering interdisciplinary collaboration. Hiring experts in robotics, cybersecurity, and mechanical engineering ensures a capable teaching staff equipped to deliver comprehensive curricula.

Infrastructure Investment

The cost of infrastructure, such as robotics labs, 3D printing facilities, and cybersecurity simulation environments, poses a significant hurdle. Institutions can implement phased rollouts, starting with virtual simulations using tools like ROS (Robot Operating System) or Gazebo, which allow students to prototype systems without physical hardware. Grants from organizations like the NSF or partnerships with industry can offset costs, while open-source tools enhance accessibility, ensuring equitable access to training.

Curriculum and Accreditation

Redesigning curricula to meet accreditation standards, such as those set by ABET, requires a modular approach that integrates interdisciplinary skills while maintaining compliance. Institutions can pilot programs to test reforms, gradually incorporating modules like robotics or cybersecurity into existing curricula.

For example, a pilot program might introduce a robotics module in year two, allowing institutions to assess outcomes before full implementation. Regular reviews ensure curricula remain aligned with industry needs and accreditation requirements.

Building Stakeholder Support

Securing stakeholder support requires demonstrating the benefits of end-to-end education, including lower unemployment rates (potentially dropping below 3% with holistic training), higher graduate earnings (10-20% above traditional programs), and societal impact through secure, sustainable systems.

Engaging alumni, industry partners, and students in curriculum design builds trust and ensures relevance. For instance, advisory boards with industry representatives can provide insights into emerging trends, aligning programs with market demands.

Promoting Equity and Access

To ensure equitable access, institutions should leverage online platforms and modular degrees, reducing costs and reaching diverse populations. Partnerships with community colleges and vocational programs can create pathways for underrepresented groups, fostering an inclusive engineering workforce.

For example, online courses in robotics or cybersecurity can provide access to students in remote or underserved areas, while modular bootcamps allow working professionals to upskill efficiently.

Addressing Scalability

Scaling end-to-end programs requires strategic planning to balance quality and accessibility. Institutions can start with small cohorts, refining curricula based on feedback before expanding. Collaborations with other universities or online education platforms can share resources, reducing costs and increasing reach. For instance, a consortium of universities could develop shared virtual labs, enabling cost-effective training across institutions.

Conclusion: A Path Forward for Engineering Education

The case for end-to-end engineering education is compelling in a world shaped by AI, interconnected systems, and evolving societal needs. Traditional software engineering programs, with their focus on intermediary roles, must evolve to prepare graduates for the complexities of modern industries.

By integrating software development with robotics, mechanical engineering, and cybersecurity, institutions can produce versatile, innovative engineers who lead in a technology-driven world.

Reforms require bold action: transforming curricula to prioritize interdisciplinary skills, investing in faculty and infrastructure, fostering ethical industry partnerships, and promoting accessible pathways.

Case studies from MIT, Stanford, Vanderbilt, and global institutions like those in Nordic countries demonstrate the transformative potential of this approach, with graduates excelling in diverse roles, founding startups, and building resilient systems. Emerging programs at institutions like ETH Zurich and the University of Toronto further highlight the global applicability of end-to-end education.

Challenges like faculty resistance, infrastructure costs, and accreditation hurdles can be addressed through strategic planning, including phased rollouts, grants, and stakeholder engagement. Online platforms and partnerships with community colleges ensure equity, fostering a diverse talent pool that drives inclusive growth.

End-to-end education is not just an opportunity – it’s a necessity for equipping engineers to navigate a complex, technology-driven world. By embracing this model, institutions can empower the next generation to build innovative, secure, and sustainable systems that benefit society, ensuring a resilient and equitable future for all.

Further Resources:

Ready to become an End-to-End Engineer – mastering software, hardware, AI deployment, robotics, and cybersecurity to build complete systems from the ground up?

Don't just integrate – innovate and lead. You can enroll in LUNARTECH AI for Executives, tailored for leaders who want to strategize, fund, and deploy cutting-edge AI solutions without falling behind in the fast-evolving tech landscape.

LunarTech AI for Executives

For leaders and frontline professionals who feel the pressure to “get AI” but don’t speak code, this 1- to 3-day program delivers exactly what you need: no fluff, no jargon. In clear language, we unpack how generative AI, large-language models, and regulatory frameworks such as the EU AI Act are reshaping compliance, risk, and client service.

Next, we roll up our sleeves. You’ll practice with ChatGPT, Phoenix, Gemini, and other curated tools to summarize 200-page reports in minutes, flag hidden risks, and automate repetitive workflows. Expect live demos, breakout labs, and case studies drawn straight from banking, asset management, and insurance.

By the final session you’ll have a road-ready playbook for piloting AI safely – from data-governance checklists to ROI metrics your CFO will love. Graduates leave with a certificate, a toolkit of prompts, and the confidence to champion AI initiatives inside their own departments.

• Format: Online or on-site, 1–3 days

• Cost: $997 per participant

Apply Here: https://www.lunartech.ai/programs/ai-for-executives

Other Resources

• Lens | LUNARTECH - https://lens.lunartech.ai/

• YouTube | LUNARTECH - https://www.youtube.com/@lunartech_ai

• Linkedin | LUNARTECH - https://www.linkedin.com/company/lunartechai/

• Substack | LUNARTECH - https://lunartech.substack.com/

This handbook takes you on a journey from fundamental logical principles to their practical applications in software development, scientific reasoning, and critical thinking.

Whether you're a high school student learning to think more clearly, a professional debugging complex systems, or simply someone curious about how sound reasoning works, this handbook provides tools for sharper, more reliable thinking.

What We’ll Cover:

Part I: Foundational Theory

We start with the bedrock of formal logic – understanding implications, truth tables, and the core rules of reasoning.

You'll learn the scaffolding for everything that follows:

• How "if-then" statements actually work (spoiler: it's not always intuitive!)

• The power of truth tables to map all possible scenarios

• Why some arguments are valid while others are logical fallacies

• The elegant relationship between Modus Ponens, Modus Tollens, and Contrapositives

Part II: Practical Applications

Here's where logic comes alive in tangible ways:

In Software Development:

• How debugging mirrors logical reasoning, and why your tests might be lying to you

• The logic behind Test-Driven Development and Mutation Testing

In Scientific Thinking:

• Karl Popper's falsification principle and why it matters beyond academia

• How Hypothesis Testing is just statistics meets Modus Tollens

In Everyday Reasoning:

• Spotting logical fallacies in arguments, media, and your thinking

• The art of considering multiple causal paths instead of jumping to conclusions

Part III: Philosophical Depths

The final section confronts the beautiful complexity of applying pure logic to an impure world:

• Why perfect "if-and-only-if" relationships are the goal but rarely achievable

• How modern software systems hide their complexity

• The butterfly effect of bugs and why root cause analysis is often harder than it seems

• Formal verification tools: from Prolog to Coq to TLA+

What You'll Gain

For Students:

• Critical thinking superpowers: Learn to spot flawed reasoning in arguments, social media, and news

• Academic advantage: These concepts appear in debates, philosophy, computer science, mathematics, and statistics

For Software Engineers:

• Debugging mastery: Modus Tollens for debugging: "If the output is wrong, what could cause it?"

• Testing philosophy: Move beyond "make the tests pass" to "prove the code is correct"

• Problem analysis: Avoid jumping to solutions before understanding the real problem

• System design: Think more rigorously about failure modes and edge cases, evaluate cause-and-effect relationships in complex systems

• Communication and career growth: Present arguments more clearly and persuasively, gain logical thinking skills that separate senior engineers from juniors

For Scientists:

• Experimental design: Strengthen your understanding of hypothesis testing and falsifiability

• Peer review: Better evaluate the logical soundness of research claims

• Grant writing: Structure arguments more persuasively using solid logical foundations

Pre-requisites

I’ll introduce code samples starting in the second half of the article, so knowing a programming language would be helpful. The concepts in this article are programming language-agnostic, but I’ve used Python throughout for readability.

No prior formal logic or philosophy background is strictly necessary, but the following will let you reap the most benefits from this article:

• Experience in testing and debugging during software development.

• Know what REPL (Read-Evaluate-Print-Loop) is if you want to try the Proof Assistants.

• Knowledge of logical operators (NOT, AND, OR), and the fact that they take 1 or 2 boolean values as input and return a single boolean value as output.

• Basic Algebraic Thinking: representing statements as variables (P, Q), the concept of NOT (¬) as an inversion of statements, and the concept that different input combinations can reach the same output.

• Exposure to deductive reasoning, where inferences are made based on some facts, and fallacies, which are some ways arguments can be flawed.

• Willingness to engage in conceptual back-and-forth between concrete English examples and abstract logical symbols.

• Holding possibly conflicting ideas between the ideal logic world and the impure real world.

• Openness to challenging intuition and following logical rules before applying your real-world experience.

Table of Contents

1. An Introduction to Logic

2. Truth Tables: Mapping All Possibilities

3. Contrapositives, Modus Ponens, Modus Tollens

4. The Origin of P⟹Q: Science and Reality

5. Revisiting Argument Forms: Valid Inferences and Common Fallacies

6. Denying the Antecedent: A Database Example

7. Assigning Real-World Meanings to Logic

8. Applying Logic to Software Testing

9. A Closer Look at Testing

10. Revisiting the Four Statements for Coding

11. The Missing Ingredient - If and Only If

12. Mutation Testing: Testing the Tests

13. Toward If-and-Only-If Confidence

14. Real-World Challenges

15. Glimmers of Hope: Tools and Practices for Clarity

16. The Power of Falsification in Testing

17. Proof Assistants

18. Food for Thought

19. Q.E.D.: The Enduring Power of Logic in an Uncertain World

20. Resources

21. Glossary

An Introduction to Logic

Imagine that the following statement is True:

If you are a coding instructor, then you have a job.

Now, do these make sense?

1. You have no job, so you are not a coding instructor

2. You have a job, so you are a coding instructor

3. You are not a coding instructor, so you have no job

Interpretations

Based on logic:

• Statement 1 is correct.

• Statement 2 is wrong because you may have other jobs without being a coding instructor.

• Statement 3 is wrong because you may or may not have a job, and as before, you may have other jobs without being a coding instructor.

Growing complexity

These statements grow increasingly complex due to:

• Changing from 2 valid statements to 2 invalid conclusions

• Moving from a clear job status (1, 2) to uncertainty about job existence or type (3).

Let’s get familiar with some notation before seeing how Truth tables help manage this complexity.

Notations

Truth Tables: Mapping All Possibilities

What is a Truth Table?

A truth table is a powerful tool in logic that helps us determine the overall truth or falsity of a compound logical statement. It does this by systematically listing all possible combinations of truth values (True or False) for its individual component propositions.

For every way the "inputs" (our propositions like P and Q) can be true or false, the truth table shows you the precise "output" (the truth value of the entire logical statement, such as P⟹Q).

Why are Truth Tables Helpful?

Truth tables offer critical benefits for clear thinking:

• Clarity and precision: They eliminate ambiguity by explicitly showing the outcome for every single scenario.

• Systematic analysis: They ensure no possible combination is missed, which is vital for sound reasoning.

• Foundation for understanding: They define how logical rules work, forming the bedrock for analyzing more complex arguments in any domain.

How to Read Our First Truth Table:

Let's examine the truth table for the implication P⟹Q ("If P then Q").

Each row represents a unique scenario, combining the truth values of P and Q to show the resulting truth value of P⟹Q.

Let's break down each row:

• P and Q Columns: These show the input truth values (True or False) for our two propositions. Since each can be one of two values, we have 2×2 = 4 unique combinations, filling all four rows.

• P ⟹ Q Column: This is the output truth value of the "If P then Q" statement for each combination of inputs P and Q.

  • Row 1: P is True, Q is True.

    • If P is true (you are a coding instructor) and Q is also true (you have a job), then the implication P⟹Q is True. (The "If...then..." statement holds).

    • This row is key for Modus Ponens.

  • Row 2: P is True, Q is False

    • If P is true (you are a coding instructor) but Q is false (you have a job), then the implication P⟹Q is False. This is the only scenario that disproves an "if-then" statement.

    • This row is key for Falsifiability.

  • Row 3: P is False, Q is True.

    • If P is False (you are not a coding instructor) but Q is True (you have a job), then the implication P⟹Q is still considered True. This can seem counter-intuitive.

    • The reason is that the implication statement only makes a claim about what happens when P is true. If P is false, the implication's claim isn't tested, so it is considered vacuously true.

  • Row 4: P is False, Q is False.

    • If P is False (you are not a coding instructor) and Q is False (you have no job), then the implication P⟹Q is also considered True.

    • Similar to Row 3, since the initial condition (P) was false, the implication's truth value remains True, as it hasn't been disproven.

    • This row is key for Modus Tollens.

The "Used In" column serves as a preview of the specific logical arguments or concepts that rely on each row's behavior, which we will explore in detail later.

Understanding the Implication (P⟹Q) Deeper

Most programmers are familiar with truth tables from logical operators like AND (∧), OR (∨), and NOT (¬), where they define the output based on combinations of inputs.

The implication (P⟹Q) works similarly, its output is defined by the rules of propositional logic, not by any real-world causal relationship or your “common sense”. For any given pair of inputs for P and Q, the result of P⟹Q is fixed.

If this feels counter-intuitive, consider that mathematical logic, like any formal system, is built upon agreed-upon axioms. These basic accepted truths allow us to construct complex systems of ideas. If later found ineffective or contradictory, these axioms can be redefined, or a new system can be developed.

In formal logic, this implication is also defined as being logically equivalent to "NOT P OR Q" (¬P∨Q).

This is the fundamental logical rule that dictates why, if P is False, P⟹Q is always True, regardless of Q's truth value. You can also understand this using the NOT P OR Q form.

• If P is False, that means NOT P is True.

• Using the rules of Logical operation:

  • True (Not P) OR True (Q) is True (NOT P OR Q)

  • True (Not P) OR False (Q) is True (NOT P OR Q)

  • NOT P OR Q is True regardless of what Q is.

The above explains rows 3 and 4 of the truth table from the NOT P OR Q form. As an exercise, you can apply the inputs (P, Q) from the first two rows of the truth table to NOT P OR Q to arrive at the same results defined in the P⟹Q column.

This formal definition allows us to use implication to reason in powerful ways, not just in the "forward" direction (P⟹Q, leading to Modus Ponens), but also in a crucial "backward" direction.

This backward form (Contrapositive) involves swapping and negating the propositions (¬Q⟹¬P).

For example, if "If you are a coding instructor, then you have a job" is true, then it must also be true that "If you have no job (¬Q), then you are not a coding instructor (¬P). ".

This "backward" way of reasoning, which underpins Modus Tollens, is a powerful tool for inferring conclusions from observed outcomes.

We'll explore the Contrapositive and two argument forms (Modus Ponens, Modus Tollens) in detail next.

Contrapositives, Modus Ponens, Modus Tollens

We've explored the fundamental implication (P⟹Q) and how truth tables reveal its behavior.

Now, we explore reasoning tools that build upon this foundation: Modus Ponens, Modus Tollens, and the concept of Contrapositives. These are bedrock principles of valid argument and efficient logical thought.

What is Logical Equivalence?

Before we dive into these specific concepts, let's clarify what logical equivalence means. Two statements are logically equivalent if they always have the same truth value under all possible circumstances. In simpler terms, if one statement is true, the other is always true. If one is false, the other is always false. They are, in essence, different ways of saying the same logical thing.

Understanding logical equivalence is incredibly useful. It:

• Simplifies logic: It allows us to substitute one statement for another without changing the truth of an argument, which simplifies complex proofs and reasoning.

• Reduces complexity: In fields like circuit design, it can lead to fewer physical gates.

• Maintains software correctness: In programming, it helps maintain code's correctness during refactoring and debugging, especially when simplifying conditional statements, by ensuring the transformed code still behaves identically to the original under all conditions.

The Contrapositive: An Equivalent Implication

One of the most important logical equivalences involves the Contrapositive of an implication. The contrapositive of an "If P then Q" (P⟹Q) statement is "If not Q, then not P" (¬Q⟹¬P).

You might intuitively question how "If P then Q" could be logically the same as "If not Q then not P." Let's demonstrate this using a truth table.

We'll start with our familiar P and Q columns and the P⟹Q implication. Then, we'll add columns for ¬P (Not P) and ¬Q (Not Q), and finally, the implication for the contrapositive, ¬Q⟹¬P.

Let's look at how the truth table explicitly shows this equivalence:

Explanation of the table

1. P, Q, P ⟹ Q (Columns 1-3): These are our standard propositions and the implication we've already defined.

2. ¬P (Column 4): This column simply shows the negation (opposite truth value) of the P column. If P is True, ¬P is False, and vice-versa.

3. ¬Q (Column 5): Similarly, this column shows the negation of the Q column.

4. ¬Q ⟹ ¬P (Column 6): This is the contrapositive. We apply the same rules for implication that we learned earlier, but now using ¬Q as our "if" part and ¬P as our "then" part. For example, in Row 2, ¬Q is True and ¬P is False. According to the implication rule (True ⟹ False yields False), the result for ¬Q⟹¬P is False.

5. The Proof of Equivalence: Now, compare Column 3 (P⟹Q) with Column 6 (¬Q⟹¬P). You'll notice that for every single row, their truth values are identical! When P⟹Q is True, ¬Q⟹¬P is also True. When P⟹Q is False, ¬Q⟹¬P is also False. This perfectly illustrates why they are logically equivalent.

So, "If you are a coding instructor, then you have a job" (P⟹Q) is logically the same as saying "If you have no job, then you are not a coding instructor" (¬Q⟹¬P). They convey the same information about the relationship between being a coding instructor and having a job.

How Modus Ponens and Modus Tollens Relate to Implication

Having defined logical equivalence and the contrapositive, we can now precisely understand two of the most fundamental and valid forms of deductive argument: Modus Ponens and Modus Tollens. Both of these argument forms rely on a core premise that an implication (P⟹Q) is true, and then use additional information to draw a valid conclusion.

1. Modus Ponens (Affirming the Antecedent): This is often considered the most intuitive and direct form of logical inference. It works in the "forward" direction of the implication.

   • Premise 1: We are given that the implication is true: If P, then Q (P⟹Q).

   • Premise 2: We are also given that the "if" part, the antecedent, is true: P is true.

   • Conclusion: Therefore, we can validly infer that the "then" part, the consequent, must also be true: Q is true.

Example:

• Premise 1: If it is raining (P), then the ground is wet (Q).

• Premise 2: It is raining (P).

• Conclusion: Therefore, the ground is wet (Q).

This directly corresponds to Row 1 (True, True) of our truth table for P⟹Q.

2. Modus Tollens (Denying the Consequent): This argument form works in the "backward" direction and relies directly on the logical equivalence of an implication and its contrapositive.

   • Premise 1: We are given that the implication is true: If P, then Q (P⟹Q).

   • Premise 2: We are also given that the "then" part, the consequent, is false: Not Q (¬Q).

   • Conclusion: Therefore, we can validly infer that the "if" part, the antecedent, must also be false: Not P (¬P).

Example:

• Premise 1: If it is raining (P), then the ground is wet (Q).

• Premise 2: The ground is not wet (¬Q).

• Conclusion: Therefore, it is not raining (¬P).

Modus Tollens is valid because if P⟹Q is true, its contrapositive (¬Q⟹¬P) must also be true. Applying Modus Ponens to this contrapositive (with ¬Q as our second premise) directly leads to the conclusion ¬P. This corresponds to Row 4 (False, False) of our original truth table for P⟹Q, where P and Q are both false but the implication is still true.

These two argument forms are central to rigorous deductive reasoning, allowing us to draw certain conclusions based on the truth of implications and related facts.

The Origin of P⟹Q: Science and Reality

In science, hypotheses often take the form "If P, then Q" where P is a cause and Q is its predicted effect –for example, "If a drug is given (P), then symptoms improve (Q)."

Ideally, P is controllable, as in experimental studies, but even in observational studies, P must be clearly defined and measurable.

Each experiment yields one observation, reflecting one of four possible truth-value combinations of P and Q.

The Falsifying Case in Science and Logic

Each experiment produces a single observation – one of the four possible combinations of P and Q.

• If P=True, Q=False is observed (row 2 of the truth table), the hypothesis is falsified

• In all other cases, the hypothesis is not falsified (yet)

Thus:

• If all observations fall in the 3 truth-preserving rows, the hypothesis remains viable.

• If at least one experiment yields P=True, Q=False, we either:

  • Conclude falsification, or

  • Re-examine the experiment and attempt replication before accepting falsification.

The Power of the Falsifying Case

In the Logical World

The falsifying case is not useful for inference with Modus Ponens or Modus Tollens because these two argument forms require starting with P⟹Q = True. I’ll explain both arguments in detail later.

But the falsifying case is useful for showing counterexamples to disprove the implication, or proof by contradiction.

In the Real Scientific world

The falsifying case embodies Falsifiability – a crucial concept in Science.

In so far as a scientific statement speaks about reality, it must be falsifiable: and in so far as it is not falsifiable, it does not speak about reality.

— Karl R. Popper, The Logic of Scientifc Discovery

Scientific theories come about through hypotheses that are continually tested and survive attempts at falsification.

Popperian Falsification and Hypothesis Testing

These two approaches, one philosophical and one statistical, are distinct but complementary in the scientific method.

• Popperian Falsification starts with a scientific hypothesis (for example, "P has an effect on Q"). Its core aim is to actively seek evidence that would disprove this hypothesis. If such disproving evidence is found, the hypothesis is falsified.

• Statistical Hypothesis Testing begins with a null hypothesis (H0​) (for example, "P has no effect on Q"). Its goal is to determine if the collected data provides sufficiently extreme evidence to reject this null hypothesis.

If the null hypothesis is rejected, it provides statistical support for the alternative hypothesis (that P does have an effect on Q). This statistically supported hypothesis then becomes a stronger candidate, continually subjected to further Popperian attempts at falsification through new experiments and observations.

The Nuance: Implication is Not Causality

P⟹Q does not inherently imply that P causes Q.

Consider these examples:

• "If the fire alarm is sounding, then there is smoke." The alarm doesn't cause the smoke.

• "If a colleague screams during code review, then the code is bad." Does the screaming cause the bad code, or merely reveal it? (Perhaps sometimes both! 😰)

Causality is a real-world concept crucial for making informed decisions, predicting outcomes, and inferring the underlying reasons for events.

It's often central to predictive modeling and supervised learning in data science, where the target variable is the effect and the predictors are proposed causes. A common pitfall here is data leakage, where predictors are inadvertently influenced by (or are themselves effects of) the target, violating the causal assumption.

Logic, however, doesn't model time, mechanisms, or interventions. It only cares about truth values and formal structure. Logic defines what is true based on premises, not what makes something true in a causal sense.

Revisiting Argument Forms: Valid Inferences and Common Fallacies

We've now established the rules of implication, understood logical equivalence, and learned about two powerful, valid argument forms: Modus Ponens and Modus Tollens. But when we try to reason using "if-then" statements, it's easy to fall into common logical traps.

In this section, we'll systematically revisit the four common ways we might try to draw conclusions from an implication P⟹Q (If you are a coding instructor, then you have a job) introduced at the start of the handbook.

Two are valid arguments (Modus Ponens and Modus Tollens), and two are common logical fallacies. Understanding the differences is crucial for sound reasoning.

First, let's quickly define the parts of an "if-then" condition:

• Antecedent: The "if" part of the condition (P).

• Consequent: The "then" part of the condition (Q).

Now, let's examine these four argument forms, using our knowledge of truth tables and the coding instructor example.

Affirming the Antecedent (Modus Ponens)

This is the first valid argument form we discussed. It's called "affirming the antecedent" because it asserts the truth of the "if" part (the antecedent, P) to conclude the "then" part (the consequent, Q).

• Argument Form:

  1. If P, then Q (P⟹Q)

  2. P is true.

  3. Therefore, Q is true.

• Examples:

  • You are a coding instructor (P), so you have a job (Q).

  • You provided invalid input data (P), so the code will show an error (Q).

• Interpretation: This argument directly aligns with Row 1 (P=True, Q=True) of our truth table, where the implication holds true. It's often the most intuitive form of logical deduction. In programming, it's natural to expect bad input to lead to error messages if the code is designed correctly.

Denying the Consequent (Modus Tollens)

This is the second valid argument form. It's called "denying the consequent" because it asserts the falsity of the "then" part (the consequent, ¬Q) to conclude the falsity of the "if" part (the antecedent, ¬P). As we learned, Modus Tollens derives its validity from the logical equivalence of P⟹Q and its contrapositive (¬Q⟹¬P).

• Argument Form:

  1. If P, then Q (P⟹Q)

  2. Not Q is true (¬Q).

  3. Therefore, Not P is true (¬P).

• Examples:

  • You have no job (¬Q), so you are not a coding instructor (¬P).

  • There are no error messages (¬Q), so the input data is valid (¬P)

• Interpretation: This argument corresponds to Row 4 (P=False, Q=False) of our truth table, where P⟹Q is true, and both P and Q are false. This form of reasoning is critical for skillful debugging, allowing you to infer reasonably true conclusions about the cause (P) from observations of the outcome (Q), assuming your program logic (P⟹Q) holds true.

Affirming the Consequent (Fallacy)

Now we move to the common pitfalls. This is an invalid argument form where we attempt to conclude that the antecedent (P) is true simply because the consequent (Q) is true. It's a fallacy because the truth of Q does not guarantee the truth of P, as Q could have been caused by something other than P.

• Argument Form (Invalid):

  1. If P, then Q (P⟹Q)

  2. Q is true.

  3. Therefore, P is true. (**Incorrect inference!**🚨)

• Examples:

  • You have a job (Q), so you are a coding instructor (P).

    • Incorrect: You could have many other jobs.

  • The code showed an error (Q), so you provided invalid data (P).

    • Incorrect: Other things besides invalid data can cause errors.

• Interpretation: This fallacy highlights the difference between a one-to-one and a one-to-many relationship. Looking at our truth table, when P⟹Q is True and Q is True, P could be True (Row 1) or False (Row 3). The argument mistakenly concludes that P must always be True. The uncertainty arises because observing Q as True doesn't uniquely point to P as the cause – there could be many other reasons or paths that lead to Q.

  • Think of walking down a forest path, unaware that another trail has merged into yours from behind you. When retracing your steps in reverse, you encounter a split (Q) at that merge and feel disoriented, unsure which path leads back to your start point (P). Just as multiple paths can converge on the same point, multiple causes can produce the same outcome.

Denying the Antecedent (Fallacy)

This is another invalid argument form. Here, we attempt to conclude that the consequent (Q) is false simply because the antecedent (P) is false. It's a fallacy because P being false does not guarantee that Q will also be false. Q could still be true for other reasons, or the implication might not cover all scenarios where Q occurs.

• Argument Form (Invalid):

  1. If P, then Q (P⟹Q)

  2. Not P is true (¬P).

  3. Therefore, Not Q is true (¬Q). (**Incorrect inference!**🚨)

• Examples:

  • You are not a coding instructor (¬P), so you have no job (¬Q).

    • Incorrect: You could have a different job.

  • You provided valid data (¬P), so you have no error (¬Q).

    • Incorrect: Valid data doesn't guarantee no error. Other factors like network issues, memory leaks, or non-idempotent operations can still cause errors.

• Interpretation: Similar to Affirming the Consequent, this fallacy stems from incorrectly assuming a unique relationship. From our truth table, when P⟹Q is True and P is False, Q could be True (Row 3) or False (Row 4). The argument mistakenly concludes Q must always be False.

Both of these fallacies (Affirming the Consequent and Denying the Antecedent) creep into our thinking when we prematurely assume a single cause for an effect. In complex real-world systems, many factors can lead to an outcome, and narrowing your thinking too soon can lead to missed bugs or incorrect conclusions.

Fallacies and Implication: A Prerequisite

Both the fallacy of affirming the consequent and denying the antecedent assume the underlying implication (P⟹Q) is true.

If this implication is false from the start, there's no logical argument to be made, and thus, no fallacy to speak of.

Exercise: Identifying an Argument Form

Which of the 4 forms of argument is this?

• Penguins can’t fly. I can’t fly. Therefore, I’m a penguin.

Hint: Rephrase the first statement into an if-then form.

Denying the Antecedent: A Database Example

We just saw that Denying the Antecedent is a logical fallacy, meaning that even if the initial implication (P⟹Q) is true, concluding ¬Q from ¬P is not a valid inference. To make this abstract concept concrete, and to illustrate why this fallacy can be particularly dangerous in real-world systems like software, let's explore a practical example involving a database.

The implication: If the database is down (P), we’ll see a connection timeout error (Q).

Now, applying the fallacy of Denying the Antecedent, we might incorrectly conclude: If the database is not down (¬P), we will not see a connection timeout error (¬Q). ❌

But even if the database itself is perfectly operational and "not down," you might still encounter a connection timeout error. This could happen due to a variety of other, independent reasons, such as:

• Network problems

• Firewall rules

• The database is up but extremely slow

• The query engine is stuck

This specific example of multiple potential causes for a "timeout" highlights a broader, critical skill in software development: thorough case analysis.

This is precisely why technical assessments, especially in areas like algorithms and system design, frequently demand that you consider exhaustive possibilities. For instance, you are often asked to handle base and recursive cases in dynamic programming, or to ensure mutually exclusive and collectively exhaustive coverage when grouping multiple scenarios in problems like interval merging.

Such strong case analysis is vital for minimizing bugs and cultivating an open-minded approach to considering multiple causal paths, driven by experience, curiosity, and a dedication to craftsmanship.

But even perfect case analysis doesn't guarantee a correct implementation. Weak language mastery or mistaken assumptions can still lead to errors, making tests a crucial last line of defense.

Before jumping into applying logic to software testing, let’s practice our agility in conceptually switching between real-world concepts in English and symbols in logic.

Assigning Real-World Meanings to Logic

We must define what P, Q, and P⟹Q refer to when applying logical theory to real-world concepts.

How we define these variables affects our truth tables.

For example:

• If P means "valid input," then ¬P means "invalid input."

• If P means "invalid input," then ¬P means "valid input."

Imagine we define P = "Good input" and Q = "No Error."

• When testing the happy path, we are verifying that the implication P⟹Q (If input is good, then no error) holds true.

• When testing the unhappy path (mutation testing, more details later), we are verifying that ¬P⟹¬Q (If input is not good, then an error occurs) holds true.

In any test, a failure indicates that the tested implication is false. This warrants investigation into whether the issue lies with the specification's interpretation, the implementation, or even the test itself.

Applying Logic to Software Testing

Software development relies on constructing systems that behave predictably. Software testing is our primary tool for validating these behaviors. At its core, testing is a process deeply rooted in logical implications, where we propose a hypothesis about our code and then run an experiment (the test) to check its truth.

A test case is carefully designed to evaluate a specific piece of code. This involves:

1. Setting up Preconditions and Inputs: Before executing the code under test, we meticulously establish a specific environment and provide particular inputs. This includes:

   • Function/Method Arguments: The precise values passed into the code being tested.

   • System State: Setting up relevant data in a database, preparing the content of a file system, configuring an object's instance variables, or dictating the responses of external services (often through "mocks" or "stubs").

   • Environmental Factors: Controlling elements like the current time, specific network conditions, or user permissions relevant to the code's execution. This precise setup ensures that the code runs under defined conditions, allowing us to evaluate its behavior consistently.

Once the setup is complete, the code under test is executed, and its output or behavior is observed. This observation is then compared against an expected result.

To precisely analyze test outcomes, let's establish our specific logical mapping:

• P: The code under test is correct for the specific scenario defined by the test. This refers to the actual, objective state of the code's internal logic and implementation when presented with the test's preconditions and inputs. If P is True, the code is without defect for this case. If P is False, there is a bug or deviation.

• Q: The test passes. This means the actual output or behavior observed from the code precisely matches the expected outcome defined in our test case. If they do not match, the test fails.

• P⟹Q: If the code under test is correct for this specific scenario, then the test will pass. In pure propositional logic, the truth value of P⟹Q is indeed defined by the truth values of P and Q. But in the context of software testing, P⟹Q represents our hypothesis or desired specification for how the code should behave. We don't directly "know" P's truth value beforehand. Instead, the test's execution provides empirical data (the actual Q) that allows us to evaluate whether this hypothesis holds true in practice, and thereby infer the actual state of P.

Understanding this mapping is vital for interpreting test results. Let's examine the different outcomes of a test run, referencing the truth table for P⟹Q:

• Row 1: P is True (Code is correct), Q is True (Test passes)

  • Interpretation in Testing: Ideal State/Validation

    • This is the desired outcome and strengthens our confidence that the code adheres to its specification.

    • This scenario directly confirms the truth of our hypothesis (P⟹Q).

• Row 2: P is True (Code is correct), Q is False (Test fails)

  • Interpretation in Testing: Logical Contradiction / Falsification of Hypothesis

    • This row means our overall hypothesis P⟹Q is false for this specific instance.

    • This demands investigation: either our initial assumption that P was True (meaning the code was correct) is wrong (i.e., there's an actual bug, so P is actually False), or the test itself is flawed (its inputs/expectations are incorrect), or the specification is wrong.

    • This is where rethinking of the P⟹Q hypothesis itself happens.

• Row 3: P is False (Code is incorrect), Q is True (Test passes)

  • Interpretation in Testing: False Positive / Inadequate Test

    • This is a problematic scenario. It implies the test is not robust enough to detect the defect in the code, or the test's expectation is flawed.

    • While P⟹Q remains true vacuously, this outcome is misleading and means the test is not effectively verifying code correctness.

• Row 4: P is False (Code is incorrect), Q is False (Test fails)

  • Interpretation in Testing: Bug Found / Confirmation of Incorrectness

    • This is a beneficial outcome, as the test has successfully identified a defect.

    • When P is truly False, P⟹Q is vacuously true.

    • This row can represent either a known, intended 'P is False' state (e.g., TDD Red phase) or the actual state discovered via deduction (explained below in Scenario 1).

Note on this Contextualized Truth Table and Probabilistic Nature

This truth table differs from a purely abstract logical truth table by being explicitly contextualized for software testing.

• Specific Definitions: Unlike a generic P and Q, here they have precise meanings within the domain of code correctness and test outcomes.

• "Interpretation in Testing" Column: This is the key distinguishing feature. It translates the raw logical outcomes of (P, Q, and P⟹Q) into actionable insights and common debugging/development scenarios for software engineers. It explains what it means when a particular row is observed in the context of testing.

• Probabilistic Confidence: While formal logic operates in binary (True/False), real-world software testing often involves probabilistic confidence. A test doesn't provide absolute logical proof of correctness (for example, a passing test doesn't guarantee P is 100% True due to the possibility of undiscovered bugs or false positives). Instead, test results increase our confidence that the code is correct, or provide strong evidence that it is incorrect. Testing is fundamentally about reducing uncertainty and increasing the probability that our code functions as intended.

Let's now explore how these logical outcomes are interpreted in two common testing scenarios:

Scenario 1: Debugging an Unexpected Defect (Applying Modus Tollens)

This scenario occurs when a test that was previously passing, or a newly written test that we strongly trust as a precise and correct specification, unexpectedly fails. In this context, we assume the validity of the implication P⟹Q for this specific test case, treating it as an unbreakable rule for how correct code should behave.

1. Our Core Premise (Trusted Specification): We operate under the assumption that the implication "P⟹Q" ("If the code is correct for this scenario, then the test passes") is True for this specific test. Our confidence stems from the test's meticulous design, its history of passing, or its role in a well-established regression suite.

2. Test Execution and Observation: We run the test, which has its preconditions and inputs set.

   • If the Test Fails (Q is False): This is the key observation. Since we trust our premise that P⟹Q is True, and we observe ¬Q (the test fails), we are logically compelled to deduce that our initial belief about P (the code being correct for this scenario) must be false.

     • Application of Modus Tollens:

       • Premise 1: If the code is correct for this scenario (P), then the test passes (Q). (P⟹Q, assumed true as a trusted specification).

       • Premise 2: The test did not pass (¬Q).

       • Conclusion: Therefore, the code is not correct for this scenario (¬P).

     • Outcome: This inference directly points us to a defect in the code. The test's failure, given its trusted nature, reveals that the actual state of the code for this scenario is P is False. This effectively places the scenario in Row 4 (P False, Q False) of our truth table, confirming the presence of a bug that needs fixing. This is typical in regression testing, where a previously correct feature suddenly breaks.

Scenario 2: Validating/Refining the Specification (Falsifying P⟹Q or Confirming Known Incorrectness)

This scenario arises when a test fails, and our primary focus is not immediately on debugging the code as if it's a regression. Instead, it's on understanding why the P⟹Q relationship (our hypothesis for this specific behavior) isn't holding, or simply confirming an expected failure. This can involve questioning the test itself, the underlying requirements, or confirming a deliberately incorrect state of the code.

1. Our Hypothesis (Being Challenged or Confirmed): We are either actively evaluating the validity of the implication "P⟹Q" for a specific behavior, or we are running a test against code we know is incomplete or incorrect.

2. Test Execution and Observation: We run the test with its defined preconditions and inputs.

3. If the Test Fails (Q is False): The interpretation here depends on our prior knowledge or intent about the code's state (P):

   • Sub-scenario 2A: Falsifying P⟹Q and Rethinking Specification (Corresponds to Row 2: P True, Q False):

     • We observe Q is False (the test fails).

     • If we then examine the code and the requirements, and we conclude that the code should have been correct for this scenario (meaning, our expectation/belief was P is True), then the test result means the specific instance of our hypothesis "P⟹Q" is FALSE.

     • This direct falsification reveals a contradiction. We must then investigate:

       • Is our initial belief that P was True mistaken (that is, is there a genuine bug in the code that makes P actually False, moving this to a Row 4 scenario)?

       • Or, is the test itself incorrect (its inputs or expected output are wrong), meaning our P⟹Q premise needs to be re-evaluated and corrected?

       • Or, have the underlying requirements changed or been misunderstood?

     • Outcome: This critical outcome prompts us to "rethink" – either the code needs fixing, or the test needs adjusting, or the specification needs clarification. This is common in exploratory testing or when working with new/evolving features where the exact behavior is still being defined.

   • Sub-scenario 2B: Confirming Known Incorrectness (Corresponds to Row 4: P False, Q False):

     • We observe Q is False (the test fails).

     • We already know or intentionally designed the code to be incorrect for this scenario (that is, we are actively developing a feature and haven't written the full code yet, or we're running a test against a known, un-fixed bug, so our expectation is P is False).

     • The test result simply confirms our prior knowledge that P is False. The test correctly highlights the missing or incorrect behavior. In this case, the P⟹Q implication is vacuously true, and the test effectively served its purpose of showing the existing defect.

     • Outcome: This is typical in Test-Driven Development (TDD) in the Red phase, where a failing test for a not-yet-implemented feature confirms the "P is False" state, guiding development to make P True. It also applies when verifying that a bug fix indeed works: the test initially fails (confirming the bug), and then passes after the fix (confirming P is now True).

A Closer Look at Testing

The Illusion of Correctness: Affirming the Consequent

Consider a common scenario where a test passes, seemingly validating our code:

def get_user_role(user_id):
    if user_id == 42:
        return "admin"
    return "guest"

# test
assert get_user_role(42) == "admin"

Here, our implicit claim (the specification) is: If the code is correct (P), then the output will match the expectation (Q).

In this example, the test passes – the output is "admin" (Q), but can we definitively conclude that the function is correct (P)? Not necessarily.

This scenario often exemplifies the logical fallacy of affirming the consequent. We see the desired outcome (Q) and mistakenly assume that our specific intended cause (P, the correctness of our specific implementation path) was the reason.

The Problem: What if the real condition for an "admin" role should be checking a database, but we have temporarily hardcoded the value for testing? The test would pass, but the correctness is illusory. If we see P as false because the code did not implement the behaviour from the full specification, this corresponds to Row 3 (P False, Q True: False Positive) in our truth table.

As I mentioned before, deliberately implementing ¬P works well if ¬Q is observed, but is not useful, or even erroneous, if Q is observed.

Even without hardcoding, the output might match by coincidence, or because of factors outside the direct logic we intended to test. This can happen due to:

• Default behavior: A broader system default might produce the expected output.

• Caching: A previous successful operation might have cached the result, bypassing the actual logic.

• Fallback logic: An unintended fallback mechanism produces the correct output despite an error in the primary path.

• Test harness bugs: Flaws in the testing setup itself might obscure real issues.

The Role and Risks of Test Doubles

The challenges highlighted above are particularly relevant when using test doubles, such as Stubs and Mocks. These are artificial components that replace real dependencies (for example, databases, external APIs, time-sensitive operations) during testing.

• Stubs focus on state: they provide pre-programmed fake data or return values to get the rest of the code under test working predictably, like the get_user_role example

• Mocks focus on behavior: they allow you to verify interactions, such as the number of calls made to a certain API, or how control flow flows through specific parts of the system.

Both remove external dependencies, allowing you to isolate and focus on the internal logic of the code without noise or side effects. But using them without understanding their limitations can lead to false confidence.

If a test double simulates a "correct" response, but the real dependency it replaces has a bug, or the way the main code interacts with that dependency is flawed, the test will pass (Q is True) – yet P (the code's overall correctness in a real environment) might be False, leading to a dangerous false positive.

Whether you encounter such logical fallacies in your testing depends on precisely what behavior or state you are attempting to verify, and whether you are over-interpreting the test results.

Test Scope and Interpretation

The choice of testing scope – from narrowly focused unit tests to broader integration tests, system tests, user acceptance tests (UAT), and even testing in production – represents a continuum. On this spectrum, various trade-offs are involved, especially concerning the effort-reward ratio. This effort is influenced by factors like individual developer skill, company engineering practices (for example, responsibility split between feature developer and dedicated tester roles), and industry regulations.

Generally:

• Smaller-scoped tests (for example, unit tests) have fewer assumptions baked in and a shorter chain of logical implications. This translates to less risk of committing fallacies in both test implementation and test result interpretation. They are excellent for quickly verifying isolated units of code.

• Larger-scoped tests (for example, end-to-end integration tests) incorporate more real-world complexities and dependencies. While providing higher confidence in the system's overall behavior, they inherently increase the potential for confounding factors that can lead to false positives or make debugging more challenging.

Being acutely aware of the assumptions implicit in each test, at every scope level, is paramount. Passing tests for the wrong reasons will inevitably cause problems down the road.

Debugging, Observability, and Mental Models

Failing tests are not failures of the testing process but are, in fact, incredibly valuable learning moments. They represent opportunities to:

• Run focused debugging experiments to pinpoint the exact cause of the failure.

• Refine your mental model of the code-to-outcome (P⟹Q) link. A failing test (where Q is False) tells you that your current understanding of P, or of the P⟹Q relationship, is flawed. Use this feedback to update your understanding of the code's actual behavior.

• Improve both the code and the tests themselves.

Enhance system observability to better detect and confirm outcomes (Q). The more clearly, from multiple angles, and through diverse methods we can observe Q (for example, logs, metrics, tracing, output inspection), the more confident we can be in its causes and, by extension, the actual state of P.

Crucially, avoid blindly fixing tests just to make them pass. Always ensure you thoroughly understand why a test failed and update your P⟹Q model accordingly. The ultimate goal is not just to fix current bugs, but to prevent them in the future by continually strengthening both the correctness of the code and the verifiability of its behavior.

Falsifiable Tests Reveal Regressions

Beyond avoiding false positives (where the code is incorrect but the test passes), a good test must also be falsifiable. This means the test must be genuinely capable of failing under certain (incorrect) conditions. An unfalsifiable test is a broken test – it cannot serve its purpose of revealing regressions or confirming the presence of bugs.

While we strive for the implication P⟹Q to hold true for all the scenarios we care about, it may not be true for all cases due to unforeseen or mistaken assumptions, or simply because the code is incorrect. The test's ability to demonstrate this incorrectness by failing under specific, well-defined conditions makes it profoundly valuable.

Some common culprits for unfalsifiable or "bad" tests include:

• Vague or Untestable Specifications: Statements like "The system should behave well under most conditions," "It shouldn't crash randomly," or "The algorithm is robust" lack clear, measurable criteria. It's impossible to design a test that definitively passes or fails against such statements, thus rendering them effectively unfalsifiable.

• Broken Implementations of the Test Suite: The test code itself might be flawed, perhaps due to logical errors or control flow issues that prevent assertions from ever being reached or correctly evaluated, inadvertently taking the same passing path regardless of the code under test.

• Insufficient Test Data or Edge Cases: If tests only cover "happy path" scenarios and fail to include challenging inputs or boundary conditions, they might pass for incorrect code that only breaks under specific, untested circumstances.

A robust specification clearly defines what constitutes success and failure. Correspondingly, a good test suite correctly implements that specification, making its tests both accurate and truly falsifiable.

Take a step back

Critical thinkers might observe that the application of the four fundamental logical argument forms to coding scenarios, as initially presented, could be misleading in the complexities of real-world software.

The next section shows some nuances that arise when we transition from the clear-cut rules of formal logic to the often messy reality of software development.

Specifically:

• The first two points below show why the seemingly valid arguments of Modus Ponens and Modus Tollens may not always lead to reliable conclusions when applied to coding scenarios.

• The last two points below show why the two common logical fallacies, Affirming the Consequent and Denying the Antecedent, may actually provide correct insights under specific real-world coding conditions.

Revisiting the Four Statements for Coding

Here are the four arguments and their associated coding examples:

1. Modus Ponens: If you provide invalid input data (P), the code will show an error (Q).

2. Modus Tollens: There are no error messages (¬Q), so the input data is valid (¬P).

3. Affirming the Consequent (Fallacy): The code showed an error (Q), so you provided invalid data (P).

4. Denying the Antecedent (Fallacy): You provided valid data (¬P), so you have no error (¬Q).

Now, let's dive into the nuances of each:

Modus Ponens

• Our coding example: If you provide invalid input data (P), then the code will show an error (Q).

• Why it may not always hold: This application of Modus Ponens assumes that either your code or any third-party code it relies upon will always properly detect and explicitly raise exceptions or show errors on bad data. In reality, systems might automatically fix or sanitize bad input, silence errors, or simply proceed with unexpected behavior without explicitly signaling an error, leading to a passing (or non-failing) state (¬Q) even when P (invalid input) was true.

Modus Tollens

• Our coding example: There are no error messages (¬Q), so the input data is valid (¬P).

• Why it may not always hold: This application of Modus Tollens assumes there are no automatic mechanisms within the system to fix or silence bad input before errors are typically displayed. If such "silent correction" or "error suppression" occurs, you might observe no error messages (¬Q), but the input data could still be invalid (P), rendering the conclusion (¬P) false despite the premise (¬Q) being true. This highlights the dangers of incomplete observability.

Affirming the Consequent (Fallacy)

• Our coding example: The code showed an error (Q), so you provided invalid data (P).

• Why it may actually be correct: While logically a fallacy, in specific, highly constrained real-world conditions, this inference can gain practical validity. If the error message is so uniquely and specifically defined that it can only be caused by invalid input data (P) and no other known factor, then this statement can become reliable. This is rare and typically requires meticulous error handling design where each error message maps unambiguously to a single root cause.

Denying the Antecedent (Fallacy)

• Our coding example: You provided valid data (¬P), so you have no error (¬Q).

• Why it may actually be correct: Although a fallacy in general logic, this inference can hold a high degree of practical confidence under certain programming paradigms (Functional Programming). If the code is sufficiently simple, purely functional (meaning outputs depend only on inputs and have no side effects), and has no external dependencies (like network or database interactions), then the absence of invalid data (¬P) can indeed make us reasonably confident that there will be no errors (¬Q). The lack of external variables and internal state makes the code's behavior highly predictable and directly tied to its inputs.

You may now be thinking: what’s the point of studying logic if it has so many loopholes and edge cases when applied to coding?

The Missing Ingredient – If and Only If

In our exploration of logical implications, we've focused primarily on the unidirectional relationship P⟹Q ("If P, then Q"). This statement tells us what happens if P is true, but it remains silent on whether Q only happens when P is true. It's like saying, "If it rains, the ground gets wet." This is true, but the ground can also get wet if a sprinkler is on, even if it's not raining.

But in many critical contexts, especially in rigorous scientific theories and robust software systems, we often seek a much stronger relationship: one where the truth of Q absolutely depends on the truth of P, and vice versa. This powerful bidirectional relationship is captured by the phrase "If and Only If" (P⟺Q).

What "If and Only If" Means: A Stronger Statement

When we assert "P⟺Q", we're making two distinct claims simultaneously:

1. If P, then Q (P⟹Q): P is a sufficient condition for Q. Whenever P is true, Q must also be true.

2. If Q, then P (Q⟹P): P is also a necessary condition for Q. Whenever Q is true, P must also be true. In other words, Q cannot be true without P being true.

Notice the significant increase in the strength of the statement. "If P, then Q" merely states a consequence. "P⟺Q" declares a definitive equivalence, where P and Q are inextricably linked. They rise and fall together – one cannot be true without the other being true, and one cannot be false without the other being false.

Bidirectional Truth Table: Unambiguous Relationships

Let's construct the truth table for P⟺Q to clearly see this strong relationship.

P⟺Q is logically equivalent to (P⟹Q)∧(Q⟹P).

Creating the Table (columns 4 and 5 are new):

• Q⟹P (Column 4): We apply the standard implication rules, but with Q as our "if" and P as our "then." For instance, in Row 3, Q is True and P is False, so Q⟹P is False.

• P⟺Q (Column 5): This is the logical AND of the P⟹Q and Q⟹P columns. For P⟺Q to be True, both component implications must be True, which explains why you see less Trues in the bidirectional implication compared to any of the unidirectional implications.

Implications for the Two Common Fallacies

The clarity provided by "If and Only If" is particularly powerful in preventing the very logical fallacies we discussed earlier: Affirming the Consequent and Denying the Antecedent. These fallacies arise from the incorrect assumption that an "if-then" statement implies an "if and only if" relationship.

Let's revisit them with the lens of P⟺Q If and Only If you provided invalid data (P), then the code will show an error (Q):

Affirming the Consequent: No More Ambiguity

• The Fallacy (assuming unidirectional P⟹Q):

  • If the code showed an error (Q), then you provided invalid data (P).

  • Previously, when P⟹Q was True and Q was True, P could be True (Row 1) or False (Row 3). This ambiguity led to the fallacy.

• With P⟺Q:

  • Now, look at the P⟺Q column in the table. When P⟺Q is True and Q is True (Row 1), P is unambiguously True. The confusion from Row 3 is gone because if Q were True while P was False, P⟺Q would be False (as Q⟹P would be False), thus making that row irrelevant for valid modus ponens inference under the P⟺Q premise.

  • In a system designed with P⟺Q in mind, knowing that Q is True (observing an error) would force the conclusion that P is True (invalid data is the cause), assuming the "if and only if" relationship holds true for that specific system design.

Denying the Antecedent: Unmistakable Consequences

• The Fallacy (assuming unidirectional P⟹Q):

  • You provided valid data (¬P), so you have no error (¬Q).

  • Previously, when P⟹Q was True and P was False, Q could be True (Row 3) or False (Row 4). This ambiguity led to the fallacy.

• With P⟺Q:

  • Now, when P⟺Q is True and P is False (Row 4), Q is unambiguously False. The problematic scenario from Row 3 (where P was False but Q was True) is irrelevant here because P⟺Q would be False in that case (specifically, Q⟹P would be False).

  • If your system genuinely adheres to "P⟺Q", then knowing that P is False (valid data provided) guarantees that Q is False (no error messages).

Practical Mitigation in Coding

The insights from "If and Only If" are more than just theoretical. Practically, both fallacies (Affirming the Consequent and Denying the Antecedent) can be mitigated by striving for conditions that approximate an "if and only if" relationship in your code and tests.

Focused Unit Tests

Design unit tests that are so granular and isolated that they effectively aim to establish an "if and only if" scenario for a tiny piece of logic. By thoroughly mocking or controlling all external dependencies and environmental factors, you reduce the impact of "other causes."

If your test for a specific input passes, you want to be as confident as possible that it passed only because the code handled that specific input correctly, and not due to some irrelevant side effect. Similarly, if it fails, you want to be sure that the failure points directly to the intended logical path.

Exception Handling and Specificity

Instead of catching broad Exception types, catch and handle specific exceptions. This helps differentiate between various "causes" (P1​,P2​,…) that might lead to a generic "error" (Q). The more precise your error handling, the closer you get to a scenario where "If X error, then Y specific cause," moving towards a bidirectional understanding of error conditions.

Test-Driven Development (TDD) and Mutation Testing

These methodologies inherently push towards P⟺Q thinking. TDD encourages writing a failing test first (¬Q), which then necessitates a specific code change (P) to make it pass.

Mutation testing, which we'll explore further, takes this a step further by ensuring that your tests are robust enough to fail when code is subtly altered (that is, proving that ¬P leads to ¬Q, and thus, that the original P was indeed necessary for Q).

By consciously aiming for "if and only if" relationships in your code's design and your testing strategies, you can build systems that are not only predictable but also much easier to debug and reason about, moving beyond mere correlation to a deeper understanding of cause and effect.

Callback to Mutation Testing

In the earlier section on Assigning Real-World Meanings to Logic, we discussed:

When testing the happy path, we are verifying that the implication P⟹Q (If input is good, then no error) holds true.

When testing the unhappy path (mutation testing), we are verifying that ¬P⟹¬Q (If input is not good, then an error occurs) holds true.

This dual view is key to understanding how mutation testing contributes to software correctness.

Mutation Testing: Testing the Tests

Mutation testing deliberately introduces small faults (mutations) in the code and checks whether the test suite detects them by failing. This process assesses not the code, but the tests themselves.

In a robust test suite, we strive for two ideal conditions:

• All correct implementations should pass the tests.

• All incorrect implementations should fail the tests.

If a mutated (wrong) version of the code is introduced and causes no test failures, that defeats the fundamental purpose of testing. It means your tests aren't sensitive enough to catch a deviation from correctness. Mutations reveal hidden assumptions or gaps in your test coverage, acting as a sensitivity probe for your test suite.

Example code mutations:

• Changing an arithmetic operator (+ to -, > to >=).

• Flipping a boolean condition (true to false).

• Deleting or duplicating a statement.

• Modifying a constant value.

Common Python mutation testing tools:

• mutmut uses Python’s built-in ast module.

• cosmic-ray uses parso, which provides a more complete AST.

These tools rely on abstract syntax trees to surgically mutate code.

You can even swap out underlying AST libraries for different precision or completeness: https://github.com/boxed/mutmut/issues/281

Logic Behind Mutation Testing

Let's formalize the logical mapping of mutation testing, recalling our definitions:

• Let P: Code is correct.

• Let Q: Tests pass.

Standard happy path testing primarily checks that P⟹Q – "if the code is correct, then tests pass."

Mutation testing focuses on the other side of the coin: we intentionally make ¬P true (by introducing a fault), and then we expect ¬Q (the tests should fail). This process rigorously checks whether the implication ¬P⟹¬Q ("if the code is not correct, then the tests fail") holds true for your test suite.

But there's a deeper, more powerful logical implication here:

As we learned earlier, the statement ¬P⟹¬Q is logically equivalent to its contrapositive, Q⟹P.

So, by successfully verifying that introducing a fault (¬P) leads to a test failure (¬Q), we are simultaneously validating the contrapositive: if tests pass (Q), then the code must be correct (P).

This is incredibly significant! It moves us much closer to establishing a bidirectional guarantee between our code and our tests: P⟺Q (code correctness is tightly coupled with test success). Mutation testing helps us confidently eliminate false positives in the test suite – situations where Q is true (the test passes) but P is false (the code is actually incorrect).

In a world where LLMs help us write and refactor code quickly, having this "if and only if" confidence in our test suite is invaluable for ensuring the generated or refactored code truly meets expectations.

Clarifying the Kinds of Failures

In software, we typically categorize errors into three main types:

• Syntax errors: Violations of the language's grammatical rules (for example, missing colon, invalid keyword). These prevent the code from running at all.

• Runtime errors: Errors that occur during program execution, often due to unexpected conditions (for example, TypeError, AttributeError, ZeroDivisionError).

• Logic errors: The program runs without crashing, but it produces an incorrect result or behaves in a way that doesn't match the intended specification (for example, wrong algorithm, wrong return value).

Mutation testing focuses on logic errors – failures where the program runs, but produces incorrect results. These are usually caught via AssertionError in the "Assert" phase of the Arrange–Act–Assert (AAA) testing pattern.

You could argue pedantically that AssertionError is a runtime error, but in testing, we treat it as a signal for logical failure:

"The function ran, but the output didn’t match the expected behavior."

Mutation testing assumes that syntax and runtime errors are already handled. Its purpose is to validate whether the test suite reliably catches logical misbehavior.

A Deeper Falsification Perspective

Now, let's connect mutation testing back to Karl Popper's principle of falsification, which we introduced earlier in the context of scientific reasoning. Recall that Popper argued scientific theories gain strength not by being "proven," but by surviving rigorous attempts to disprove them. The core idea of falsification logic is that to disprove an implication like P⟹Q, you only need to find one instance where P is True and Q is False.

Mutation testing applies this same powerful principle, but to our test suite's effectiveness:

Instead of trying to prove directly that our tests are perfect, mutation testing takes a falsification approach to the implication ¬P⟹¬Q ("If the code is incorrect, then the tests fail"). It actively tries to falsify this crucial relationship.

If we introduce a mutation (making ¬P true, that is, the code is now incorrect) but the existing test suite still passes (meaning Q is true), then we have found an instance where:

1. ¬P is True (the code is incorrect due to the mutation).

2. Q is True (the test still passes).

In this scenario, the implication ¬P⟹¬Q is falsified because we have a True antecedent (¬P) leading to a False consequent (¬Q is false, because Q is true).

And, critically, if ¬P⟹¬Q is falsified, then its logically equivalent contrapositive, Q⟹P ("If the tests pass, then the code is correct"), is also falsified. This means we can no longer trust that a passing test suite reliably indicates correct code. Our desired P⟺Q relationship is broken – the test suite is no longer fully effective at guaranteeing correctness.

By pushing for zero surviving mutants, mutation testing forces us to minimize the surface area of these "hidden assumptions" in our test suite. It demands highly sensitive and specific tests that can pinpoint even subtle logical flaws, thereby moving us closer to building truly resilient systems.

Comparing TDD (Red Phase) and Mutation Testing

Both methodologies, albeit through different means and at different stages of the development cycle, aim to establish confidence in the ¬P ⟹ ¬Q relationship.

Key Differences Summarized:

Toward If-and-Only-If Confidence

Ultimately, the goal in software development is to establish if-and-only-if relationships whenever possible, both in the code implementation and especially in the sensitivity of the test suite to the code under test.

This means if a certain condition (P) is true, then a specific outcome (Q) must occur, and if Q occurs, then P must have been the cause. Achieving this level of clarity comes from:

• A deep understanding of the problem.

• Aligned expectations during requirements gathering.

• Logical analysis and interpretation of well-designed experiments.

• Adherence to Single Responsibility Principle in SOLID

• Rigorous tests with meaningful coverage.

This allows us to understand how control flow and data flow work with greater depth and confidence, leading to better inferences throughout the entire software development lifecycle.

Real-World Challenges

While striving for perfect "if-and-only-if" relationships provides a powerful logical ideal, the messy reality of modern software development presents significant hurdles. The very characteristics that make large systems powerful and scalable – their intricate interconnections and inherent dynamism – simultaneously obscure clear cause-and-effect relationships, making precise logical reasoning and debugging an ongoing battle.

A Web of Complexity

Fan-In, Fan-Out: The Nature of Modern Systems

Any reasonably large software system rarely operates through purely linear control and data flows. Fan-out and fan-in patterns – where many components are called and then their results merged – are inevitable.

For example:

• In ETL pipelines, data may be ingested from multiple sources (external APIs, CSVs) and logged to multiple destinations (files, databases).

• In concurrent programming, Python’s ProcessPoolExecutor splits data into chunks processed in parallel, then recombines the results.

SRP Meets Real-World Boundaries

Just as functional programming must eventually perform I/O, the Single Responsibility Principle (SRP) runs into real-world boundaries, whether conceptual or infrastructural. At some point, something must glue these isolated units together.

Orchestration logic might live in a single function, span multiple files, or even distribute across microservices and machines communicating over networks. While this decomposition enhances modularity, it also increases surface area for bugs involving:

• Side effects: Unintended changes to system state outside a component's explicit outputs.

• Circular dependencies: Components relying on each other in a loop, leading to difficult-to-trace behavior.

• Interface drift: Changes in one component's input/output expectations not being correctly reflected elsewhere.

• Race conditions: Timing-dependent bugs in concurrent operations.

• Serialization issues: Problems translating data between different formats or systems.

• Network unreliability: Unpredictable latency, packet loss, or disconnections in distributed systems.

The Double-Edged Sword of Abstraction

This web of dependencies is the price of progress, made manageable only through better tooling and abstractions.

• If boundaries are well-designed, observable, and testable, they enable asynchronous collaboration, improve long-term maintainability, and increase developer confidence. (See GitHub Playbook in References)

• If systems lack architectural coherence or fall behind evolving needs, they calcify into technical debt that demoralizes even the most motivated teams.

Clean Code Is Contextual

While abstractions and orchestration help manage complexity, overusing design patterns or creating unnecessary class layers can introduce needless indirection. This is a common counterargument to architectural purism.

Ultimately, what counts as "clean code" is context-dependent. It varies with programmer skill, the tooling at hand (linters, tests, Copilot), and whether the project is a throwaway script or a multi-year infrastructure investment. Architectural practices like SRP should evolve alongside those constraints.

The Butterfly Effect of Bugs

From SRP to Reasoning Chains

Previously, we focused on simple, direct cause-effect logic (P ⟹ Q), but real-world systems are messier.

The more we adhere to SRP through small, focused functions, the more we create longer chains of logic. This improves separation of concerns but also extends the reasoning required to debug behavior.

Debugging in a Causal Fog

A seemingly minor trigger (O) can cascade through a chain like O⟹P⟹Q⟹R, which we may not fully understand due to knowledge silos, evolving requirements, or runtime dynamism.

Even when we understand the components, precisely identifying “P” is hard, much like how redefining a research question shifts the statistical population being studied. In complex systems with feedback loops (recommender engines), there might not be a single "root cause" at all.

Short-Term Triage vs. Long-Term Insight

Finding the true origin of a bug often demands experimentation, telemetry, and broad system insight. These investigations produce robust, future-proof fixes but take time.

In on-call scenarios, however, urgency reshapes priorities. Fast mitigations and clear communication often take precedence over deep diagnosis.

Masked by Design and Debt

As systems scale, failure stops looking like a crash. Instead, it shows up as a retry spike, a slow metric drift, or silent fallback behavior.

Modern fault-tolerant systems, built with retries, failovers, circuit breakers, and autoscaling, are designed to recover quickly. This resilience often masks deeper problems, delaying detection for weeks and making root cause analysis harder.

Operating in non-deterministic environments with flaky networks, race conditions, or dynamic routing adds further ambiguity. Small symptoms become harder to link back to specific causes.

Compounding this, technical debt driven by weak technical leadership, shifting priorities or time pressure weakens the system’s observability and test coverage. Teams inherit brittle, poorly understood code, making it hard to draw clean lines between cause and effect.

Even the best engineers struggle in such conditions. When a system resists clarity, it doesn’t just block debugging. It erodes trust, slows learning, and fuels long-term burnout.

Glimmers of Hope: Tools and Practices for Clarity

Despite these challenges, several strategies and practices offer a path toward more robust and understandable software.

Leveraging Design Patterns

Design patterns offer a shared vocabulary and time-tested strategies for structuring systems. When applied well, they tame complexity, reduce technical debt, and make behavior more predictable.

They also tend to concentrate similar failure modes. The same bug might appear across companies or industries, creating a wealth of prior art and solution playbooks. Familiarity with patterns can accelerate debugging and deepen shared understanding across teams.

Nurturing Expert Mentorship

Promoting mentors based on real technical impact instead of tenure builds stronger teams and avoids the Peter Principle (people in a hierarchy tend to rise to a level of respective incompetence).

Great mentors teach more than skills – they model falsifiability, independent thinking, and an ability to reason under uncertainty.

They help others challenge assumptions, navigate tradeoffs, and grow both technically and interpersonally. In systems where root causes are murky, this kind of leadership is essential.

One of the most powerful techniques that scales from mentorship to code is falsification: the disciplined search for counterexamples. Whether applied in design reviews, debugging sessions, or automated tests, this mindset anchors reasoning in reality.

The Power of Falsification in Testing

The deliberate search for counterexamples is core to building reliable systems.

• In algorithm design, testing edge cases is just falsification in disguise: finding where your logic breaks.

• In code, fuzz testing (Atheris) throws diverse inputs at functions to expose falsifying examples.

• Property-based testing (Hypothesis) goes further by generating inputs that satisfy certain rules, then shrinks failures to their minimal form. This greatly improves reproducibility and helps stress-test concurrency issues.

The more rigorously we attempt to falsify our assumptions, the more confidently we can reason about behavior using tools like Modus Ponens and Modus Tollens.

Assumptions are always present in software to simplify complexity. The question is whether they're explicitly codified in tests or left hidden and fragile.

Of course, no test is ever bulletproof: our assumptions could be mistaken, or the world could change. That’s why critical thinking, discerning "what should be" versus "what is", remains essential as newer generations increasingly rely on AI tools like Large Language Models.

This deliberate, falsification-driven approach is paramount for building reliable software. It underpins sophisticated testing techniques designed to expose hidden assumptions and break our logical chains.

While testing helps us uncover where our reasoning might falter, some domains demand an even higher degree of certainty. For those critical systems, we turn to the ultimate tools for logical rigor: Proof Assistants.

Proof Assistants

While traditional testing and fuzzing are powerful for finding bugs, they fundamentally cannot guarantee correctness for all possible inputs or scenarios. They can only prove the presence of bugs, not their absence.

To achieve formal, mathematically verified proofs of program behavior – providing the strongest possible guarantees – we turn to proof assistants. These tools allow us to build step-by-step logical proofs, ensuring that a program or system design adheres to its specification with absolute rigor.

Prolog

Prolog offers a relatively straightforward entry point into the world of logic programming and theorem proving. SWI-Prolog is a common interpreter (a REPL, or Read-Eval-Print Loop) for Prolog.

You interact with Prolog by providing it with a knowledge base composed of facts and rules (which are a type of logical clause called Horn clauses). You then pose queries.

Installing SWI-Prolog

You can download SWI-Prolog from its official website: https://www.swi-prolog.org/download/stable
Follow the instructions for your operating system (Windows, macOS, or Linux).

On Ubuntu/Debian, you can usually install it via:

sudo apt update
sudo apt install swi-prolog

Using Prolog: REPL vs. File

• REPL (swipl) is best for: Quick, interactive tests of single facts or rules, and posing queries to an already loaded knowledge base.

• A File (.pl extension) is best for: Defining your entire knowledge base (multiple facts and rules) and storing your program for reusability. This is the standard way to work with Prolog for anything beyond a few lines.

Example: A Simple Knowledge Base

Let's define a knowledge base to represent who has a job and who is a coding instructor.

1. Create a file named knowledge.pl with the following content:

% knowledge.pl
% This file defines a small knowledge base in Prolog.
% In Prolog, all statements (facts and rules) about the same predicate
% (identified by its name AND number of arguments, e.g., 'has_job' with 1 argument is 'has_job/1')
% must be written consecutively without other predicate definitions in between.

% --- Definitions for the 'has_job' predicate (takes 1 argument) ---

% Fact: Alice has a job.
has_job(alice).

% Fact: Bob has a job.
has_job(bob).

% Rule: Anyone (represented by variable X) has a job IF they are a coding instructor.
% ':-' means 'if'. 'X' is a variable (starts with uppercase).
has_job(X) :- is_coding_instructor(X).

% --- Definitions for the 'is_coding_instructor' predicate (takes 1 argument) ---

% Fact: Alice is a coding instructor.
is_coding_instructor(alice).

What each line does:

• Lines starting with %: These are comments for human readability, ignored by Prolog. They explain the file's purpose and key rules like predicate grouping.

• has_job(alice). / has_job(bob).: These are facts. They assert simple truths, like "Alice has a job." The . at the end is mandatory for every statement.

• has_job(X) :- is_coding_instructor(X).: This is a rule. It states a conditional truth: "For any X, X has a job if X is a coding instructor." X is a variable (always starts with an uppercase letter), and :- means "if." This rule allows Prolog to deduce new information.

• is_coding_instructor(alice).: Another fact, asserting "Alice is a coding instructor." It's placed after all has_job/1 clauses to satisfy Prolog's grouping rule.

2. Load and Query in the REPL:

Open your terminal and type swipl. Once at the ?- prompt, load the file and then pose your queries:

$ swipl
?- [knowledge].   % Load the 'knowledge.pl' file (omit .pl, use square brackets and a period)
% Press Enter. Prolog will confirm it loaded the file, e.g., '% knowledge.pl compiled...'
True.

?- has_job(alice). % Query: Does Alice have a job?
% Press Enter. Prolog gives you a solution, then waits.
True.              % Output: Yes, because it's a fact.
% After 'True.', you'll see the '?- ' prompt again, indicating Prolog is ready for your next query.
% If there were multiple ways to prove 'True.', Prolog would present the first 'True.' then wait for you to press ';' for alternatives, then Enter to confirm the final 'True.' or 'False.'.

?- has_job(carol). % Query: Does Carol have a job?
% Press Enter.
False.             % Output: No, Prolog cannot prove it from its knowledge.

?- has_job(X).     % Query: Who has a job? (Find values for X)
% Press Enter
X = alice ;        % Prolog finds Alice as the first solution. Type ';' and press Enter to ask for the next solution.
X = bob ;          % It finds Bob. Type ';' and press Enter for the next solution.
X = alice          % It finds Alice again (this time deduced via the rule and is_coding_instructor(alice)).
% Press Enter. This accepts the current set of solutions and stops searching for more.
False.             % Output: Indicates no more solutions found after the last 'Enter' (or if you explicitly chose not to search further).

?- halt.           % Type 'halt.' to exit the Prolog REPL cleanly.
% Alternatively, you can often use Ctrl+D (press and hold Ctrl, then D) to exit most REPLs.

The Prolog example clearly demonstrates:

• "Is P(X) true for a specific X?": Shown by ?- has_job(alice). (returns True.) and ?- has_job(carol). (returns False.).

• "Is there an X for which P(X) is true?": Shown by ?- has_job(X). (provides solutions like X = alice, X = bob).

Prolog Limitations

Prolog's limitations become evident when attempting to reason about falsity or non-existence. You cannot directly ask "Is there any X for which P(X) is false?"

Instead, Prolog operates on the principle of negation as failure. This means that if Prolog cannot prove a statement, it considers that statement false.

For example, if you ask ?- \+ has_job(carol). (meaning "Is it not true that Carol has a job?"), Prolog will say True, because it simply cannot find any proof that Carol has a job in its knowledge base.

This is a significant distinction: it doesn't mean Carol definitely doesn't have a job, nor does Prolog provide a formal counterexample. It merely reflects a lack of provable information.

This fundamental constraint means Prolog, while powerful for logic programming, falls short of being a full-fledged proof assistant for comprehensive formal verification.

Coq

After experimenting with Prolog and seeing its limitations, you can move on to a more powerful proof assistant like Coq. Coq is employed in safety-critical domains where absolute mathematical certainty is paramount. coqtop is the standard REPL for Coq.

A fundamental difference from Prolog is Coq's lack of a Closed World Assumption. In Coq, anything not explicitly proven is simply unknown, not automatically false.

Unlike Prolog, Coq's primary purpose isn't solving computational problems by searching a knowledge base. Its true power lies in its ability to construct and verify formal mathematical proofs and programs with absolute rigor. Its interaction involves managing a proof state (your remaining goals) and applying tactics (logical inference steps) until the proof is complete.

Installing Coq

Coq can be installed in several ways, often via package managers or a tool called opam (the OCaml package manager, as Coq is written in OCaml).

• Official Downloads: Visit the Coq website for detailed instructions for your OS: https://coq.inria.fr/download

• Using a system package manager (for example, Ubuntu/Debian): Bash

    sudo apt update
    sudo apt install coq
  

Using Coq: REPL vs. File

• REPL (coqtop) is best for: Trying out single tactics, inspecting the current proof state, or learning basic syntax for very short commands.

• A File (.v extension) is best for: Almost all Coq development and proof construction. This is how complex proofs and verified programs are structured and managed.

Coq's Comprehensive Question Answering

Unlike Prolog, Coq can directly address all three types of logical questions we've discussed, providing robust answers backed by formal proof:

• "Is P(X) true for a specific X?": Coq allows you to define a precise statement (a theorem) like "Alice has a job." You then build a step-by-step logical proof that formally confirms whether this statement is true based on your definitions. If the proof succeeds, Coq formally verifies it: if it fails, Coq clearly shows where your logic breaks down.

• "Is there an X for which P(X) is true?": Coq handles questions of existence. If you ask, "Does someone have a job?", you can construct a proof by explicitly providing an example (like "Alice") and then proving that your chosen example indeed satisfies the condition ("Alice has a job").

• "Is there any X for which P(X) is false?": This is a key capability where Coq excels over Prolog. Coq allows you to formally prove that a statement is false, or that a counterexample exists. For instance, you could prove "Carol does not have a job" by showing it contradicts the definition, or prove "there exists someone who doesn't have a job" by explicitly identifying such a person and proving that they indeed lack a job. This direct ability to reason about negation and provide formal counterexamples (or prove their non-existence) is what makes Coq a full-fledged proof assistant.

While Coq's core doesn't automatically generate counterexamples when a proof fails, plugins like QuickChick can be integrated for property-based testing to find falsifying examples.

It's a Coq library that allows you to specify properties about your Coq definitions and then randomly generate inputs to try and find a counterexample that falsifies your property.

This is a powerful way to find bugs early in your formalization before you invest a lot of time trying to prove a false theorem.

TLA+, Isabelle, and Lean: A Spectrum of Formal Verification

Beyond Prolog and Coq, other powerful proof assistants and formal specification languages cater to different needs and paradigms:

• TLA+: This is a formal specification language developed by Leslie Lamport. It focuses on modeling and verifying system designs (especially concurrent and distributed ones) using temporal logic, rather than proving low-level code. It helps ensure critical properties like safety (nothing bad ever happens) and liveness (something good eventually happens). Its practicality and accessibility make it popular in industry, notably at Amazon and Microsoft for robust system design.

• Isabelle and Lean: These are modern, highly advanced proof assistants.

  • Isabelle, grounded in higher-order logic, is widely used by researchers and institutions (for example, in projects like the seL4 verified microkernel) for formal theorem proving and software verification in academic and safety-critical domains demanding extreme rigor.

  • Lean, based on dependent type theory, is favored by mathematicians for formalizing proofs in pure mathematics (for example, number theory, algebra). It's known for its powerful automation and active community.

These tools represent the pinnacle of applying formal logic to ensure the correctness and reliability of both mathematical theories and complex software systems.

Now that you have a good lay of the land in both theory and practice, here are some thought experiments to enrich your education.

Food for Thought

The journey into formal logic and its intersection with practical domains like software and science offers many avenues for deeper exploration.

Hypothesis Testing in Science and the Implication Truth Table

Statistical hypothesis testing uses a probabilistic form of Modus Tollens. We start with a null hypothesis (H0​): "If H0​ is true, then observing this data (or more extreme data) is likely." We then observe data that is highly unlikely/unexpected if H0​ were true (that is, a small p-value). This serves as our probabilistic "not Q." Therefore, we conclude that H0​ is likely not true (we reject H0​). This is our probabilistic "∴¬P."

Here, the "truthiness" of P⟹Q is being tested, rather than simply assumed to be true for developing arguments, as in Modus Ponens or Modus Tollens. There's no absolute truth or anything to "prove" definitively.

Inferences are drawn from prior experiments (which inform the test data distribution) and context-specific experiment setups (which determine the significance level α), together defining the threshold (critical value) for what is considered an unlikely observation of Q.

The experiment's result is a rejection (or lack thereof) of H0​, not a definitive proof that H0​ is true.

Inductive Reasoning's Relationship to Deductive Arguments

• Induction generates general rules (for example, "P is always followed by Q") from specific observations or cases.

• Deduction then tests or applies those general rules in new situations.

If deduction leads to wrong predictions (that is, a rule is falsified), induction may need to revise the original rule, which forms a continuous feedback loop that refines our understanding.

Necessity and Sufficiency in Implication

The implication P⟹Q ("If you crossed the border, you must have had a passport") unpacks into two fundamental logical concepts:

• P is sufficient for Q: Crossing the border guarantees you had a passport. (P alone is enough for Q.)

• Q is necessary for P: If you didn't have a passport (¬Q), you couldn't have crossed (¬P). (Q is required for P to happen.)

Q.E.D.: The Enduring Power of Logic in an Uncertain World

Throughout this handbook, we’ve journeyed from the foundational concepts of propositional logic and truth tables to the powerful argument forms of Modus Ponens and Modus Tollens. We explored how these tools enable valid deductions and identified common logical fallacies like Affirming the Consequent and Denying the Antecedent, understanding why they lead to incorrect inferences when an "if-then" relationship isn't a strict "if and only if." We learned the profound importance of falsifiability – the ability for a statement or hypothesis to be disproven – a cornerstone of both scientific inquiry and robust software testing.

We then delved into the practical application of these logical principles in software development, mapping code correctness to test outcomes. We discovered how a failing test, when trusted, becomes a powerful application of Modus Tollens, pinpointing defects. We also confronted the "illusion of correctness" that arises from the affirming the consequent fallacy when tests pass for the wrong reasons, especially when using test doubles.

Crucially, we introduced the "If and Only If" (P⟺Q) relationship, highlighting its unparalleled power in establishing unambiguous connections between cause and effect. This bidirectional guarantee is the ideal we strive for in test suite quality, moving beyond mere correlation to a deeper understanding of causality. We saw how mutation testing rigorously pushes us towards this "if and only if" confidence by actively trying to falsify the assumption that "incorrect code leads to failing tests," thereby strengthening the inverse: "passing tests guarantee correct code."

We also acknowledged the "messy reality" of modern software. Large systems are webs of complexity, with fan-in/fan-out patterns, side effects, and unforeseen interactions that can obscure clear logical chains. Technical debt and the double-edged sword of abstraction often mask the true origins of bugs, turning debugging into a "causal fog."

Logic as Your Compass

Despite these formidable challenges, the logical principles we've explored remain your most vital tools. They provide the mental framework to navigate uncertainty.

When confronted with a bug, your ability to reason logically allows you to formulate hypotheses, design focused experiments (your tests), and interpret their outcomes with precision. Whether you're debugging a complex microservice or reasoning about a simple function, applying Modus Tollens to a failing test or designing tests that aim for P⟺Q clarity helps you cut through the noise.

We also touched upon advanced tools like Proof Assistants (Prolog, Coq, TLA+, Isabelle, Lean), which represent the pinnacle of applying formal logic to guarantee system correctness – a testament to the enduring power of logical rigor in critical domains.

In the intricate dance between theory and practice, the principles of logic stand as an unshakeable foundation. They are the "rocks" upon which you can meticulously build your understanding and your systems. The more consistently you apply this critical thinking, driven by curiosity and a commitment to rigorous validation, the clearer your path becomes.

This clarity is not just about fixing today’s bugs, it’s about continually refining your mental models, fostering trust in your codebase, and equipping yourself to build increasingly robust and predictable systems in an ever-evolving technological landscape.

If you love problem solving, critical thinking, or have experiences on how you fixed an issue that looked different from how it initially seemed, feel free to connect with me at https://linkedin.com/in/hanqi91.

Resources

1. Article that motivated this handbook: Classical Reasoning and Debugging

2. 3 Formal proofs of modus tollens: https://en.wikipedia.org/wiki/Modus_tollens

3. Table of 24 syllogisms: https://en.wikipedia.org/wiki/Syllogism

4. Challenging Assumptions: Falsehoods software teams believe about user feedback

5. How assumptions and software evolve beyond your control: https://www.tdda.info/why-code-rusts

6. Relationship to Hypothesis Testing: https://sites.google.com/view/reasonedwriting/home/FRAMEWORK_FOR_SCIENTIFIC_PAPERS/HYPOTHESES/HOW_TO_TEST_HYPOTHESES/MODUS_TOLLENS

7. The Troubleshooting Mindset: https://www.autodidacts.io/troubleshooting/

8. Causal Diagrams from The Effect Book: https://theeffectbook.net/ch-CausalDiagrams.html

9. A systematic guide to the mindsets and practices of debugging: https://www.amazon.sg/Debug-Find-Repair-Prevent-Bugs/dp/193435628X

10. Constructing P in a way to ensure software correctness: https://www.hillelwayne.com/post/constructive/

11. Fail Fast by explicitly representing assumptions as assertions: https://www.martinfowler.com/ieeeSoftware/failFast.pdf

12. Deterministic Simulation Testing to tackle complex systems: https://pierrezemb.fr/posts/learn-about-dst/

13. GitHub’s Engineering System Success Playbook (ESSP) - Quality, Velocity, Developer Happiness on Business Outcomes: https://assets.ctfassets.net/wfutmusr1t3h/us6AUuwawrtNGTlwlT9Ac/f0fce86712054fc87f10db28b20f303b/GitHub-ESSP.pdf

14. Closed-world assumption: https://en.wikipedia.org/wiki/Closed-world_assumption

Glossary

• Axiom: A fundamental truth or rule accepted as a starting point for a logical or mathematical system, without requiring proof.

• Contrapositive: A logically equivalent form of an "if-then" statement (P⟹Q), which is ¬Q⟹¬P ("If not Q, then not P").

• Deductive Reasoning: A type of logical reasoning where a conclusion is necessarily true if its premises are true.

• Falsification: The principle, especially in science (from Karl Popper), that a hypothesis or theory must be capable of being proven false by empirical observation or experiment.

• Formal Logic: The study of abstract systems of reasoning and arguments based on their structure, independent of content.

• Hypothesis Testing: A statistical method for making inferences about a population based on sample data, typically by testing a null hypothesis (e.g., "P has no effect on Q") against an alternative hypothesis.

• Logical Fallacy: A flaw in the structure or content of an argument that makes it unsound or invalid, even if its conclusion might seem plausible.

  • Affirming the Consequent (Fallacy): An invalid argument form that mistakenly assumes if P⟹Q is true, and Q is true, then P must be true.

  • Denying the Antecedent (Fallacy): An invalid argument form that mistakenly assumes if P⟹Q is true, and P is false, then Q must be false.

• Modus Ponens: A valid argument form: If P⟹Q is true and P is true, then Q must be true.

• Modus Tollens: A valid argument form: If P⟹Q is true and ¬Q is true, then ¬P must be true.

• Mutation Testing: A software testing technique that involves deliberately introducing small, single-point faults (mutations) into code to assess the effectiveness and coverage of a test suite.

• Propositional Logic: A branch of logic that deals with propositions and their relationships using logical operators.

• Test-Driven Development (TDD): A software development methodology where tests are written before the code, guiding the development process and ensuring correctness.

• Truth Table: A table that systematically lists all possible truth values for a set of propositions and shows the resulting truth value of a complex logical statement.

• Vacuously True: Describes an implication (P⟹Q) that is considered true simply because its antecedent (P) is false.

Technologies and industries that were once dominated by one or two companies or were very much “human-focused” are coming under threat.

Google is losing ground to AI search, truck drivers may soon be a thing of the past, and low-skilled clerical jobs are being lost every day.

Will this disruption destroy the Software Engineering industry? I don’t think so, and I’ll tell you why.

Here’s what we’ll discuss:

1. The Phenomenon of "Vibe Coding"

2. How AI Has Changed Software Development

3. The Productivity Paradox

4. Why Human Engineers Are Still Critical

5. AI as a “Capability Multiplier”

6. Critical Skills for the AI Era

7. The Path Forward

The Phenomenon of "Vibe Coding"

If you follow tech discussions on X, you've likely seen the term "vibe coding" – the practice of building software through trial and error, intuition, and AI-generated code snippets without deep technical knowledge.

Modern AI assistants such as GitHub Copilot and ChatGPT can generate full functions, fix bugs, and create components based on simple descriptions. “Vibe Coders” are claiming that human coders will soon become obsolete.

From my perspective, these AI tools function more as skill multipliers than replacements.

They help talented developers work faster while exposing gaps in knowledge for less skilled programmers. Those lacking technical foundations will face problems they can't solve, but engineers who blend AI assistance with solid expertise will be able to be incredibly productive.

How AI Has Changed Software Development

The software industry has seen rapid adoption of AI coding tools based on Large Language Models that analyze code repositories to predict and suggest next steps.

These tools have transformed daily programming work by:

• Suggesting complete functions as you type

• Creating API endpoints from plain language descriptions

• Eliminating hours spent on standard code patterns

• Automating documentation tasks

• Handling repetitive logic quickly

This shift toward "vibe coding" speeds up feature delivery. Programmers can now build without mastering every technical detail – they describe what they want, get AI suggestions, and adjust until the code works.

The risk? Developers often push code they can't explain. They move quickly during building but struggle when systems break or need changing.

There's also a concerning trend of non-programmers selling AI-built applications. Recently, someone with zero coding background launched a paid service created entirely through AI prompts, only to face a data breach days later when hackers exploited basic security flaws. This is dangerous. It has wasted people's money and exposed their data. Imagine if this became common place due to the rise of “vibe coders”?

For anyone considering building software who isn’t a software engineer, there are a few basic levels of security that you need to consider:

• Adding authentication to your API endpoints: People can scan for open ports and endpoints across the internet. If they can then call your API endpoints without being authenticated, it can cause all sorts of problems

• Do not store passwords in plain text. This is a big no no. If you do this and your database gets exposed, those passwords are there for all to see. And if we’re being real, people re-use passwords, so those passwords will be their passwords for other sites too.

• SSL: Make sure your website is secure and has an up to date SSL certificate. Transmitting data in plain text is dangerous.

• Lock down unused ports: If you are hosting a backend service, make sure that any ports that you don’t use are locked down and people aren’t able to connect to them.

• If you have areas where people can upload files, limit the uploads to specific file types.

Those are just a few considerations around security for your site or product, but there are many more.

The Productivity Paradox

AI assistance dramatically increases code output – but volume doesn't equal value in software engineering.

These tools excel at syntax but have no understanding about system architecture, scalability concerns, and maintenance requirements. Just as typing speed doesn't create a better novel, code generation speed doesn't produce better software systems.

AI works for individual functions but struggles with architectural decisions, security planning, and long-term support needs. Without proper review and understanding, AI-generated code often becomes tomorrow's tech-debt and maintenance burden.

Consider this scenario: A developer implements an AI-created authentication system that works in isolation but causes subtle failures in users signing up to the product. Finding and fixing these integration issues might take experienced staff several days – negating any initial time savings. This is a quick path to losing money and trust.

Why Human Engineers Are Still Critical

While AI tools handle syntax well, they cannot:

1. Plan systems that grow with user demand

2. Create reliable deployment and testing pipelines

3. Anticipate unusual but critical failure cases

4. Make smart tradeoffs between performance and cost

5. Find non-obvious security weaknesses

Great engineers think beyond code. They develop patterns that help entire teams, select the right technologies, and plan both for success and failure scenarios.

Software creation involves complex tradeoffs: Do we prioritize speed or stability? Flexibility or simplicity? These decisions require both technical expertise and business knowledge.

The highest value engineers I work with spend more time thinking than typing. They consider: How will requirements evolve? What stress points might emerge? How will the system recover from failures?

As basic code generation becomes widely accessible, your value comes from understanding system interactions. The competitive edge will be with those who know why certain approaches succeed, where they might fail, and how to build resilient solutions.

Per the above, there are some things that AI without proper oversight can not do.

AI can, for instance, be great at passing simple prompts to create solutions to well known problems. If you use this prompt, for example:

Here’s the code you might get:

import json

def is_valid_json(input_string: str) -> bool:
    """
    Checks if the input string is valid JSON.

    Args:
        input_string (str): The JSON string to validate.

    Returns:
        bool: True if valid, False otherwise.
    """
    try:
        json.loads(input_string)
        return True
    except json.JSONDecodeError:
        return False

def main():
    user_input = input("Enter a JSON string to validate: ")
    if is_valid_json(user_input):
        print("✅ The input is valid JSON.")
    else:
        print("❌ The input is NOT valid JSON.")

if __name__ == "__main__":
    main()

Pretty basic.

I would not, though, ask an AI to build me a complex Forex Trading platform.

I’ll save you reading through all of the response, but suffice it to say that it was not able to do this for me.

I thought I’d break that down to a slightly smaller subset of a financial trading platform:

Now, I will admit, this got closer. Here was the result:

from fastapi import FastAPI, Request, HTTPException, Header
from pydantic import BaseModel, Field
from typing import Optional
import hmac
import hashlib
import logging

app = FastAPI()

# === CONFIGURATION ===
SHARED_SECRET = b"super_secure_shared_secret"  # Keep this secure

# === LOGGING SETUP ===
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ems.fix")

# === Pydantic Model for Execution Report (simplified) ===
class ExecutionReport(BaseModel):
    msg_type: str = Field(default="J", const=True)
    cl_ord_id: str  # Tag 11
    order_id: str   # Tag 37
    exec_id: str    # Tag 17
    exec_type: str  # Tag 150
    ord_status: str # Tag 39
    symbol: str     # Tag 55 (e.g., EUR/USD)
    side: str       # Tag 54 (1=Buy, 2=Sell)
    leaves_qty: float  # Tag 151
    cum_qty: float     # Tag 14
    avg_px: float      # Tag 6
    transact_time: str # Tag 60 (ISO8601 string)

# === Helper: HMAC Validator ===
def verify_signature(payload: bytes, signature: str) -> bool:
    computed = hmac.new(SHARED_SECRET, payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(computed, signature)

# === Endpoint ===
@app.post("/fix/execution-report")
async def receive_execution_report(
    request: Request,
    x_signature: Optional[str] = Header(None)
):
    body = await request.body()

    if not x_signature or not verify_signature(body, x_signature):
        logger.warning("Invalid or missing signature.")
        raise HTTPException(status_code=403, detail="Unauthorized")

    try:
        report_data = ExecutionReport.parse_raw(body)
        logger.info(f"Execution Report received: {report_data.dict()}")
        # ✅ Here, you'd handle order updates, booking, etc.
        return {"status": "ok", "message": "Execution Report received"}
    except Exception as e:
        logger.exception("Invalid payload format.")
        raise HTTPException(status_code=400, detail=f"Invalid FIX message format: {str(e)}")

But this does not come close to the final version. Even if you could use this code as a part of your trading system, unless you know how to build and architect and entire software system, where do you go from here? You can tell your clients that they can send you an “ExecutionReport”, but then what?

I’ve read and heard the arguments that you “just need to be better at prompting to build bigger systems”. But where does the benefit come from then? The complexity of financial trading systems is beyond comprehension. Prompting a system well enough, with enough information to make it fully featured, scalable, secure, and extensible (not to mention able to be debugged) would itself be a mammoth task. So where is the time being saved? Is it even possible?

I’m yet to see any proof anywhere that anyone has build such a complex system without the oversight of a human, and I’m not convinced we will see it at any point in the near future.

AI as a “Capability Multiplier”

These AI tools help magnify existing capabilities rather than replacing them. Skilled developers become far more productive, while less skilled ones generate problems more quickly.

Effective engineers use AI to:

• Handle basic implementation tasks

• Create initial project frameworks

• Compare different solution approaches

• Move past challenging problems

Meanwhile, less capable developers use AI to mask skill gaps, implementing solutions they neither understand nor can modify. When these implementations fail, they lack the knowledge to fix them independently.

This widens the skill gap. Top engineers leverage AI for mechanical tasks while focusing on higher-value thinking. Those using AI as a substitute for learning face limitations when working beyond the AI's knowledge boundaries.

A good example of something that AI is perfect for is translation logic:

Let’s say I have Python Dataclass representing an" “InternalUser”. I also have a Django ORM representation of the same entity. If I wanted to convert one to the other, I can just paste both representations in to ChatGPT and get it create me a conversion function. Notice that the conversion function also takes into account that the field names aren’t exact matches:

@dataclass
class InternalUser:
    id: str
    email: str
    hashed_password: str
    full_name: str
    role: UserRole
    status: AccountStatus
    created_at: datetime
    updated_at: datetime
    address: Optional[Address] = None
    preferences: Preferences = field(default_factory=Preferences)
    login_activity: LoginActivity = field(default_factory=LoginActivity)
    tags: List[str] = field(default_factory=list)
    notes: Optional[str] = None

class InternalUser(models.Model):
    id = models.CharField(primary_key=True, max_length=100)  # Use UUIDField if preferred
    email = models.EmailField(unique=True)
    hashedPassword = models.CharField(max_length=256)
    fullName = models.CharField(max_length=255)
    role = models.CharField(max_length=20, choices=UserRole.choices, default=UserRole.USER)
    status = models.CharField(max_length=20, choices=AccountStatus.choices, default=AccountStatus.PENDING)
    createdAt = models.DateTimeField(default=timezone.now)
    updatedAt = models.DateTimeField(auto_now=True)

    address = models.OneToOneField(Address, null=True, blank=True, on_delete=models.SET_NULL)
    preferences = models.OneToOneField(Preferences, on_delete=models.CASCADE)
    loginActivity = models.OneToOneField(LoginActivity, on_delete=models.CASCADE)

    tags = models.JSONField(default=list, blank=True)  # List of strings
    notes = models.TextField(null=True, blank=True)

    def __str__(self):
        return f"{self.fullName} ({self.email})"

from typing import Optional
from myapp.models import InternalUser as ORMInternalUser, Address as ORMAddress, Preferences as ORMPreferences, LoginActivity as ORMLoginActivity
from mydataclasses import InternalUser as DCInternalUser  # assuming your dataclass lives here


def convert_internal_user_to_model(dc_user: DCInternalUser) -> ORMInternalUser:
    # Create or update Address if provided
    orm_address: Optional[ORMAddress] = None
    if dc_user.address:
        orm_address = ORMAddress.objects.create(
            street=dc_user.address.street,
            city=dc_user.address.city,
            state=dc_user.address.state,
            postalCode=dc_user.address.postal_code,
            country=dc_user.address.country
        )

    # Create Preferences
    orm_preferences = ORMPreferences.objects.create(
        newsletterOptIn=dc_user.preferences.newsletter_opt_in,
        darkModeEnabled=dc_user.preferences.dark_mode_enabled,
        language=dc_user.preferences.language
    )

    # Create LoginActivity
    orm_login_activity = ORMLoginActivity.objects.create(
        lastLogin=dc_user.login_activity.last_login,
        loginAttempts=dc_user.login_activity.login_attempts,
        lastIpAddress=dc_user.login_activity.last_ip_address
    )

    # Create InternalUser
    orm_user = ORMInternalUser.objects.create(
        id=dc_user.id,
        email=dc_user.email,
        hashedPassword=dc_user.hashed_password,
        fullName=dc_user.full_name,
        role=dc_user.role.value,
        status=dc_user.status.value,
        createdAt=dc_user.created_at,
        updatedAt=dc_user.updated_at,
        address=orm_address,
        preferences=orm_preferences,
        loginActivity=orm_login_activity,
        tags=dc_user.tags,
        notes=dc_user.notes
    )

    return orm_user

This is a basic example, but extrapolate this out to an entire internal data model and you’ve saved hours and hours of code.

You can even then ask it to create you the function stubs of the tests:

import pytest
from datetime import datetime
from yourmodule.dataclasses import InternalUser, Address, Preferences, LoginActivity, UserRole, AccountStatus
from yourmodule.conversion import dataclass_to_django_internal_user


@pytest.mark.django_db
def test_internal_user_conversion_basic_fields():
    # Test that basic fields (email, name, etc.) are correctly copied
    pass


@pytest.mark.django_db
def test_internal_user_conversion_with_address():
    # Test that address fields are properly mapped to the ORM model
    pass


@pytest.mark.django_db
def test_internal_user_conversion_with_preferences():
    # Test preferences like dark mode, newsletter opt-in, and language
    pass


@pytest.mark.django_db
def test_internal_user_conversion_with_login_activity():
    # Test login attempts, last IP, and last login datetime
    pass


@pytest.mark.django_db
def test_internal_user_conversion_with_tags_and_notes():
    # Test tags list and optional notes field
    pass


@pytest.mark.django_db
def test_internal_user_conversion_with_missing_optional_fields():
    # Ensure None fields like address or lastLogin don’t break conversion
    pass


@pytest.mark.django_db
def test_internal_user_conversion_saves_correctly():
    # Save all related models and main InternalUser model and check database
    pass

Now, I’m not suggesting that you take these as is and don’t add your own thought in to each possible test scenario, but it’s a great start.

These pieces of “grunt work” were never what we paid the top engineers for. These were just the things that they had to do to get the job done. People didn’t enjoy these tasks. They weren’t fulfilling.

Critical Skills for the AI Era

As AI handles more coding tasks, successful engineers must develop strengths in areas where human judgment remains essential:

Systems thinking becomes the primary skill – understanding component interactions, identifying potential failures, and designing for future growth. This capability comes from experience, not prompting.

You should build expertise in infrastructure and deployment processes. Software that works in development but fails in production creates no value. So, learn about continuous integration, monitoring systems, and cloud platform capabilities.

You should also master API design – the interfaces between systems. Well-designed APIs enable team independence. Poor interfaces create bottlenecks affecting everyone.

Another key skill is being able to integrate security throughout the development process. A single oversight can result in breaches, damaging both customer trust and business standing.

Make sure you develop communication skills for both technical and non-technical audiences. You’ll need to explain complex decisions clearly across different stakeholder groups.

And study how AI tools function to understand their limitations and strengths, allowing you to use them most effectively.

For senior developers, mentoring becomes increasingly important. New engineers need guidance on responsible AI usage – knowing when to accept suggestions and when to question them.

The Path Forward

The software field is entering a significant transition. AI will generate more code more quickly, transforming development practices. This shift presents both opportunities and challenges.

The most valuable positions will go to those good at tasks machines cannot handle. These engineers will determine what to build, how to design it, and how to balance technical constraints with business objectives.

"Vibe coding" serves as a useful technique for specific needs – like quickly building standard components. But it fails as a comprehensive strategy for complex system development.

Skilled engineers will advance by delegating routine work to AI while addressing more challenging problems. Less skilled engineers will struggle as fundamental knowledge gaps become apparent.

With regards to learning how to use AI effectively, also use caution and judgement when following advice from people online. It’s still a fairly new field and changes constantly.

People online are giving away “free prompts” to generate code. These prompts may be great or may have problems. The prompts may have worked when they used them, but the AI models may have changed and maybe they’ll produce different results now. Be cautious and use your best judgement.

The future belongs to those who view AI as a collaborative tool rather than a replacement. Software development remains fundamentally human-driven, now supported by increasingly powerful assistance.

In his spare time, Ben writes his tech blog Just Another Tech Lead and runs a site on SEO, SmoothSEO.

Tools such as Swagger, JSON Schema, or OpenAPI are powerful, but they can be verbose, inflexible, or not conducive to reuse.

Well, I recently discovered TypeSpec. In this guide, I’ll show you how to take advantage of TypeSpec to create modern, maintainable, and well-documented REST APIs.

We'll take a look at:

• Prerequisites

• What is TypeSpec?

• Why use TypeSpec?

• How to Install and Configure TypeSpec

• TypeSpec Basic Syntax

• How to Create a REST API Model

• How to Build the API in Express and ASP.NET Core

• Best Practices for Structuring TypeSpec Projects and Components

• Conclusion

Prerequisites

Before we dive into using TypeSpec to document and model APIs, here are a few things you'll need to familiarize yourself with and/or have:

• Node.js (version 18 or higher)

• npm for dependency management

• Visual Studio Code (recommended to take advantage of the official TypeSpec extension). For an optimal experience, to create your project easily, it provides syntax highlighting, validation, autocompletion, navigation, and more.

• TypeSpec Extension in VS Code (You can install the extension via Visual Studio Marketplace)

• An understanding of how to use and create APIs

What is TypeSpec?

TypeSpec is an open-source declarative language, developed by Microsoft, designed to describe APIs in an explicit, reusable, scalable, and standards-based way. It’s designed to model REST, gRPC, GraphQL, and other types of APIs, and offers a modern syntax close to TypeScript.

It can automatically generate:

• OpenAPI, JSON Schema, or Protobuf specifications

• server and client code

• API documentation

• and other interface-related artifacts

TypeSpec isn't just a language – it's an API design platform that favors abstraction, encourages code reuse, and integrates with modern tools like Visual Studio Code via a dedicated extension. You can install the extension via the VS Code Visual Studio Marketplace.

Why use TypeSpec?

Before diving into the code, let's take a minute to understand the TypeSpec philosophy. Microsoft uses TypeSpec internally to deliver high-quality API services to millions of customers, across tens of thousands of endpoints, while ensuring code quality, governance, and scalability.

Unlike generators such as Swagger, Codegen, or Postman, which start from an OpenAPI file to generate code, TypeSpec does the opposite: you first write your API design in a DSL (Domain Specific Language), then generate everything you need.

TypeSpec has been designed to meet the major challenges of large-scale API design and governance:

• Simplification: clear, concise syntax to focus on business logic.

• Reusability: encapsulates types, request/response models, and directives in modular components.

• Productivity: automatically generates the necessary resources from a single source definition.

• Consistency: maintains compliance with internal standards thanks to shared libraries.

• Interoperability: integrates with the OpenAPI ecosystem and supports multi-format generation.

• Scalability: designed to handle thousands of endpoints like those used by Microsoft Azure.

Let's take a look at how to install and configure the development environment

How to Install and Configure TypeSpec

Before you can start writing your first API with TypeSpec, you need to set up your development environment. Here's how to install TypeSpec on your machine.

Requirements:

• Node.js (version 18 or higher)

• npm for dependency management

• Visual Studio Code (recommended to take advantage of the official TypeSpec extension). For an optimal experience, it provides syntax highlighting, validation, autocompletion, navigation, and more.

TypeSpec CLI global installation:

npm install -g @typespec/compiler

How to Create a TypeSpec Project

The easiest way to create a project is to use Visual Studio Code via the TypeSpec extension you've installed (if you're not comfortable with the command line (CMD)).

Create a folder containing the project and open it with Visual Studio Code. Then click on the View tab, and next on Comment Palette .

In the search bar that appears, enter TypeSpec: Create TypeSpec Project.

Follow the quick selections to select the root folder of the project you've just created. Then choose the Template – for our case this will be Generic REST API – and enter the project name. Leave the emitter OpenAPI 3.1 document (3.1 is the current version at the time of writing) selected by default. This will put us @typespec/http@typespec/openapi3. Finally, wait for the project configuration to finish.

You should have a basic TypeSpec project configuration with a structure that looks like this:

• node_modules/: Directory where npm installs project dependencies.

• main.tsp: the entry point for your TypeSpec build. This file generally contains the main definitions of your models, services, and operations.

• package.json: Contains project metadata, including dependencies, scripts, and other project-related information.

• tspconfig.yaml: TypeSpec compiler configuration file, specifying options and parameters for the generation process.

You can also run tsp compile . to compile the project, but it's better to run tsp compile . --watch to automatically compile changes during development each time you save.

Once the project has been compiled, you'll see the tsp-output and schema folders generated and a file added openai.yaml.

• tsp-output/: Directory where the TypeSpec compiler generates files.

• openapi.yaml: OpenAPI specification file generated for your API, detailing API endpoints, templates, and operations. Output may vary depending on the target format specified in the tspconfig.yaml file.

emit:
  - "@typespec/openapi3"
options:
  "@typespec/openapi3":
    emitter-output-dir: "{output-dir}/schema"
    openapi-versions:
      - 3.1.0

Thanks to this configuration of the tspconfig.yaml file, one of TypeSpec's major assets is its ability to automatically generate OpenAPI specifications from clear, typed, and modular source code. This means you can write your API as you would in TypeScript (or a well-structured DSL), and get output in .yaml files compatible with the whole OpenAPI ecosystem: Swagger UI, Postman, Redoc, and so on.

In the next section, we'll look at the basic syntax of TypeSpec.

TypeSpec Basic Syntax

Now that you've got a clear idea of what TypeSpec is and what its benefits are in the world of API design, it's time to get to the heart of the matter: the basic syntax.

TypeSpec is a declarative language, inspired by TypeScript, that lets you model the resources, routes, data structures, and behaviors of an API in an explicit, readable, and modular way. Its syntax is based on simple keywords and clear file organization, making it easy to learn yet powerful.

Language Basics

Here's a very simple example of defining a model with TypeSpec:

model Book {
  id: string;
  title: string;
  author: string;
}

This block defines a Book resource with three typed fields. The model keyword is used to describe the JSON objects manipulated by the API. It is equivalent to schemas in JSON Schema or type definitions in OpenAPI.

Defining an HTTP operation

TypeSpec lets you bind operations to models using the @route keyword. Here's a minimal example of an endpoint:

@route("/books")
op listBooks(): Book[];

This syntax declares a REST operation that returns a list of books. @route indicates the URL path, op introduces an operation, and Book[] is the return type.

You can also define path, query, or body parameters very easily.

@route("/books/{id}")
op getBook(@path id: string): Book;

In this example, we declare that id is a URL parameter (path parameter).

Fundamental Concepts

model Defining data structures

A model represents an API entity, like a JSON object. Models are the basis of your information exchanges.

model User {
  id: string;
  email: string;
  age?: int32;
}

interface Group operations

An interface groups together a set of logically linked operations. This is useful for structuring large API sets.

interface BookOperations {
  @get op listBooks(): Book[];
  @get op getBook(@path id: string): Book;
}

service Entry point of the API

A service defines publicly exposed interfaces, their version, and the basic path.

@service({ title: "Book API", version: "1.0.0" })
namespace BookApi {
  interface BookOperations;
}

Import and Organize Your Code with Namespaces

TypeSpec provides clear organization through namespaces, similar to modules or packages.

namespace CommonModels {
  model Error {
    message: string;
  }
}

Then you can import them into another file like this:

import CommonModels from "./common.tsp";

Complete Example of a REST Service

Let's take a complete example of a REST service in TypeSpec.

@service({ title: "Book Service", version: "1.0.0" })

@route("/books")

namespace BookService {

  model Book {
    id: string;
    title: string;
    author: string;
    publishedYear?: int32;
  }

  @get()
  op listBooks(): Book[];

  @post()
  op createBook(@body book: Book): Book;

  @get("/{id}")
  op getBook(@path id: string): Book;

  @put("/{id}")
  op updateBook(@path id: string, @body book: Book): Book;

  @delete("/{id}")
  op deleteBook(@path id: string): void;
}

Here’s what’s going on:

• @service({ title, version }): Defines service metadata (name, version), useful for generated documentation (for example, Swagger UI).

• @route("/books"): Defines the basic path for all operations of this API.

• namespace BookService { ... }: Encapsulates all models and operations linked to this service under a single logical name.

Next come the operations:

• @get() op listBooks(): Endpoint GET /books qui retourne un tableau de livres.

• @post() op createBook(): Endpoint POST /books which accepts a Book object in the request body (@body) and returns the created book.

• @get("/{id}"): Endpoint GET /books/{id} which retrieves a book via its identifier (@path).

• @put("/{id}"): Endpoint PUT /books/{id} which updates a book's data.

• @delete("/{id}"): Deletes a book via its id. The void type means that no data is returned.

With just a few lines, you get a complete, well-organized, easily readable REST service, ready to be automatically converted into OpenAPI documentation, a client SDK, or backend code.

Add Validation Annotations

TypeSpec makes it easy to add validation annotations to your models using:

model Book {
  id: string;
  title: string @minLength(3);
  author: string @minLength(3);
  publishedYear?: int32 @minValue(1800);
}

This adds validation rules directly to the schema, which will be taken into account during OpenAPI generation.

Comparison with Other Tools (OpenAPI / Swagger)

So you might wonder – why should you use TypeSpec rather than writing directly in OpenAPI?

Let's take the example of OpenAPI 3 (YAML):

paths:
  /books:
    get:
      summary: Get list of books
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Book'
    post:
      summary: Create a new book
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Book'
      responses:
        '201':
          description: Created
  /books/{id}:
    get:
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: string
      responses:
        '200':
          description: OK
    put:
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: string
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Book'
    delete:
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: string
components:
  schemas:
    Book:
      type: object
      properties:
        id:
          type: string
        title:
          type: string
        author:
          type: string
        publishedYear:
          type: integer

As you can see, the OpenAPI definition is much more verbose. Relationships between paths, methods, schemas, and parameters are scattered, which complicates reading and maintenance. Also, it's less typed, given that OpenAPI remains YAML (or JSON), without the typing security or modularity of a real language.

Why TypeSpec is useful here

With TypeSpec, everything is centralized in a declarative, modular, typed, and intuitive format.

• Greater legibility: less noise, more intent.

• Reusability: you can create modular components and share them between projects.

• Productivity: you write less code and generate more (OpenAPI, client, server, doc).

• Consistency: errors are detected early thanks to strong typing.

Note: TypeSpec doesn't replace OpenAPI, but complements it: you write to TypeSpec, then automatically generate OpenAPI files, SDKs, specs and so on. It gives you a source language for accurately describing your API.

In the next section, we'll look at how to create a REST API template.

How to Create a REST API Model

To deepen our understanding of REST API creation with TypeSpec, let's continue with the example of managing books. In this example, we'll create a Book model, define a service to manage the books, and add validations to ensure that the data respects the right constraints.

Define a Data Model for Book

First, we'll define a data model for the Book resource. A book can have the following properties:

• id: A unique identifier for the book.

• title: The title of the book.

• author: The author of the book.

• publicationYear: The book's year of publication.

• isbn: The book's ISBN number.

Book model in TypeSpec

model Book {
  id: integer;
  @minLength(1)
  title: string;
  @minLength(1)
  author: string;
  publicationYear: integer;
  @pattern("^\\d{3}-\\d{1,5}-\\d{1,7}-\\d{1,7}-\\d{1}$")
  isbn: string;
}

• id: Unique book identifier (integer type).

• title and author: Character strings representing the book's title and author, validated by @minLength(1) to ensure they are not empty.

• publicationYear: The book's year of publication (integer type).

• isbn: The book's ISBN number, validated with a regular expression that matches the standard format of an ISBN.

Define a REST Service to Manage Books

Now that we have a Book model, we'll create a service to manage CRUD operations on this resource. This service will contain methods for retrieving a book by its identifier, creating a new book, updating an existing book, and deleting a book.

BooksService service in TypeSpec

service BooksService {

  @get("/books/{id}")
  getBook(id: integer): Book;

  @post("/books")
  createBook(book: Book): Book;

  @put("/books/{id}")
  updateBook(id: integer, book: Book): Book;

  @delete("/books/{id}")
  deleteBook(id: integer): void;
}

The BooksService contains four methods for performing actions on books:

• @get("/books/{id}"): Method for retrieving a book by its id.

• @post("/books"): Method for creating a new book.

• @put("/books/{id}"): Method for updating an existing book by its id.

• @delete("/books/{id}"): Method for deleting a book based on its id.

These methods use HTTP annotations to indicate the type of operation they perform (GET, POST, PUT, DELETE).

Add Additional Validations for the Book Model

As in the previous example for users, we can add additional validations on Book template properties.

Example of validation on publicationYear and isbn

model Book {
  id: integer;
  @minLength(1)
  title: string;
  @minLength(1)
  author: string;
  @minValue(1000)
  publicationYear: integer;
  @pattern("^\\d{3}-\\d{1,5}-\\d{1,7}-\\d{1,7}-\\d{1}$")
  isbn: string;
}

• @minValue(1000) guarantees that the year of publication is greater than or equal to 1000.

• Validation of the isbn remains the same, using a regular expression to validate a standard ISBN format.

A Complete Service for Managing Books

Now that we have the Book model and the necessary validations, here's a complete service for managing books, with all the essential operations.

Complete BooksService in TypeSpec

model Book {
  id: integer;
  @minLength(1)
  title: string;
  @minLength(1)
  author: string;
  @minValue(1000)
  publicationYear: integer;
  @pattern("^\\d{3}-\\d{1,5}-\\d{1,7}-\\d{1,7}-\\d{1}$")
  isbn: string;
}

service BooksService {
  @get("/books/{id}")
  getBook(id: integer): Book;

  @post("/books")
  createBook(book: Book): Book;

  @put("/books/{id}")
  updateBook(id: integer, book: Book): Book;

  @delete("/books/{id}")
  deleteBook(id: integer): void;
}

• The Book model defines properties and validations for a book.

• The BooksService provides endpoints for retrieving, creating, updating, and deleting a book.

• Each service method is correctly annotated with the corresponding HTTP verbs (GET, POST, PUT, DELETE).

And here’s a summary of everything we’ve done:

• We created a Book model with properties such as title, author, year of publication, and ISBN number.

• We defined a BooksService to provide CRUD operations on books.

• We added validations to ensure that the data respected specified constraints (for example, ISBN and year of publication).

• We designed a complete REST API to manage books with TypeSpec, using a minimum amount of code and staying true to standards.

This example shows just how quickly and efficiently TypeSpec can be used to model a REST API, while ensuring a clear structure and robust validations.

How to Build the API in Express and ASP.NET Core

Now that we've defined a book management REST service with TypeSpec, let's see how we'd implement this same API using two popular frameworks:

• ExpressJS (Node.js / TypeScript)

• ASP.NET Core (C#)

This will allow us to better compare TypeSpec's conciseness and readability with traditional implementations.

Manual implementation with ExpressJS (Node.js / TypeScript):

//server.ts
import express from 'express';

const app = express();
app.use(express.json());

interface Book {
  id: number;
  title: string;
  author: string;
  publicationYear: number;
  isbn: string;
}

const books: Book[] = [];

// GET /books/:id
app.get('/books/:id', (req, res) => {
  const id = parseInt(req.params.id);
  const book = books.find(b => b.id === id);
  if (!book) return res.status(404).send({ message: 'Book not found' });
  res.send(book);
});

// POST /books
app.post('/books', (req, res) => {
  const newBook: Book = req.body;
  books.push(newBook);
  res.status(201).send(newBook);
});

// PUT /books/:id
app.put('/books/:id', (req, res) => {
  const id = parseInt(req.params.id);
  const index = books.findIndex(b => b.id === id);
  if (index === -1) return res.status(404).send({ message: 'Book not found' });

  books[index] = req.body;
  res.send(books[index]);
});

// DELETE /books/:id
app.delete('/books/:id', (req, res) => {
  const id = parseInt(req.params.id);
  const index = books.findIndex(b => b.id === id);
  if (index === -1) return res.status(404).send({ message: 'Book not found' });

  books.splice(index, 1);
  res.status(204).send();
});

app.listen(3000, () => {
  console.log('Server is running on port 3000');
});

Observations:

• A lot of repetitive logic.

• No automatic validation.

• Routes must be maintained manually.

• No automatically generated API documentation.

Manual implementation with ASP.NET Core (C#):

// Book.cs
public class Book
{
    public int Id { get; set; }

    [Required]
    public string Title { get; set; } = string.Empty;

    [Required]
    public string Author { get; set; } = string.Empty;

    [Range(1000, int.MaxValue)]
    public int PublicationYear { get; set; }

    [RegularExpression(@"^\d{3}-\d{1,5}-\d{1,7}-\d{1,7}-\d{1}$")]
    public string Isbn { get; set; } = string.Empty;
}

// BooksController.cs
[ApiController]
[Route("books")]
public class BooksController : ControllerBase
{
    private static readonly List<Book> books = new();

    [HttpGet("{id}")]
    public IActionResult GetBook(int id)
    {
        var book = books.FirstOrDefault(b => b.Id == id);
        if (book == null) return NotFound("Book not found");
        return Ok(book);
    }

    [HttpPost]
    public IActionResult CreateBook([FromBody] Book book)
    {
        books.Add(book);
        return CreatedAtAction(nameof(GetBook), new { id = book.Id }, book);
    }

    [HttpPut("{id}")]
    public IActionResult UpdateBook(int id, [FromBody] Book updatedBook)
    {
        var index = books.FindIndex(b => b.Id == id);
        if (index == -1) return NotFound("Book not found");

        books[index] = updatedBook;
        return Ok(updatedBook);
    }

    [HttpDelete("{id}")]
    public IActionResult DeleteBook(int id)
    {
        var book = books.FirstOrDefault(b => b.Id == id);
        if (book == null) return NotFound("Book not found");

        books.Remove(book);
        return NoContent();
    }
}

Observations:

• More formal and structured than Express, thanks to C# annotations ([HttpPost], [Required], and so on).

• Validation is handled automatically via Data Annotations.

• Once again, no automatic OpenAPI generation or SDK client without additional configuration.

Comparison with TypeSpec:

Best Practices for Structuring TypeSpec Projects and Components

When you start writing API definitions in TypeSpec, it's easy to put everything in a single file. But as with any software project, as the application grows, a good structure becomes essential to guarantee the readability, reusability and maintainability of the code.

Here's a set of best practices I strongly recommend:

Organize by Functional Area

Use namespaces to group models, interfaces, and operations by business domain: book, user, auth, payment, and so on.

namespace MyApi.Books;

Create a /books folder with the following files:

src/
├── books/
│   ├── models.tsp
│   ├── routes.tsp
│   └── service.tsp

This ensures a clear separation of responsibilities, just like in a well-structured Node.js project.

A Single main.tsp Entry Point

This is the main file that orchestrates:

// main.tsp
import "./books/service.tsp";
import "./users/service.tsp";
import "./auth/service.tsp";

This allows you to compile the entire project from a single point.

Create Reusable Components

Define common models and types in a shared file. Example:

// common/models.tsp
model ErrorResponse {
  code: string;
  message: string;
}

@defaultResponse
op Error(): ErrorResponse;

Then import them into your other files:

import "../common/models.tsp";

This is handy for centralizing errors, standard answers, pagination types, and so on.

Use Decorators to Enrich Your Components

Decorators such as @doc, @minLength, @server, @route or @tag can be used to generate valid, documented APIs without any extra effort:

@route("/books")
@doc("Get all books")
op listBooks(): Book[];

A well-annotated API is one that is ready for automatic generation of documentation or clients.

Define Servers in the Right Place

Add your @server directive to a service.tsp or global api.tsp file:

@server("Production", "https://api.mysite.com")
@server("Staging", "https://staging.mysite.com")

This allows you to target different environments without duplicating definitions.

Validate Regularly

Integrate tsp compile into your CI/CD to ensure that your definitions are always valid. Example with an npm script:

npm run tsp compile src/main.tsp --emit=./dist

This avoids last-minute errors and guarantees the consistency of your API over time.

Example of a recommended complete structure:

project-root/
├── src/
│   ├── books/
│   │   ├── models.tsp
│   │   ├── routes.tsp
│   │   └── service.tsp
│   ├── users/
│   │   ├── models.tsp
│   │   └── service.tsp
│   ├── common/
│   │   └── models.tsp
│   └── main.tsp
├── tspconfig.yaml
├── package.json
└── README.md

In summary:

Conclusion

TypeSpec represents a real evolution in the way we design, document and maintain APIs. By adopting a declarative, modular, and typed approach, it simplifies the definition of APIs while enhancing their quality, readability, and consistency on a large scale.

Whether you're a front-end developer consuming APIs, a software architect looking to standardize your team's practices, or a technical documentation enthusiast, TypeSpec offers you a robust, modern, and extensible solution.

The TypeSpec ecosystem is still young but very promising, supported by Microsoft and used internally on a large scale. So now's the time to start exploring and adopting it for your projects.

Ressources

1. TypeSpec official website
   https://typespec.io
   Full documentation, guides, syntax references and APIs.

2. TypeSpec GitHub repository (Microsoft)
   https://github.com/microsoft/typespec
   Source code, examples and community discussions.

3. Playground TypeSpec (essayer dans le navigateur)
   https://typespec.io/playground
   Quickly test your models without installing anything.

4. TypeSpec documentation — Microsoft Learn
   https://learn.microsoft.com/en-us/azure/developer/typespec/overview
   Learn how to use TypeSpec to create consistent, high-quality APIs efficiently and integrate them seamlessly with existing toolchains.

5. OpenAPI Specification
   https://swagger.io/specification
   To compare with current API description standards.

6. TypeSpec 101 by Mario Guerra Product Manager for TypeSpec at Microsoft
   https://www.youtube.com/playlist?list=PLYWCCsom5Txglkl_I1XvwzrzM5G3SuVsR
   A tutorial series, hosted by Mario Guerra, TypeSpec product manager at Microsoft, will guide you through the process of building a REST API using TypeSpec, and generating an OpenAPI specification from our code.

7. APIs at Scale with TypeSpec
   https://youtu.be/yfCYrKaojDo
   A talk given by Mandy Whaley from Microsoft at the 2024 Austin API Summit in Austin, Texas.

Thanks for reading. You can find me on LinkedIn, and follow me on all socials @AdalbertPungu.

A buffer overflow happens when a program writes more data into a buffer than it was allocated, leading to memory corruption, crashes, or even security vulnerabilities. A buffer corruption occurs when unintended modifications overwrite unread data or modify memory in unexpected ways.

In safety-critical systems like cars, medical devices, and spacecraft, buffer overflows can cause life-threatening failures. Unlike simple software bugs, buffer overflows are unpredictable and depend on the state of the system, making them difficult to diagnose and debug.

To prevent these issues, it's important to understand how buffer overflows and corruptions occur, and how to detect and fix them.

Article Scope

In this article, you will learn:

1. What buffers, buffer overflows, and corruptions are. I’ll give you a beginner-friendly explanation with real-world examples.

2. How to debug buffer overflows. You’ll learn how to use tools like GDB, LLDB, and memory maps to find memory corruption.

3. How to prevent buffer overflows. We’ll cover some best practices like input validation, safe memory handling, and defensive programming.

I’ll also show you some hands-on code examples – simple C programs that demonstrate buffer overflow issues and how to fix them.

What this article doesn’t cover:

1. Security exploits and hacking techniques. We’ll focus on preventing accidental overflows, not hacking-related buffer overflows.

2. Operating system-specific issues. This guide is for embedded systems, not general-purpose computers or servers.

3. Advanced RTOS memory management. While we discuss interrupt-driven overflows, we won’t dive deep into real-time operating system (RTOS) concepts.

Now that you know what this article covers (and what it doesn’t), let’s go over the skills that will help you get the most out of it.

Prerequisites

This article is designed for developers who have some experience with C programming and want to understand how to debug and prevent buffer overflows in embedded systems. Still, beginners can follow along, as I’ll explain key concepts in a clear and structured way.

Before reading, it helps if you know:

1. Basic C programming.

2. How memory works – the difference between stack, heap, and global variables.

3. Basic debugging concepts – if you’ve used a debugger like GDB or LLDB, that’s a plus, but not required.

4. What embedded systems are – a basic idea of how microcontrollers store and manage memory.

Even if you’re not familiar with these topics, this guide will walk you through them in an easy-to-understand way.

Before you dive into buffer overflows, debugging, and prevention, let’s take a step back and understand what a buffer is and why it’s important in embedded systems. Buffers play a crucial role in managing data flow between hardware and software but when handled incorrectly, they can lead to serious software failures.

Table of Contents

• What is a Buffer, and How Does it Work?

• What is a Buffer Overflow?

• Common Causes of Buffer Overflows and Corruption

• Consequences of Buffer Overflows

• How to Debug Buffer Overflows

• How to Prevent Buffer Overflows

• Conclusion

What is a Buffer, and How Does it Work?

A buffer is a contiguous block of memory used to temporarily store data before it is processed. Buffers are commonly used in two scenarios:

1. Data accumulation: When the system needs to collect a certain amount of data before processing.

2. Rate matching: When the data producer generates data faster than the data consumer can process it.

Buffers are typically implemented as arrays in C, where elements are indexed from 0 to N-1 (where N is the buffer size).

Let’s look at an example of a buffer in a sensor system.

Consider a system with a sensor task that generates data at 400 Hz (400 samples per second or 1 sample every 2.5 ms). But the data processor (consumer) operates at only 100 Hz (100 samples per second or 1 sample every 10 ms). Since the consumer task is slower than the producer, we need a buffer to store incoming data until it is processed.

To determine the buffer size, we calculate:

Buffer Size = Time to consume 1 sample / Time to generate 1 sample = 10 ms/ 2.5 ms = 4

This means the buffer must hold at least 4 samples at a time to avoid data loss.

Once the buffer reaches capacity, there are several strategies to decide which data gets passed to the consumer task:

1. Max/min sampling: Use the maximum or minimum value in the buffer.

2. Averaging: Compute the average of all values in the buffer.

3. Random access: Pick a sample from a specific location (for example, the most recent or the first).

In real-world applications, it’s beneficial to use circular buffers or double buffering to prevent data corruption.

• Circular buffer approach: A circular buffer (also called a ring buffer) continuously wraps around when it reaches the end, ensuring old data is overwritten safely without exceeding memory boundaries. The buffer size should be multiplied by 2 (4 × 2 = 8) to hold 8 samples. This allows the consumer task to process 4 samples while the next 4 samples are being filled, preventing data overwrites.

• Double buffer approach: Double buffering is useful when data loss is unacceptable. It allows continuous data capture while the processor is busy handling previous data. A second buffer of the same size is added. When the first buffer is full, the write pointer switches to the second buffer, allowing the consumer task to process data from the first buffer while the second buffer is being filled. This prevents data overwrites and ensures a continuous data flow.

Buffers help manage data efficiently, but what happens when they are mismanaged? This is where buffer overflows and corruptions come into play.

What is a Buffer Overflow?

A buffer overflow occurs when a program writes more data into a buffer than it was allocated, causing unintended memory corruption. This can lead to unpredictable behavior, ranging from minor bugs to critical system failures.

To understand buffer overflow, let's use a simple analogy. Imagine a jug with a tap near the bottom. The jug represents a buffer, while the tap controls how much liquid (data) is consumed.

The jug is designed to hold a fixed amount of liquid. As long as water flows into the jug at the same rate or slower than it flows out, everything works fine. But if water flows in faster than it flows out, the jug will eventually overflow.

Similarly, in software, if data enters a buffer faster than it is processed, it exceeds the allocated memory space, causing a buffer overflow. In the case of a circular buffer, this can cause the write pointer to wrap around and overwrite unread data, leading to buffer corruption.

Buffer Overflows in Software

Unlike the jug, where water simply spills over, a buffer overflow in software overwrites adjacent memory locations. This can cause a variety of hard-to-diagnose issues, including:

1. Corrupting other data stored nearby.

2. Altering program execution, leading to crashes.

3. Security vulnerabilities, where attackers exploit overflows to inject malicious code.

When a buffer overflow occurs, data can overwrite variables, function pointers, or even return addresses, depending on where the buffer is allocated.

Buffer overflows can occur in different memory regions:

1. Buffer overflows in global/static memory (.bss / .data sections)

   • These occur when global or static variables exceed their allocated size.

   • The overflow can corrupt adjacent variables, leading to unexpected behavior in other modules.

   • Debugging is easier because memory addresses are fixed at compile time unless the compiler optimizes them. Map files provide a memory layout of variables during the compilation and linking.

2. Stack-based buffer overflow (more predictable, easier to debug):

   • Happens when a buffer is allocated in the stack (for example, local variables inside functions).

   • Overflowing the stack can affect adjacent local variables or return addresses, potentially crashing the program.

   • In embedded systems with small stack sizes, this often leads to a crash or execution of unintended code.

3. Heap-based buffer overflow (harder to debug):

   • Happens when a buffer is dynamically allocated in the heap (for example, using malloc() in C).

   • Overflowing a heap buffer can corrupt adjacent dynamically allocated objects or heap management structures.

   • Debugging is harder because heap memory is allocated dynamically at runtime, causing memory locations to vary.

Buffer Overflow vs Buffer Corruption

Buffer overflow and buffer corruption are of course related, but refer to different situations.

A buffer overflow happens when data is written beyond the allocated buffer size, leading to memory corruption, unpredictable behavior, or system crashes.

A buffer corruption happens when unintended data modifications result in unexpected software failures, even if the write remains within buffer boundaries.

Both issues typically result from poor write pointer management, lack of boundary checks, and unexpected system behavior.

Now that we've covered what a buffer overflow is and how it can overwrite memory, let’s take a closer look at how these issues affect embedded systems.

In the next section, we’ll explore how buffer overflows and corruption happen in real-world embedded systems and break down common causes, including pointer mismanagement and boundary violations.

Common Causes of Buffer Overflows and Corruption

Embedded systems use buffers to store data from sensors, communication interfaces (like UART (Universal Asynchronous Receiver-Transmitter), SPI (Serial Peripheral Interface), I2C (Inter-integrated Circuit), and real-time tasks. These buffers are often statically allocated to avoid memory fragmentation, and many implementations use circular (ring) buffers to efficiently handle continuous data streams.

Here are three common scenarios where buffer overflows or corruptions occur in embedded systems:

Writing Data Larger Than the Available Space

Issue: The software writes incoming data to the buffer without checking if there is enough space.

Example: Imagine a 100-byte buffer to store sensor data. The buffer receives variable-sized packets. If an incoming packet is larger than the remaining space, it will overwrite adjacent memory, leading to corruption.

So why does this happen?

• Some embedded designs increment the write pointer after copying data, making it too late to prevent overflow.

• Many low-level memory functions (memcpy, strcpy, etc.) do not check buffer boundaries, leading to unintended writes.

• Without proper bound checking, a large write can exceed the buffer size and corrupt nearby memory.

Here’s a code sample to demonstrate buffer overflow in a .bss / .data section:

  #include <stdint.h>
  #include <stdio.h>
  #include <string.h>

  #define BUFFER_SIZE 300

  static uint16_t sample_count = 0;
  static uint8_t buffer[BUFFER_SIZE] = {0};

  // Function to simulate a buffer overflow scenario
  void updateBufferWithData(uint8_t *data, uint16_t size)
  {
      // Simulating a buffer overflow: No boundary check!
      printf("Attempting to write %d bytes at position %d...\n", size, sample_count);

      // Deliberate buffer overflow for demonstration
      if (sample_count + size > BUFFER_SIZE)
      {
          printf("WARNING: Buffer Overflow Occurred! Writing beyond allocated memory!\n");
      }

      // Copy data (unsafe, can cause overflow)
      memcpy(&buffer[sample_count], data, size);

      // Increment sample count (incorrectly, leading to wraparound issues)
      sample_count += size;
  }

  int main()
  {   
      // Save 1 byte to buffer
      uint8_t data_to_buffer = 10;
      updateBufferWithData(&data_to_buffer, 1);

      // Save an array of 20 bytes to buffer
      uint8_t data_to_buffer_1[20] = {5};
      updateBufferWithData(data_to_buffer_1, sizeof(data_to_buffer_1));

      // Intentional buffer overflow: Save an array of 50 x 8 bytes (400 bytes)
      uint64_t data_to_buffer_2[50] = {7};
      updateBufferWithData((uint8_t*)data_to_buffer_2, sizeof(data_to_buffer_2));

      return 0;
  }

Interrupt-Driven Overflows (Real-time Systems)

Issue: The interrupt service routine (ISR) may write data faster than the main task can process, leading to buffer corruption or buffer overflow if the write pointer is not properly managed.

Example: Imagine a sensor ISR that writes incoming data into a buffer every time a new reading arrives. Meanwhile, a low-priority processing task reads and processes the data.

What can go wrong?

• If the ISR triggers too frequently (due to a misbehaving sensor or high interrupt priority), the buffer may fill up faster than the processing task can keep up.

• This can result in one of two failures:

  1. Buffer Corruption: The ISR overwrites unread data, leading to loss of information.

  2. Buffer Overflow: The ISR exceeds buffer boundaries, causing memory corruption or system crashes.

So why does this happen?

• In real-time embedded systems, ISR execution preempts lower-priority tasks.

• If the processing task doesn't not get enough CPU time, the buffer may become overwritten or overflow beyond its allocated scope.

System State Changes & Buffer Corruption

Issue: The system may unexpectedly reset, enter low-power mode, or changes operating state, leaving the buffer write pointers in an inconsistent state. This can result in buffer corruption (stale or incorrect data) or buffer overflow (writing past the buffer’s limits.

Example Scenarios:

1. Low-power wake-up issue (Buffer Overflow risk): Some embedded systems enter deep sleep to conserve energy. Upon waking up, if the buffer write pointer is not correctly reinitialized, it may point outside buffer boundaries, leading to buffer overflow and unintended memory corruption.

2. Unexpected mode transitions: If a sensor task is writing data and the system suddenly switches modes, the buffer states and pointers may not be cleaned up. The next time the sensor task runs, it may continue writing without clearing previous data. This can cause undefined behavior due to presence of stale data.

Now that you understand how buffer overflows and corruptions happen, let’s examine their consequences in embedded systems ranging from incorrect sensor readings to complete system failures, making debugging and prevention critical.

Consequences of Buffer Overflows

Buffer overflows can be catastrophic in embedded systems, leading to system crashes, data corruption, and unpredictable behavior. Unlike general-purpose computers, many embedded devices lack memory protection, making them particularly vulnerable to buffer overflows.

A buffer overflow can corrupt two critical types of memory:

1. Data Variables Corruption

A buffer overflow can overwrite data variables, corrupting the inputs for other software modules. This can cause unexpected behavior or even system crashes if critical parameters are modified.

For example, a buffer overflow could accidentally overwrite a sensor calibration value stored in memory. As a result, the system would start using incorrect sensor readings, leading to faulty operation and potentially unsafe conditions.

2. Function Pointer Corruption

In embedded systems, function pointers are often used for interrupt handlers, callback functions, and RTOS task scheduling. If a buffer overflow corrupts a function pointer, the system may execute unintended instructions, leading to a crash or unexpected behavior.

As an example, a function pointer controlling motor speed regulation could be overwritten. Instead of executing the correct function, the system would jump to a random memory address, causing a system fault or erratic motor behavior.

Buffer overflows are among the hardest bugs to identify and fix because their effects depend on which data is corrupted and the values it contains. A buffer overflow can affect memory in different ways:

• If a buffer overflow corrupts unused memory, the system may seem fine during testing, making the issue harder to detect.

• if a buffer overflow alters critical data variables, it can cause hidden logic errors that cause unpredictable behavior.

• If a buffer overflow corrupts function pointers, it may crash immediately, making the problem easier to identify.

During development, if tests focus only on detecting crashes, they may overlook silent memory corruption caused by a buffer overflow. In real-world deployments, new use cases not covered in testing can trigger previously undetected buffer overflow issues, leading to unpredictable failures.

Buffer overflows can cause a chain reaction, where one overflow leads to another overflow or buffer corruption, resulting in widespread system failures. So how does this happen?

1. A buffer overflow corrupts a critical variable (for example, a timer interval).

2. The corrupted variable disrupts another module (for example, triggers the timer interrupt too frequently, causing it to push more data into a buffer than intended.).

3. This increased interrupt frequency forces a sensor task to write data faster than intended, eventually causing another buffer overflow or corruption by overwriting unread data.

This chain reaction can spread across multiple software modules, making debugging nearly impossible. In real-word applications, buffer overflows in embedded systems can be life-threatening:

• In cars: A buffer overflow in an ECU (Electronic Control Unit) could cause brake failure or unintended acceleration.

• In a spacecraft: A memory corruption issue could disable navigation systems, leading to mission failure.

Now that we’ve seen how buffer overflows can corrupt memory, disrupt system behavior, and even cause critical failures, the next step is understanding how to detect and fix them before they lead to serious issues.

How to Debug Buffer Overflows

Debugging buffer overflows in embedded systems can be complex, as their effects range from immediate crashes to silent data corruption, making them difficult to trace. A buffer overflow can cause either:

1. A system crash, which is easier to detect since it halts execution or forces a system reboot.

2. Unexpected behavior, which is much harder to debug as it requires tracing how corrupted data affects different modules.

This section focuses on embedded system debugging techniques using memory map files, debuggers (GDB/LLDB), and a structured debugging approach. Let’s look into the debuggers and memory map files.

Memory Map File (.map file)

A memory map file is generated during the linking process. It provides a memory layout of global/static variables, function addresses, and heap/stack locations. It provides a memory layout of Flash and RAM, including:

• Text section (.text): Stores executable code.

• Read-only section (.rodata): Stores constants and string literals.

• BSS section (.bss): Stores uninitialized global and static variables.

• Data section (.data): Stores initialized global and static variables.

• Heap and stack locations, depending on the linker script.

If a buffer overflow corrupts a global variable, the .map file can identify nearby variables that may also be affected, provided the compiler has not optimized the memory allocation. Similarly, if a function pointer is corrupted, the .map file can reveal where it was stored in memory.

Debuggers (GDB & LLDB)

Debugging tools like GDB (GNU Debugger) and LLDB (LLVM Debugger) allow:

• Controlling execution (breakpoints, stepping through code).

• Inspecting variable values and memory addresses.

• Getting backtraces (viewing function calls before a crash).

• Extracting core dumps from microcontrollers for post-mortem analysis.

If the system halts on a crash, a backtrace (bt command in GDB) can reveal which function was executing before failure. If the overflow affects a heap-allocated variable, GDB can inspect heap memory usage to detect corruption.

The Debugging Process

Now, let’s go through a step-by-step debugging process to identify and fix buffer overflows. Once a crash or unexpected behavior occurs, follow these techniques to trace the root cause:

Step 1: Identify the misbehaving module

If the system crashes, use GDB or LLDB backtrace (bt command) to locate the last executed function. If the system behaves unexpectedly, determine which software module controls the affected functionality.

Step 2: Analyze inputs and outputs of the module

Every function or module has inputs and outputs. Create a truth table listing expected outputs for all possible inputs. Check if the unexpected behavior matches any undefined input combination, which may indicate corruption.

Step 3: Locate memory corruption using address analysis

If a variable shows incorrect values, determine its physical memory location. Depending on where the variable is stored:

1. Global/static variables (.bss / .data): Look up the memory map file for nearby buffers.

2. Heap variables: Snapshot heap allocations using GDB.

   Here’s an example of using GDB to find corrupted variables:

    (gdb) print &my_variable  # Get memory address of the variable
    $1 = (int *) 0x20001000
    (gdb) x/10x 0x20001000   # Examine memory near this address, Display 10 memory words in hexadecimal format starting from 0x20001000
   

Step 4: Identify the overflowing buffer

If a buffer is located just before the corrupted variable, inspect its usage in the code. Review all possible code paths that write to the buffer. Check if any design limitations could cause an overflow under a specific use cases.

Step 5: Fix the root cause

If the buffer overflow happened due to missing bounds checks, add proper input validation to prevent it. Buffer design should enforce strict memory limits. The module should implement strict boundary checks for all inputs and maintain a consistent state.

In addition to GDB/LLDB, you can also use techniques like hardware tracing and fault injection to simulate buffer overflows and observe system behavior in real-time.

While debugging helps identify and fix buffer overflows, prevention is always the best approach. Let’s explore techniques that can help avoid buffer overflows altogether.

How to Prevent Buffer Overflows

You can often prevent buffer overflows through good software design, defensive programming, hardware protections, and rigorous testing. Embedded systems, unlike general-purpose computers, often lack memory protection mechanisms, which means that buffer overflow prevention critical for system reliability and security.

Here are some key techniques to help prevent buffer overflows:

Defensive Programming

Defensive programming helps minimize buffer overflow risks by ensuring all inputs are validated and unexpected conditions are handled safely.

First, it’s crucial to validate input size before writing to a buffer. Always check the write index by adding the size of data to be written prior to writing data to make sure more data is not written than the available buffer space.

Then you’ll want to make sure you have proper error handling and fail-safe mechanisms in place. If an input is invalid, halt execution, log the error, or switch to a safe state. Also, functions should indicate success/failure with helpful error codes to prevent misuse.

Sample Code:

   #include <stdint.h>
   #include <string.h>
   #include <stdbool.h>
   #include <stdio.h>

   #define BUFFER_SIZE 300

   static uint16_t sample_count = 0;
   static uint8_t buffer[BUFFER_SIZE] = {0};

   typedef enum
   {
       SUCCESS = 0,
       NOT_ENOUGH_SPACE = 1,
       DATA_IS_INVALID = 2,
   } buffer_err_code_e;


   buffer_err_code_e updateBufferWithData(uint8_t *data, uint16_t size)
   {
       if (data == NULL || size == 0 || size > BUFFER_SIZE)  
       {
           return DATA_IS_INVALID; // Invalid input size
       }

       uint16_t available_space = BUFFER_SIZE - sample_count;
       bool can_write = (available_space >= size) ? true : false;

       if (!can_write)  
       {
           return NOT_ENOUGH_SPACE;
       }

       // Copy data safely
       memcpy(&buffer[sample_count], data, size);
       sample_count += size;

       return SUCCESS;
   }

   int main()
   {   
       buffer_err_code_e ret;

       // Save 1 byte to buffer
       uint8_t data_to_buffer = 10;
       ret = updateBufferWithData(&data_to_buffer, sizeof(data_to_buffer));
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       // Save an array of 20 bytes to buffer
       uint8_t data_to_buffer_1[20] = {5};
       ret = updateBufferWithData(data_to_buffer_1, sizeof(data_to_buffer_1));
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       // Save an array of 50 x 8 bytes, Intentional buffer overflow
       uint64_t data_to_buffer_2[50] = {7};
       ret = updateBufferWithData((uint8_t*)data_to_buffer_2, sizeof(data_to_buffer_2));  
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       return 0;
   }

Choosing the Right Buffer Design And Size

Some buffer designs handle overflow better than others. Choosing the correct buffer type and size for the application reduces the risk of corruption.

• Circular Buffers (Ring Buffers) prevent out-of-bounds writes by wrapping around. They overwrite the oldest data instead of corrupting memory. These are useful for real-time streaming data (for example, UART, sensor readings). This approach is ideal for applications where data loss is unacceptable.

• Ping-Pong Buffers (Double Buffers) use two buffers. One buffer fills up with data. Then, once it’s full, it switches to the second buffer while the first one is processed. This approach is beneficial for application that have strict requirements on no data loss. The buffer design should be based on the speed of write and read tasks.

Hardware Protection

Memory Protection Unit (MPU)

An MPU (Memory Protection Unit) helps detect unauthorized memory accesses, including buffer overflows, by restricting which regions of memory can be written to. It prevents buffer overflows from modifying critical memory regions and triggers a MemManage Fault if a process attemps to write outside an allowed region.

But keep in mind that, an MPU does not prevent buffer overflows – it only detects and stops execution when they occur. Not all microcontrollers have an MPU, and some low-end MCUs lack hardware protection, making software-based safeguards even more critical.

Modern C compilers provide several flags to identify memory errors at compile-time:

1. -Wall -Wextra: Enables useful warnings

2. -Warray-bounds: Detects out-of-bounds array access when the array size is known at compile-time

3. -Wstringop-overflow: Warns about possible overflows in string functions like memcpy and strcpy.

Testing and Validation

Testing helps detect buffer overflows before deployment, reducing the risk of field failures. Unit testing each function independently with valid inputs, boundary cases, and invalid inputs helps detect buffer-related issues early. Automated testing involves feeding random and invalid inputs into the system to uncover crashes and unexpected behavior. Static Analysis Tools like Coverity, Clang Static Analyzer help detect buffer overflows before runtime. Run real-world inputs on embedded hardware to detect issues.

Now that we've explored how to identify, debug, and prevent buffer overflows, it’s clear that these vulnerabilities pose a significant threat to embedded systems. From silent data corruption to catastrophic system failures, the consequences can be severe.

But with the right debugging tools, systematic analysis, and preventive techniques, you can effectively either prevent or mitigate buffer overflows in your systems.

Conclusion

Buffer overflows and corruption are major challenges in embedded systems, leading to crashes, unpredictable behavior, and security risks. Debugging these issues is difficult because their symptoms vary based on system state, requiring systematic analysis using memory map files, GDB/LLDB, and structured debugging approaches.

In this article, we explored:

• The causes and consequences of buffer overflows and corruptions

• How to debug buffer overflows using memory analysis and debugging tools

• Best practices for prevention

Buffer overflow prevention requires a multi-layered approach:

1. Follow a structured software design process to identify risks early.

2. Apply defensive programming principles to validate inputs and handle errors gracefully.

3. Use hardware-based protections like MPUs where available.

4. Enable compiler flags that help identify memory errors.

5. Test extensively, unit testing, automated testing, and code reviews help catch vulnerabilities early.

By implementing these best practices, you can minimize the risk of buffer overflows in embedded systems, improving reliability and security.

In embedded systems, where reliability and safety are critical, preventing buffer overflows is not just a best practice, it is a necessity. A single buffer overflow can compromise an entire system. Defensive programming, rigorous testing, and hardware protections are essential for building secure and robust embedded applications.

From healthcare to finance, robust software systems power our daily operations, making good software design essential to avoid inefficiencies and bottlenecks. This involves not just writing code but also designing systems that are easy to scale, maintain, and debug, while allowing others to contribute effectively.

Inefficient or ineffective software design can lead to significant issues, like scope creep, miscommunication within teams, project delays, resource misallocation, and complex systems that are difficult to maintain or understand. Without a strong design, teams often accumulate technical debt, which hinders long-term progress and increases maintenance costs.

This article will introduce you to key software design elements that will help you and your team address these challenges and guide you in building efficient, scalable systems. By understanding and applying these elements correctly, you can set up a project for both short-term and long-term success.

Prerequisites

I’ll explain these concepts through examples, but a basic understanding of programming in any language is required for this article (knowledge of Python will be especially beneficial).

Scope

The article will introduce key software design elements and explain them using an example. While I won’t provide a full software design for the example problem, I will include enough details to effectively illustrate each design element.

Table of Contents

• Overview of Key Software Design Elements

• A Walkthrough of the Software Design Process

  • Problem Statement

  • Use Cases

  • Requirements

  • High Level System Architecture

  • Detailed Software Design and Component Breakdown

• Conclusion: The Value of Thoughtful Software Design

Overview of Key Software Design Elements

To fully understand the benefits of the software design process, you’ll need to understand some key elements and their scope.

Once you have a good grasp of these, the next step is to define them for the specific problem at hand. Accurately defining these elements reduces risks and simplifies the implementation phase.

Doing this groundwork before implementation helps prevent late discoveries, minimizes the need for rewriting, and makes sure that the design can handle constraints and corner cases.

Now let’s briefly go over the key elements of the software design process:

1. Creating a problem statement: This step involves creating a clear and concise description of the problem that needs to be solved, along with its scope. The scope is essential because it focuses on the exact problem to be addressed and includes assumptions that must be considered during design.

2. Identifying use cases: This step outlines all possible user interactions with the software to achieve the desired outcome. It is a critical input to the architecture, as it helps create a design that addresses both general and edge-case use cases.

3. Stating requirements: This step defines the expectations of the software, such as its limitations, behaviors, and capabilities for different use cases.

4. Designing the architecture: This step provides a high-level structure of the software design, focusing on how to meet the requirements. The architecture typically includes components, how they interact, and how data flows through the system.

5. Drafting a detailed design: This step refines the high-level architecture into detailed, component-specific designs, ready for implementation.

In addition to these core elements, there are two important factors you need to consider throughout the design phase.

First, you’ll need to identify and state any assumptions you have. Assumptions can be present at any stage in the design process. Making correct assumptions increases the likelihood of success, improves focus, and reduces complexity in the design.

Second, you’ll need to create good documentation. Documentation is one of the most important elements in the software design process. It’s essential to document each stage as you go along. Documentation serves as the only formal record of the software design and is invaluable for presentations to management, for onboarding new team members, and for anyone returning to the project after a break. It saves valuable time and ensures continuity, as we often overestimate our own memory.

The figure below provides a visual summary of the key software design elements discussed in this section.

Next, we’ll apply these key software design elements to a practical example, demonstrating how each element contributes to building a robust and scalable system.

A Walkthrough of the Software Design Process

In any well-structured software project, clearly defining the problem is the first crucial step before diving into design and implementation. A well-defined problem ensures that the software meets user needs, remains maintainable, and scales effectively over time.

For this walkthrough, we will focus on designing a financial expense categorization system that processes and analyzes transaction data. This system is a part of a larger financial management solution and needs to be easy to debug, maintain, and scale.

Problem Statement

The problem statement provides a high-level goal for the software that we’ll design.

For this example, here’s our statement: Design a software solution that categorizes monthly expenses and generates a report from a list of transactions.

Define the scope

Defining the scope clarifies the smaller tasks that must be accomplished to meet the high-level goal. It outlines the focus of the software design and includes some assumptions.

Includes:

1. Implementing a parser to process a list of transactions provided as input.

2. Filtering transactions for a given month.

3. Analyzing, categorizing, and generating a report for each expense category.

Excludes:

Performance and memory optimization (excluded due to the limited scope of this article). While performance and memory optimizations are not the primary focus here, it’s important to keep future scalability in mind. Small design choices made now, such as selecting data structures, can help avoid significant refactoring later when the system grows.

Assumptions:

1. The list of transactions will be provided as a CSV file in the following format:
   Columns: "Date, Description, Amount, Type, Category Label".

2. Expense categories will be provided as input through a JSON file.

3. The software will run in a shell environment, and inputs will be taken as command-line arguments.

Now that the scope is clear, let’s examine how users will interact with the system through various use cases.

Use Cases

Use cases define how users will interact with the system to accomplish specific goals. Identifying accurate and valid use cases is critical to creating comprehensive requirements. Failing to capture enough use cases can lead to a design that is incomplete and lacks robustness. This may result in the need for redesigns, which increases time and resource consumption.

On the other hand, identifying too many use cases without considering their feasibility can lead to overly complex designs that are difficult to maintain and implement in the short term.

For our specific problem, the user will need to provide the following inputs while running the software in a shell:

1. A CSV file containing a list of transactions.

2. A month number.

3. A JSON file containing expense categories.

We need to consider all possible ways the user can interact with the script to achieve the desired outcome. For each of the three inputs, there are two possibilities: valid input or invalid input. This gives us 8 potential use cases (2 possibilities per input: valid and invalid). It's important to define what constitutes valid and invalid inputs for this problem:

• CSV File: Valid if it is in the format described in Assumption 1 (columns: "Date, Description, Amount, Type, Category Label").

• Month Number: Valid if the value is between 1 and 12.

• JSON File: Valid if it contains expense categories in the correct JSON format.

An input is invalid if it doesn't meet these definitions or if the input is absent.

It’s also crucial to consider the correlation between inputs when evaluating the feasibility of certain use cases, as they may interact with each other in unforeseen ways. Based on these use cases, we can now define the specific requirements that the system must meet.

Requirements

Now, let’s define the expected behaviors, limitations, and capabilities for each use case. Requirements serve as the foundation for architecture, specifications, and implementation. Based on our problem statement, the software will need to accomplish the following tasks:

1. The script shall take three inputs: a CSV file of transactions, a month number, and a JSON file of expense categories.

2. The script shall verify all inputs.

3. The script shall throw an error and exit if the CSV file cannot be opened or if it does not match the format in Assumption 1.

4. The script shall throw an error and exit if the JSON file cannot be opened.

5. The script shall throw an error if the month number is not between 1 and 12.

6. The script shall parse each transaction and load it into a data structure.

7. The script shall filter transactions by the specified month.

8. The script shall load the expense categories from the JSON file into a data structure.

9. The script shall categorize transactions based on the category label provided in the CSV file.

10. The script shall throw an exception if a category label in the CSV file is not present in the expense categories.

11. The script shall use a categorizing function to assign transactions to categories from the JSON file.

12. A class shall encapsulate categorized transactions, providing APIs to modify or access them.

13. The script shall support statistics calculation and report generation for categorized transactions.

With the requirements in place, we can now design a high-level architecture to meet those needs.

High Level System Architecture

In this stage, we will design the system at a high level, much like creating a master plan. Architecture involves organizing the software's functions into distinct components, illustrating how they interact, and mapping the flow of control and data through the system. While designing the architecture in this tutorial, we’ll incorporate good design principles.

For this example, the high-level requirements include:

1. Loading inputs and verifying them.

2. Applying time-based filtering.

3. Categorizing transactions based on category labels and descriptions.

4. Managing categorized transactions in a finance registry.

5. Generating reports from the categorized data.

One important component of software architecture is telemetry. Telemetry gathers data on the software's behavior, which is invaluable for debugging and performance assessment in real-world environments.

For smaller systems, simpler logging mechanisms may be sufficient to track basic errors and monitor performance. The decision to implement telemetry should depend on the complexity of the system and operational requirements.

Since telemetry provides such a helpful feedback loop for improving the design in future iterations, we’ll add it to the list of components here.

We’ll build our system architecture around a Test-Driven Development (TDD) approach. We’ll design each component with testing in mind to ensure it meets our requirements.

Just keep in mind that while TDD is a strong practice for ensuring code quality, it may not be the best fit for all projects. In scenarios where you need rapid prototyping or exploratory development, testing might be prioritized after initial iterations. Balancing between TDD and other methodologies depends on the project context and team preferences.

Our architecture will follow a modular structure, meaning the system will be divided into self-contained components. Each component will be responsible for specific functionality, making the system easier to test, maintain, and scale.

To achieve this, the architecture will emphasize loose coupling between components. Each component will interact with others through well-defined interfaces or APIs, ensuring minimal dependencies. We’ll abstract and encapsulate internal implementation details, exposing only the necessary information for interaction. Also, each component will handle its own errors and exceptions to ensure robustness and fault isolation.

But it is also important to consider a centralized error-handling strategy in some cases. Centralizing error handling can reduce redundancy, improve consistency, and make maintenance easier. The choice between local and centralized error handling should depend on the system's complexity and how components interact. This will contribute to the overall scalability and maintainability of the system.

Below is a summary of each component's functionality in this architecture:

• Load and verify input: This component will take the CSV file, JSON file, and month number as input, verify their validity, and load the data into structures.

• Time-based filter: This component will filter transactions based on the input month and store the filtered transactions in a data structure.

• Label-based categorization: This component will categorize transactions based on the category label in the CSV file.

• Description-based categorization: This component will categorize transactions using an algorithm based on the transaction description.

• Finance registry: This component will store all categorized transactions for further processing. It isolates the post-processing of categorized transactions from the categorization process and provides methods for updating or retrieving datasets.

• Report generation: This component will generate expense reports from the categorized transaction data.

• Telemetry: This component will monitor the performance of other components. It will track the flow of transactions, ensuring that all transactions are categorized either by label or description. Additional parameters can be added as needed to monitor specific functionalities.

The diagram below demonstrates the flow of data through these components:

Detailed Software Design and Component Breakdown

While we won't cover the full system design, this section will highlight key components and their specifications. For this example, I will assume the role of both the designer and implementer of the software.

Software design and specifications depend on several factors, including the designer's knowledge, skill set, available time, and resources. We’ll define some of the design details for the system, starting with the choice of the implementation language.

Choosing the right language is based on several important factors:

1. The language must meet the software requirements.

2. It should be stable, and have strong support from an active developer community.

3. Additional considerations include performance (speed and memory), scalability (ability to grow with future requirements), and platform support (ability to run on all major operating systems).

If you’re the one implementing this design, you’ll need to be familiar with and confident using that programming language. For this project, I chose Python because it meets all the project requirements, has a robust developer community for support, it’s stable, and I’m confident in using it to complete the implementation successfully.

Data Structures

Now, let’s look at the fundamental data structures that we’ll use in the design. We need to load the contents of the CSV file into a data structure for further analysis and processing. In Python, the Pandas DataFrame from the Pandas library is ideal for analyzing and processing tables, so we will use it to store the transactions.

For generating report, we will encapsulate categorized transactions along with relevant statistics, such as the total number of transactions, mean amount, and maximum amount, within a dedicated dataset class. This approach ensures a clear separation of concerns, where the dataset class manages data processing, while the reporting component focuses on presentation.

By structuring the system this way, we enhance reusability, maintainability, and scalability, making it easier to extend and modify in the future.

This dataset class will include:

• Member variables: category name, category description, a Pandas DataFrame for transactions, total number of transactions, mean amount, and max amount of transactions.

• Member functions: set/get DataFrame, save dataset to CSV (useful for debugging).

Here’s an example of a Dataset class in Python for structured data management and processing:

import pandas as pd  # Import Pandas for data handling

class Dataset:
    """
    A class representing a structured dataset with a name, predefined keys, 
    and a Pandas DataFrame.
    """

    def __init__(self, name, keys):
        """
        Initializes the Dataset object.

        Parameters:
        name (str): The name of the dataset.
        keys (list): A list of expected column names for the dataset.

        Attributes:
        self.name (str): Stores the dataset name as a string.
        self.keys (list): Stores the expected column names for data organization.
        self.mean_amt (float): Tracks the mean (average) transaction amount.
        self.max_amt (float): Tracks the maximum transaction amount.
        self.count (int): Stores the total number of transactions in the dataset.
        self.dataframe (pd.DataFrame): A Pandas DataFrame initialized with the specified column names.
        """
        self.name = str(name)  # Convert and store dataset name as a string
        self.keys = keys  # Store expected column names for consistency
        self.mean_amt = 0  # Initialize mean transaction amount to zero
        self.max_amt = 0  # Initialize max transaction amount to zero
        self.count = 0  # Initialize transaction count to zero
        self.dataframe = pd.DataFrame(columns=keys)  # Initialize empty DataFrame with predefined columns

    def getName(self):
        """
        Returns the name of the dataset.

        Returns:
        str: The name of the dataset.
        """
        return self.name  # Fixed: Removed incorrect parentheses

    def getValue(self, key):
        """
        Retrieves a specific column from the DataFrame.

        Parameters:
        key (str): The column name to retrieve.

        Returns:
        pandas.Series or None: The column data if the key exists, otherwise None.
        """
        if key in self.dataframe.columns:
            return self.dataframe[key]
        else:
            print(f"Warning: Key '{key}' not found in DataFrame.")
            return None  # Prevents KeyError

    def getKeys(self):
        """
        Returns the list of expected keys (column names) of the dataset.

        Returns:
        list: The keys defining the dataset.
        """
        return self.keys

    def setDataFrame(self, dataframe):
        """
        Sets the dataset's DataFrame while ensuring it contains only expected keys.

        Parameters:
        dataframe (pandas.DataFrame): The DataFrame to assign to the dataset.
        """
        if not isinstance(dataframe, pd.DataFrame):
            raise TypeError("Provided data is not a valid pandas DataFrame.")

        # Ensure only the expected columns are included
        self.dataframe = dataframe[self.keys].copy() if set(self.keys).issubset(dataframe.columns) else dataframe.copy()

    def getDataFrame(self):
        """
        Returns the DataFrame associated with the dataset.

        Returns:
        pandas.DataFrame: The dataset's DataFrame.
        """
        return self.dataframe

    def save_to_csv(self, file_name):
        """
        Saves the dataset's DataFrame to a CSV file.

        Parameters:
        file_name (str): The name of the CSV file to save.
        """
        self.dataframe.to_csv(file_name, mode='w', index=False)  # Save the DataFrame to CSV

In the previous section, we outlined the high-level system architecture, detailing the core components and their interactions. Now, let’s dive into the detailed design of some of the individual components, specifying how we’ll implement each one and how it’ll function within the system. We’ll also break down the components to explain how they work together to process the input and generate the report.

Below, you can see the flow diagram for the software, illustrating the interaction between the core components and the flow of data through the system.

Category Label-Based Filtering Component

The Category Label-Based Filtering Component classifies transactions by matching their "Category Label" with predefined expense categories from a JSON file. Transactions with valid category labels are stored in the finance registry, while unmatched ones remain for further processing.

• Input: DataFrame of time-filtered transactions, expense categories from JSON.

• Libraries used: Pandas DataFrame.

• Software design: Filters transactions based on the "Category Label" column and assigns them to corresponding categories. Transactions that cannot be categorized remain for further processing.

• Output: DataFrame of remaining transactions with empty values in the "Category Label" field.

• Component tests: Validate handling of valid, invalid, and missing category labels.

Finance Registry Component

The Finance Registry Component manages categorized transactions by storing them as datasets for each expense category. It maintains a structured collection of DataFrames, each containing transactions and summary statistics such as total count, max amount, and mean amount.

• Input: Expense categories from JSON.

• Libraries used: Pandas DataFrame.

• Software design: Implements a class that organizes datasets for all expense categories, providing methods to set and retrieve DataFrames.

• Component tests: Validate dataset creation, ensuring correct storage and retrieval of categorized transactions.

Here’s a simple and efficient Finance Registry implementation in Python for managing categorized financial datasets:

from Dataset import Dataset
import pandas as pd  # Ensure Pandas is imported if used elsewhere

# Define column structure for datasets
KEYS = ("Date", "Description", "Amount", "Transaction Type", "Category", "Account Name", "Labels", "Notes")

# Define dataset names for different financial categories
EXAMPLE_DATASET_NAMES = ("Investment", "Expense", "Savings")

class FinanceRegistry:
    """
    A class to manage categorized financial datasets, including investment, expense, and savings datasets.
    This registry allows structured access to transaction data and maintains aggregated financial metrics.
    """

    def __init__(self):
        """
        Initializes the FinanceRegistry object.

        Attributes:
        self.example_dataset (dict): A dictionary storing Dataset objects for financial datasets.
        """
        self.example_dataset = {name: Dataset(name, KEYS) for name in EXAMPLE_DATASET_NAMES}  # Create datasets for categories

    def setExampleDatasetToRegistry(self, name, dataframe):
        """
        Merges a new dataframe into the existing dataset for a given financial category.

        Parameters:
        name (str): The category name (e.g., "Investment", "Expense", or "Savings").
        dataframe (pd.DataFrame): The new data to be added.

        If the dataset already contains data, it concatenates the new dataframe to the existing one.

        Raises:
        ValueError: If the provided name is not a valid dataset category.
        """
        if name not in self.example_dataset:
            raise ValueError(f"Invalid dataset name: '{name}'. Expected one of {EXAMPLE_DATASET_NAMES}")

        df = self.example_dataset[name].getDataFrame()  # Get existing dataset

        if not dataframe.empty:  # Ensure the new dataframe is not empty
            dataframe = pd.concat([df, dataframe], axis=0, ignore_index=True)  # Append new data

        self.example_dataset[name].setDataFrame(dataframe)  # Update dataset in registry

    def getExampleDatasetFromRegistry(self, name):
        """
        Retrieves the dataset for a given financial category.

        Parameters:
        name (str): The category name (e.g., "Investment", "Expense", or "Savings").

        Returns:
        Dataset: The dataset corresponding to the given name.

        Raises:
        ValueError: If the provided name is not a valid dataset category.
        """
        if name not in self.example_dataset:
            raise ValueError(f"Invalid dataset name: '{name}'. Expected one of {EXAMPLE_DATASET_NAMES}")

        return self.example_dataset[name]

The diagram below illustrates how the Finance Registry organizes these datasets for further processing in the Report Generation component.

Report Generation Component

The Report Generation Component processes categorized transaction datasets from the finance registry and generates summary statistics. It calculates key financial metrics such as maximum amount, mean amount, and total transaction count. It also provides functionality to display categorized transactions in a structured format within the shell.

• Input: Datasets of categorized transactions from the finance registry.

• Libraries used: Numpy for calculations, Tabulate for formatted shell output (if needed).

• Software design: Implements a class with methods to compute financial statistics and display transaction summaries per expense category.

• Component tests: Validate correct calculation of mean, max, and total transactions, and ensure accurate display of categorized datasets in the shell.

Here’s a function to compute transaction statistics, including mean, max, and count, from a dataset in the report generation component:

from Dataset import Dataset
import numpy as np

def calculateStats(dataset):
    """
    Computes statistical metrics for a given dataset.

    Parameters:
    dataset: The dataset containing transaction data.

    Updates:
    - dataset.mean: Mean transaction amount.
    - dataset.max: Maximum transaction amount.
    - dataset.count: Number of transactions.
    """

    # Return early if the dataset has no transactions
    if dataset.dataframe.empty:
        return

    # Extract transaction amounts as a list
    tx_amount_list = dataset.dataframe['Amount'].astype(float).round(2).tolist()

    # Adjust transaction amounts based on "Transaction Type"
    for i, tx_type in enumerate(dataset.dataframe['Transaction Type']):
        if tx_type == 'debit':
            tx_amount_list[i] *= -1  # Convert debit transactions to negative values

    # Compute statistical metrics
    dataset.mean = round(np.mean(tx_amount_list), 2)
    dataset.max = max(tx_amount_list)
    dataset.count = len(tx_amount_list)

This concludes the design section, where we explored key software design elements with a practical example. The next step, implementation, is beyond the scope of this article. But it's crucial to recognize that new challenges often emerge during development, requiring updates to requirements, architecture, and specifications.

The purpose of this article is not to provide a full implementation, but to teach you some basic software design principles through an example. The focus is on understanding how to structure software, define clear requirements, and create scalable architectures, all before writing code.

By following a structured design process, you can shift complex problem-solving from implementation to the architecture phase, where you can explore solutions more effectively using flowcharts, block diagrams, and documentation. This makes the development process more organized, efficient, and maintainable, a crucial skill for real-world software engineering.

If you're learning to code, remember that good design is just as important as writing code itself!

Conclusion: The Value of Thoughtful Software Design

With well-defined problem statements, scope, requirements, specifications, and design, even complex problems can be solved and maintained in a sustainable way.

The steps we went through in this article can help you break down any problem, regardless of its complexity, into smaller, actionable tasks that you and your team can efficiently tackle.

Without proper planning, projects are often plagued by scope creep, wasted time and resources, miscommunication between teams, overly complicated designs, technical debt, and frequent redesigns.
Good design is often simple design, but achieving simplicity is difficult without thorough planning.

Approaching each problem with the mindset of defining a Problem Statement, Scope, Use Cases, Requirements, Architecture, and Specifications helps cultivate a strong software design mindset. This mindset is crucial for developing software that is scalable, maintainable, and high quality.

We talk about his experience getting laid off as a dev, and how we prepared for his mid-career job search.

We talk about:

• How Caleb got laid off and went about landing his next developer job

• How most people sleep on networking and recruiters, but shouldn't

• Why Caleb is so serious about teaching system design concepts

• How Caleb pairs his deep focus with broad extracurricular learning through podcasts and white papers

Support comes from the 11,343 kind folks who support freeCodeCamp through a monthly donation. Join these kind folks and help our mission by going to https://www.freecodecamp.org/donate

You can watch the interview on YouTube:

Or you can listen to the podcast in Apple Podcasts, Spotify, or your favorite podcast app. Be sure to follow the freeCodeCamp Podcast there so you'll get new episodes each Friday.

Links we talk about during our conversation:

• Caleb's course on Database Design: https://www.freecodecamp.org/news/database-design-full-course-43233664125b/

• Caleb's system design lecture playlist: https://www.youtube.com/watch?v=0e7yQ43bUtg&list=PL_c9BZzLwBRLSs6x50D5WIH76VCUxJs9E

• Caleb on LinkedIn: https://www.linkedin.com/in/calebcurry/