The first time I tried this, I did what seemed obvious to me. I put a <canvas> on top of the map and drew my animation on it.

But it looked wrong in a way I couldn't fix with more polish. The effect had edges. It floated above the ground. And as soon as I tilted or rotated the map, it stopped lining up with anything underneath.

I hit this while building HazardSim, a browser simulator that runs natural disasters on a real map. The tsunami scenario went through about a dozen rewrites before it stopped looking like a cut-out pasted over the city.

Much of the testing, and a good share of the bug reports, came from my 12-year-old son Tim, who turned out to be a more patient tester than I am. Most of those rewrites taught me the same thing from a different direction: draw your simulation in map coordinates, not screen coordinates.

That rule carried me through the tsunami. Then I started on the earthquake and ran into the opposite problem. Mapbox GL JS will draw a city in 3D for you: turn on a fill-extrusion layer and every footprint in the vector tiles rises to its real height.

It looks good until you try to make one of those buildings fall down, and you find out that the map's 3D is a rendering trick, not a scene. There are no objects to grab, rotate or break. Things that lie on the ground and things that have height turned out to need two different techniques, and this guide covers both.

Part 1 builds the ground layer. You'll set up a grid aligned to the map's own tiles, load real elevation data into it, run a flood fill that respects hills and valleys, hand the canvas to the map as a raster source, write pixels into it every frame, layer video textures on top with canvas compositing modes, and move the camera into the scene.

Part 2 adds height. You'll pull building footprints and heights out of the map's tiles, make the extrusions look like buildings, decide which ones fall from the magnitude and distance, animate the collapse without rebuilding any layers, set up a three.js custom layer inside the map's own WebGL context, turn a footprint into a box that tips over, throw debris and dust, and shake the camera.

At the end we'll go through the mistakes from both halves, several of which produce no error message and cost me a day each.

Prerequisites

You should be comfortable with JavaScript, requestAnimationFrame, and the basics of the 2D canvas API: getContext("2d"), drawImage, and ImageData.

You don't need any background in web maps. Everything map-specific is explained where it comes up.

Part 2 also assumes you've used three.js at least once: meshes, materials, a scene and a camera. If you haven't, the three.js fundamentals are enough to follow along, and the parts that interact with the map are explained in full.

The examples use Mapbox GL JS, which is what runs on HazardSim. You'll need a free Mapbox access token for the base map and the vector tiles. Every API used here also exists in MapLibre GL JS, the open-source fork, under the same names. The one place the two differ is called out in Part 2.

Table of Contents

• Part 1: Drawing on the Ground

• The Core Idea: Map Space, Not Screen Space

• Step 1: Choose a Grid Aligned to Map Tiles

• Step 2: Load Real Elevation Into the Grid

• Step 3: Attach a Canvas as a Map Source

• Step 4: Simulate a Flood That Respects the Terrain

• Step 5: Animate by Writing Pixels

• Step 6: Layer Video Textures With Canvas Compositing

• Step 7: Move the Camera Into the Scene

• Where the Ground Layer Stops

• Part 2: Things with Height

• Two Kinds of 3D, and Why You Need Both

• Step 8: Get Buildings Out of the Map's Own Tiles

• Step 9: Make Extrusions Look Like Buildings

• Step 10: Decide What Falls

• Step 11: Animate Collapse Without Rebuilding Layers

• Step 12: A three.js Scene Inside the Map

• Step 13: From Footprint to a Box That Tips Over

• Step 14: Debris and Dust

• Step 15: The Camera Has to Shake

• Keeping It Fast

• What Went Wrong, and Why

• Conclusion

Part 1: Drawing on the Ground

The first half of this guide is about anything that belongs to the surface of the map: water, fire spread, a smoke stain, or a shockwave. The technique is one idea applied carefully, and the next seven steps build it up from a grid to a camera move.

The Core Idea: Map Space, Not Screen Space

A web map is a camera looking at a surface. When the user drags, zooms, tilts or rotates, the camera moves and the surface stays where it is. Roads, buildings, and your flood all belong to the surface, so the camera has to transform them together.

A canvas placed over the map's DOM element is attached to the camera, not to the surface. Think of it as a sheet of glass in front of the lens. Whatever you draw on it stays where you drew it while the world moves underneath.

At a top-down zoom with no rotation, you can fake alignment by reprojecting your points every frame, and it almost works. Add pitch and it falls apart: a flat sprite has no perspective, no foreshortening, and nothing in front of or behind it.

The fix is to give your canvas to the map as a data source. Mapbox GL JS has a canvas source type. You pass it a canvas element and four geographic corners, and from then on the map drapes that canvas over the rectangle the same way it drapes satellite imagery. Rotate the map and the canvas rotates. Tilt it and the canvas foreshortens. Turn on 3D terrain and the canvas follows the hills.

Once the drawing surface lives in map space, several hard problems disappear. The rest of this handbook is about what you can do with that.

Step 1: Choose a Grid Aligned to Map Tiles

Your simulation needs a grid: a 2D array of cells, each holding elevation, water depth, or whatever your model tracks. The decision that makes everything downstream simpler is to align that grid to the map's own tile pyramid.

Web maps use the Web Mercator projection, cut into 256×256-pixel tiles at zoom levels 0 to 22. At zoom z the world is 2^z tiles wide. If one grid cell is exactly one tile pixel at some zoom, converting between grid indices and longitude/latitude takes four short functions, and every dataset that ships as tiles (elevation, population, imagery) drops into your grid without resampling.

Here are the four conversions. lon2x and lat2y return fractional tile coordinates. Multiply by 256 to get pixels.

function lon2x(lon, z) {
  return (lon + 180) / 360 * Math.pow(2, z);
}

function lat2y(lat, z) {
  const rad = lat * Math.PI / 180;
  return (1 - Math.log(Math.tan(rad) + 1 / Math.cos(rad)) / Math.PI) / 2 * Math.pow(2, z);
}

function x2lon(x, z) {
  return x / Math.pow(2, z) * 360 - 180;
}

function y2lat(y, z) {
  const n = Math.PI - 2 * Math.PI * y / Math.pow(2, z);
  return 180 / Math.PI * Math.atan(0.5 * (Math.exp(n) - Math.exp(-n)));
}

Next, pick the zoom. You want the area of interest to fit in a grid that's fine enough to look smooth but small enough to iterate over sixty times a second. In HazardSim, I start at zoom 15 and step down until the bounding box fits in roughly 780 pixels across and under about 620,000 cells:

function chooseZoom(bbox) {
  for (let z = 15; z >= 8; z--) {
    const pw = (lon2x(bbox.e, z) - lon2x(bbox.w, z)) * 256;
    const ph = (lat2y(bbox.s, z) - lat2y(bbox.n, z)) * 256;
    if (pw <= 780 && pw * ph <= 620000) return z;
  }
  return 8;
}

With the zoom chosen, snap the bounding box outward to whole tiles. That rectangle is what your canvas will cover, and those are the tiles you'll download:

function gridFor(bbox) {
  const z = chooseZoom(bbox);
  const x0 = Math.floor(lon2x(bbox.w, z)), x1 = Math.floor(lon2x(bbox.e, z));
  const y0 = Math.floor(lat2y(bbox.n, z)), y1 = Math.floor(lat2y(bbox.s, z));
  const nx = x1 - x0 + 1, ny = y1 - y0 + 1;
  const latMid = (bbox.n + bbox.s) / 2;

  return {
    z, x0, y0, nx, ny,
    W: nx * 256,
    H: ny * 256,
    // metres per cell at this latitude and zoom
    cellM: 156543.03392 * Math.cos(latMid * Math.PI / 180) / Math.pow(2, z),
    // exact geographic corners of the snapped rectangle
    corners: {
      w: x2lon(x0, z), e: x2lon(x1 + 1, z),
      n: y2lat(y0, z), s: y2lat(y1 + 1, z),
    },
  };
}

Keep cellM around. You'll need it to turn "the wave has travelled 400 metres" into "the wave has travelled 3.8 cells". For Odesa, the city in most of the screenshots here, the grid ends up at zoom 10 with 105-metre cells.

Step 2: Load Real Elevation Into the Grid

A flood that ignores hills is a blue blob. What makes a simulation on a real map feel real is real terrain, and terrain is free.

The AWS Open Data program hosts the Mapzen Terrarium elevation tiles. They're ordinary PNGs with height encoded in the pixel colour:

height_m = R * 256 + G + B / 256 - 32768

Because the grid is tile-aligned, loading elevation means fetching the tiles, drawing them onto one big canvas in their natural positions, reading the pixels back, and decoding. Nothing is resampled. The grid cell and the elevation sample are the same pixel.

function terrariumUrl(z, x, y) {
  return `https://s3.amazonaws.com/elevation-tiles-prod/terrarium/${z}/${x}/${y}.png`;
}

function loadImage(url) {
  return new Promise((resolve, reject) => {
    const img = new Image();
    img.crossOrigin = "anonymous";
    img.onload = () => resolve(img);
    img.onerror = reject;
    img.src = url;
  });
}

async function loadElevation(grid) {
  const cv = document.createElement("canvas");
  cv.width = grid.W;
  cv.height = grid.H;
  const cx = cv.getContext("2d", { willReadFrequently: true });

  const jobs = [];
  for (let iy = 0; iy < grid.ny; iy++) {
    for (let ix = 0; ix < grid.nx; ix++) {
      jobs.push(
        loadImage(terrariumUrl(grid.z, grid.x0 + ix, grid.y0 + iy))
          .then(img => cx.drawImage(img, ix * 256, iy * 256))
      );
    }
  }
  await Promise.all(jobs);

  const d = cx.getImageData(0, 0, grid.W, grid.H).data;
  const elev = new Float32Array(grid.W * grid.H);
  for (let i = 0; i < elev.length; i++) {
    const o = i * 4;
    elev[i] = d[o] * 256 + d[o + 1] + d[o + 2] / 256 - 32768;
  }
  return elev;
}

crossOrigin = "anonymous" is required. Without it the canvas becomes "tainted" and getImageData throws. The Terrarium bucket sends the right CORS headers, so this works as written.

Since you already have a Mapbox token, you can use Mapbox's own Terrain-RGB tiles instead. The URL is https://api.mapbox.com/v4/mapbox.terrain-rgb/{z}/{x}/{y}.pngraw?access_token=… and the decode formula is different:

// Mapbox Terrain-RGB
elev[i] = -10000 + (d[o] * 65536 + d[o + 1] * 256 + d[o + 2]) * 0.1;

HazardSim supports both and switches with a config flag. Terrain-RGB is cleaner data with fewer spikes. Terrarium is free of per-tile billing. For a tutorial, I'd start with Terrarium.

Two practical notes. Terrarium data has occasional single-pixel spikes, and a 3×3 median pass over cells that differ from all their neighbours by more than a few metres cleans them up. And cells at or below zero are sea, except where they aren't, like in the Netherlands, New Orleans, or Death Valley. I treat elev <= 0 as sea only when the cell connects to the map edge through other sea cells. That handles the common cases.

Step 3: Attach a Canvas as a Map Source

Now the central move. Create a canvas the size of your grid and register it with the map as a canvas source whose four corners are the snapped rectangle from Step 1.

<div id="map"></div>
<canvas id="floodcanvas" width="10" height="10" style="display:none"></canvas>

mapboxgl.accessToken = "YOUR_MAPBOX_TOKEN";

const map = new mapboxgl.Map({
  container: "map",
  style: "mapbox://styles/mapbox/satellite-streets-v12",
  center: [30.73, 46.48],
  zoom: 11,
});

const fc = document.getElementById("floodcanvas");
const fctx = fc.getContext("2d");
let fimg = null;

function attachCanvas(grid) {
  fc.width = grid.W;
  fc.height = grid.H;
  fimg = fctx.createImageData(grid.W, grid.H);

  const c = grid.corners;
  const coords = [[c.w, c.n], [c.e, c.n], [c.e, c.s], [c.w, c.s]];

  if (map.getSource("flood")) {
    map.getSource("flood").setCoordinates(coords);
    return;
  }
  map.addSource("flood", {
    type: "canvas",
    canvas: "floodcanvas",
    coordinates: coords,
    animate: true,
  });
  map.addLayer({
    id: "flood",
    type: "raster",
    source: "flood",
    paint: {
      "raster-opacity": 0.94,
      "raster-resampling": "linear",
      "raster-fade-duration": 0,
    },
  });
}

Three options matter here:

• animate: true tells Mapbox GL to re-upload the canvas as a texture every frame. Without it, the animation freezes on the first frame.

• "raster-resampling": "linear" blends between cells when the user zooms in past your grid resolution. With 105-metre cells at a city zoom, it's the difference between water and a pixel-art game.

• "raster-fade-duration": 0 turns off the cross-fade Mapbox GL applies to raster tiles. You're redrawing every frame, and a fade only smears it.

The canvas is display:none because the map reads it directly. It never needs to be visible in the DOM.

From here on, anything you write into fimg and push with fctx.putImageData(fimg, 0, 0) shows up on the map in the right place, at any pitch and bearing, draped over terrain if terrain is on. The rendering problem has turned into a pixel-array problem.

Step 4: Simulate a Flood That Respects the Terrain

A tsunami doesn't flood everything below the wave height. Water has to get there. It enters from the sea, flows over low ground, and stops where the ground rises above the water level, even if there's lower ground on the far side of a ridge. That's a connectivity problem, and breadth-first search is the tool for it.

First, a distance field: how far is each cell from the coast? Seed the queue with every sea cell adjacent to land, then expand into the sea. You'll use this to animate the wave approaching, and the land-side equivalent to animate it spreading inland.

function seaDistance(grid, sea) {
  const { W, H, cellM } = grid;
  const N = W * H;
  const dist = new Float32Array(N).fill(-1);
  const q = new Int32Array(N);
  let head = 0, tail = 0;

  // seed: sea cells that touch land
  for (let i = 0; i < N; i++) {
    if (!sea[i]) continue;
    const x = i % W, y = (i / W) | 0;
    const touchesLand =
      (x > 0 && !sea[i - 1]) || (x < W - 1 && !sea[i + 1]) ||
      (y > 0 && !sea[i - W]) || (y < H - 1 && !sea[i + W]);
    if (touchesLand) { dist[i] = 0; q[tail++] = i; }
  }

  // expand
  while (head < tail) {
    const c = q[head++];
    const cx = c % W, cy = (c / W) | 0;
    for (let dy = -1; dy <= 1; dy++) {
      for (let dx = -1; dx <= 1; dx++) {
        if (!dx && !dy) continue;
        const nx = cx + dx, ny = cy + dy;
        if (nx < 0 || ny < 0 || nx >= W || ny >= H) continue;
        const n = ny * W + nx;
        if (!sea[n] || dist[n] >= 0) continue;
        dist[n] = dist[c] + (dx && dy ? cellM * Math.SQRT2 : cellM);
        q[tail++] = n;
      }
    }
  }
  return dist;
}

The flood is the same search in the other direction, with one extra rule: a cell is reachable only if the water surface is above its ground. waveH is the wave height at the shore in metres. The water loses height as it travels inland, and a linear decay is enough to look right:

function floodFill(grid, elev, sea, waveH) {
  const { W, H, cellM } = grid;
  const N = W * H;
  const depth = new Float32Array(N);       // metres of water above ground
  const reach = new Float32Array(N).fill(-1); // distance from shore, metres
  const q = new Int32Array(N);
  let head = 0, tail = 0;

  for (let i = 0; i < N; i++) {
    if (!sea[i]) continue;
    const x = i % W, y = (i / W) | 0;
    const nb = [i - 1, i + 1, i - W, i + W];
    for (const n of nb) {
      if (n < 0 || n >= N || sea[n] || reach[n] >= 0) continue;
      if ((n === i - 1 && x === 0) || (n === i + 1 && x === W - 1)) continue;
      reach[n] = 0;
      q[tail++] = n;
    }
  }

  const decayPerM = waveH / 12000; // loses its height over ~12 km

  while (head < tail) {
    const c = q[head++];
    const level = waveH - reach[c] * decayPerM;
    const d = level - elev[c];
    if (d <= 0) continue;               // ground is above the water here: stop
    depth[c] = d;

    const cx = c % W, cy = (c / W) | 0;
    for (let dy = -1; dy <= 1; dy++) {
      for (let dx = -1; dx <= 1; dx++) {
        if (!dx && !dy) continue;
        const nx = cx + dx, ny = cy + dy;
        if (nx < 0 || ny < 0 || nx >= W || ny >= H) continue;
        const n = ny * W + nx;
        if (sea[n] || reach[n] >= 0) continue;
        reach[n] = reach[c] + (dx && dy ? cellM * Math.SQRT2 : cellM);
        q[tail++] = n;
      }
    }
  }
  return { depth, reach };
}

Run this once per scenario, not per frame. On a 700×500 grid it takes a few milliseconds. The outputs, depth for every cell and reach for how far inland it is, are all the animation needs.

Because the fill follows real elevation, it goes up river valleys and stops at embankments. Multiply the flooded cell count by cellM² and you get a flooded area in square kilometres you can stand behind.

Step 5: Animate by Writing Pixels

The frame loop does one thing. For each flooded land cell, it decides whether the water has arrived yet, and if so, what colour it is.

"Arrived" is a comparison of the cell's reach against a front position F that grows with time. Add some per-cell noise to reach so the front isn't a perfect contour line, and the water starts to look like water:

let t0 = null;
const T_LANDFALL = 3.9;   // seconds until the wave reaches the shore
const T_FULL = 10.9;      // seconds until it reaches its maximum extent

function ease(k) {
  return k <= 0 ? 0 : k >= 1 ? 1 : 1 - Math.pow(1 - k, 2.2);
}

function frame(now) {
  if (t0 === null) t0 = now;
  const t = (now - t0) / 1000;
  const d = fimg.data;
  const { W, H } = grid;

  // how far inland the front has got, in metres
  const F = t < T_LANDFALL
    ? -1e9
    : maxReach * ease((t - T_LANDFALL) / (T_FULL - T_LANDFALL));

  for (let i = 0; i < W * H; i++) {
    const o = i * 4;
    if (sea[i] || depth[i] <= 0) { d[o + 3] = 0; continue; }

    const arrival = reach[i] + noise[i] * 180;   // ±90 m of jitter
    if (arrival > F) { d[o + 3] = 0; continue; }

    const age = (F - arrival) / 150;             // how long the cell has been under
    const dp = depth[i];

    // base colour: shallow is lighter, deep is darker
    let r = 40 + Math.max(0, 60 - dp * 10);
    let g = 70 + Math.max(0, 50 - dp * 8);
    let b = 95 + Math.max(0, 40 - dp * 6);
    let a = dp < 0.5 ? 215 + dp * 60 : 245;

    // the leading edge is foam: push toward white for the first moments
    if (age < 1.4) {
      const k = Math.pow(1 - age / 1.4, 2) * 0.8;
      r += (240 - r) * k; g += (246 - g) * k; b += (246 - b) * k;
    }

    d[o] = r; d[o + 1] = g; d[o + 2] = b; d[o + 3] = a;
  }

  fctx.putImageData(fimg, 0, 0);
  if (t < T_FULL + 3) requestAnimationFrame(frame);
}

requestAnimationFrame(frame);

noise is a Float32Array of random values in [-0.5, 0.5] generated once per run. maxReach is the largest reach value among flooded cells.

There are two performance points: A 700×500 grid is 350,000 cells, and this loop runs at 60 fps on my laptop and around 30 fps on a mid-range phone. If you need more, iterate only over the list of flooded land cells (collect the indices once after floodFill) instead of the whole grid. In a coastal city, that's usually a quarter of the cells. And keep fimg between frames. Calling createImageData every frame is the first thing a profiler will point at.

If you stop here, you have a working, terrain-aware simulation. The rest is about making it look like footage instead of a diagram.

Step 6: Layer Video Textures With Canvas Compositing

Flat colour reads as a diagram. Real water has texture that moves. The cheapest way to get that is to drape a looping video of real water over the flooded area, and the restriction "over the flooded area only" comes free from two canvas compositing modes:

• source-atop draws the new image only where the destination already has pixels. Draw a video onto the flood canvas with source-atop and it appears on water only. Dry land stays untouched, because dry land is transparent.

• destination-in keeps the destination only where the new image has pixels. Draw a black-and-transparent mask onto a buffer with destination-in, and the buffer is cropped to the mask's shape.

You need a hidden <video> (muted, looping, and playsInline. You need all three, or mobile Safari refuses to autoplay) and one scratch canvas the size of the grid:

const vid = document.createElement("video");
vid.muted = true; vid.loop = true; vid.playsInline = true;
vid.setAttribute("muted", ""); vid.setAttribute("playsinline", "");
vid.src = "/fx/water.webm";
vid.style.cssText = "position:absolute;width:1px;height:1px;opacity:0;pointer-events:none";
document.body.appendChild(vid);
vid.load();

const buf = document.createElement("canvas");
buf.width = grid.W; buf.height = grid.H;
const bctx = buf.getContext("2d");

function drawWaterTexture(mud) {
  if (vid.readyState < 2) return;         // not enough data yet: skip, don't block
  if (vid.paused) vid.play().catch(() => {});

  // 1. video into the scratch buffer, stretched to the grid
  bctx.globalCompositeOperation = "source-over";
  bctx.clearRect(0, 0, grid.W, grid.H);
  bctx.drawImage(vid, 0, 0, grid.W, grid.H);

  // 2. tint it, only where the video is (everywhere, at this point)
  bctx.globalCompositeOperation = "source-atop";
  bctx.globalAlpha = 0.5 + 0.38 * mud;
  bctx.fillStyle = `rgb(${28 + 108 * mud | 0}, ${112 - 10 * mud | 0}, ${178 - 126 * mud | 0})`;
  bctx.fillRect(0, 0, grid.W, grid.H);
  bctx.globalAlpha = 1;

  // 3. crop the buffer to land cells, so the open sea keeps its own colour
  bctx.globalCompositeOperation = "destination-in";
  bctx.drawImage(landMask, 0, 0);

  // 4. onto the flood canvas, only where there's already water
  fctx.save();
  fctx.globalCompositeOperation = "source-atop";
  fctx.globalAlpha = 0.8;
  fctx.drawImage(buf, 0, 0);
  fctx.restore();
}

Call this right after fctx.putImageData(fimg, 0, 0) in the frame loop. Step 4 is where it pays off. Because the flood canvas is transparent wherever there's no water, source-atop puts the texture exactly inside the flood outline, following every bay and valley the BFS found, with no per-pixel work on your side.

landMask is a canvas you build once per run: white where sea[i] is false, transparent elsewhere. Without it the texture also covers the open sea, and the whole gulf turns the colour of the flood. I shipped that version for about an hour before I noticed.

The mud parameter is a time ramp from 0 to 1. Tsunami water arrives as seawater and turns dark brown as it picks up sediment off the land. Interpolating the tint from sea blue to mud brown over a few seconds after landfall gives you that for free.

The first time I built this, I had baked a brown grade into the video file itself, and no amount of tinting could make the water blue at the start. Keep the source footage neutral and do colour in code.

The same destination-in trick gives you a depth-driven darkening (a mask whose alpha is proportional to depth[i], filled with a dark colour) and a foam band along the moving front (a mask whose alpha is a Gaussian of reach[i] - F, filled with a second video of churning foam). Each is a few lines, and each one makes the water read a little more as a volume and a little less as a tint.

Step 7: Move the Camera Into the Scene

A top-down map is an illustration. Tilt the camera and the same pixels become a place. The flood canvas is draped on the ground, so it foreshortens correctly. Your only job is to point the camera somewhere worth looking at.

For a tsunami, that means standing on the shore near the city, looking out to sea, and watching it come in. "Near the city" is the coastline cell closest to the map centre. "Out to sea" is the average direction to nearby sea cells from that point. Compute it locally rather than for the whole coastline, or a curved bay will point your camera at the wrong horizon. I found that out with Odesa, where the citywide average pointed the camera at empty fields next to the airport.

function shoreCamera(grid, sea, center) {
  const { W, H, z, x0, y0, cellM } = grid;

  // 1. nearest coastline cell to the map centre
  let best = -1, bestD = Infinity;
  for (let i = 0; i < W * H; i++) {
    if (!sea[i]) continue;
    const x = i % W, y = (i / W) | 0;
    const coast = (x > 0 && !sea[i - 1]) || (x < W - 1 && !sea[i + 1]) ||
                  (y > 0 && !sea[i - W]) || (y < H - 1 && !sea[i + W]);
    if (!coast) continue;
    const lng = x2lon(x0 + (x + 0.5) / 256, z), lat = y2lat(y0 + (y + 0.5) / 256, z);
    const d = (lng - center.lng) ** 2 + (lat - center.lat) ** 2;
    if (d < bestD) { bestD = d; best = i; }
  }
  const sx = best % W, sy = (best / W) | 0;

  // 2. local seaward direction: average offset to sea cells within ~2.5 km
  const r = Math.max(6, Math.round(2500 / cellM));
  let vx = 0, vy = 0;
  for (let dy = -r; dy <= r; dy++) {
    for (let dx = -r; dx <= r; dx++) {
      const x = sx + dx, y = sy + dy;
      if (x < 0 || y < 0 || x >= W || y >= H || !sea[y * W + x]) continue;
      const len = Math.hypot(dx, dy);
      if (len < 1 || len > r) continue;
      vx += dx / len; vy += dy / len;
    }
  }
  const len = Math.hypot(vx, vy) || 1;
  const east = vx / len, north = -vy / len;   // grid y grows southward

  const shoreLng = x2lon(x0 + (sx + 0.5) / 256, z);
  const shoreLat = y2lat(y0 + (sy + 0.5) / 256, z);
  const mLng = 111320 * Math.cos(shoreLat * Math.PI / 180), mLat = 111320;

  return {
    // stand 900 m inland of the shore...
    center: [shoreLng - east * 900 / mLng, shoreLat - north * 900 / mLat],
    // ...facing the sea
    bearing: Math.atan2(east, north) * 180 / Math.PI,
  };
}

Then it's two easeTo calls: dive in before landfall, pull back out before the results appear.

const before = {
  center: map.getCenter(), zoom: map.getZoom(),
  pitch: map.getPitch(), bearing: map.getBearing(),
};
const cam = shoreCamera(grid, sea, map.getCenter());

map.easeTo({
  center: cam.center,
  zoom: Math.min(13.6, before.zoom + 2.8),
  pitch: 72,
  bearing: cam.bearing,
  duration: 1200,
});

setTimeout(() => map.easeTo({ ...before, duration: 2600 }), (T_LANDFALL + 3.2) * 1000);

Here is what those two calls produce. The camera has dropped to a steep pitch above the shore of Odesa and turned to face the Gulf. The sea pulls back first, exposing a wide band of sand, and then the front comes in over the beach and into the port. Everything you see moving is the same flood canvas from Step 3, foreshortened by the camera and draped over the terrain.

Notice that nothing in the frame loop knows or cares that the camera moved. The loop keeps writing the same pixels into the same canvas, and the map does the rest. That's what Step 3 bought you.

Where the Ground Layer Stops

Everything so far lives on the ground. The flood canvas is the map's own surface, and that's exactly why it works: perspective, rotation and terrain come free. The moment you need something with height, the trick runs out. You can't drape a tower onto a raster.

I learned where that line is the expensive way. I built a real 3D wall of water as a three.js custom layer, and it worked. I also cut it, for a reason worth knowing before you spend the time.

A 30-metre wave seen from 250 metres away covers about 7 degrees of a 37-degree field of view, a fifth of the screen. To make it fill the frame, you have to put the camera a hundred metres from it and nearly on the ground, where satellite imagery smears into mud. The geometry was correct and the shot was bad. Draped water under a pitched camera turned out to read as more water than a correctly-scaled wall did.

So the ground layer stayed a ground layer. But the earthquake scenario had no such escape: a building that collapses has to be an object. That is Part 2.

Part 2: Things with Height

For the earthquake scenario, I needed buildings to fall. This half is what I ended up with after roughly ten rounds of rebuilding it: a mix of map-native extrusions for the thousands of buildings that only need to stand there, and a three.js scene rendered into the map's own WebGL context for the ones that come down.

Both paths read the same vector tiles, share the same depth buffer and move together under every camera move.

Two Kinds of 3D, and Why You Need Both

A fill-extrusion layer is the map engine's own 3D. It's fast, it handles tens of thousands of buildings, it's lit consistently with the rest of the map, and it's driven by data expressions: height from a property, colour from a property, and, the part that matters here, height multiplied by a per-feature state you can change at runtime. What it can't do is move. An extrusion is always a vertical prism standing on the ground.

A custom layer gives you the map's WebGL context and projection matrix every frame and lets you draw whatever you want into it. With three.js on the other end, that means real objects: meshes with position, rotation, and scale, instanced by the thousand, writing into the same depth buffer as the map so that a falling block disappears correctly behind a building that's still standing.

The split that works is simple. Every building is an extrusion. When one is damaged, its extrusion shrinks (that's the per-feature state). When one collapses close enough to the camera that you'd see the difference, the extrusion shrinks to nothing and a three.js object takes its place for the two seconds of the fall. Then the object is replaced with a pile of rubble, also three.js, and the frame rate never notices.

Step 8: Get Buildings Out of the Map's Own Tiles

You don't need a building dataset. The map is already downloading one. Mapbox Streets vector tiles carry a building source layer with a polygon footprint, a height, a min_height for buildings that start above ground, and an extrude flag. querySourceFeatures reads whatever is currently loaded:

function buildingSource(map) {
  const sources = map.getStyle().sources;
  if (sources.composite) return "composite";         // Mapbox styles
  if (sources.openmaptiles) return "openmaptiles";   // MapLibre + OpenMapTiles
  for (const id in sources) if (sources[id].type === "vector") return id;
  return null;
}

function collectBuildings(map, center, kx, ky) {
  const src = buildingSource(map);
  if (!src) return [];
  const feats = map.querySourceFeatures(src, { sourceLayer: "building" });
  const out = [], seen = new Set();

  for (const f of feats) {
    const p = f.properties || {}, g = f.geometry;
    if (!g || p.extrude === "false" || p.underground === "true") continue;

    const polys = g.type === "Polygon" ? [g.coordinates]
                : g.type === "MultiPolygon" ? g.coordinates : null;
    if (!polys || !polys[0] || polys[0][0].length < 4) continue;

    // centroid and bounds of the outer ring
    const ring = polys[0][0], n = ring.length - 1;
    let sx = 0, sy = 0, x0 = Infinity, x1 = -Infinity, y0 = Infinity, y1 = -Infinity;
    for (let j = 0; j < n; j++) {
      const [x, y] = ring[j];
      sx += x; sy += y;
      if (x < x0) x0 = x; if (x > x1) x1 = x;
      if (y < y0) y0 = y; if (y > y1) y1 = y;
    }
    sx /= n; sy /= n;

    // the same building appears once per tile it touches
    const key = sx.toFixed(6) + "," + sy.toFixed(6) + "," + n;
    if (seen.has(key)) continue;
    seen.add(key);

    let h = +(p.height ?? p.render_height);
    if (!(h > 0)) h = 8;                              // no data: assume two storeys
    const mh = Math.min(h, +(p.min_height ?? p.render_min_height) || 0);

    out.push({
      geometry: g, polys, lng: sx, lat: sy, h, mh,
      d: Math.hypot((sx - center.lng) * kx, (sy - center.lat) * ky),   // metres from epicentre
      rf: 0.5 * Math.sqrt(Math.max(4, (x1 - x0) * kx * (y1 - y0) * ky)), // rough radius, metres
    });
  }
  out.sort((a, b) => a.d - b.d);
  return out.slice(0, 9000);
}

kx and ky are metres per degree of longitude and latitude at the city's latitude: 111320 * cos(lat) and 110574. Everything downstream works in metres from the epicentre.

Three things to know about this data. First, querySourceFeatures returns a building once for each tile it intersects, so you have to deduplicate. A rounded centroid plus vertex count makes a good key.

Second, the features have no stable IDs in Mapbox Streets. That matters enormously in Step 11.

Third, you only get what's loaded, which means you have to zoom in to where the buildings are before you collect them, and wait for the tiles to arrive.

In HazardSim, the camera dives in two steps, holds until map.areTilesLoaded() is true and the feature count stops changing between polls, and only then runs collectBuildings. If you collect too early, you get the ten buildings that happened to be in the first tile.

Step 9: Make Extrusions Look Like Buildings

The default extrusion is a flat-shaded box in one colour. It reads as a diagram. The trick that fixes it costs almost nothing: fill-extrusion-pattern takes an image from the style's sprite and tiles it over the walls, and you can add images to the sprite at runtime with map.addImage.

Draw a few façade textures on a canvas (rows of windows on a wall colour, with slight noise so they don't look computer-generated), register them, and pick one per building from its height:

function makeFacade(map, id, wall, glass) {
  const cv = document.createElement("canvas");
  cv.width = 128; cv.height = 64;
  const c = cv.getContext("2d");
  c.fillStyle = wall; c.fillRect(0, 0, 128, 64);

  // four floors of windows, 16px per floor
  for (let f = 0; f < 4; f++) {
    for (let x = 3; x + 6 <= 128; x += 12) {
      const lit = Math.random() < 0.07;
      c.fillStyle = lit ? "#c6b68c" : glass;
      c.fillRect(x, f * 16 + 3, 6, 9);
    }
  }
  // film grain
  const im = c.getImageData(0, 0, 128, 64), d = im.data;
  for (let i = 0; i < d.length; i += 4) {
    const n = (Math.random() - 0.5) * 12;
    d[i] += n; d[i + 1] += n; d[i + 2] += n;
  }
  map.addImage(id, im);
}

makeFacade(map, "fac-glass",  "#566a78", "#3c4e5c");
makeFacade(map, "fac-light",  "#cdc8be", "#364048");
makeFacade(map, "fac-brick",  "#8a5a44", "#2e2a26");

Then the layer. Height and base are expressions, and both are multiplied by a feature-state value k that defaults to 1. That multiplier is the whole collapse mechanism, so it goes in now:

const H  = ["coalesce", ["get", "height"], ["get", "render_height"], 8];
const MH = ["coalesce", ["get", "min_height"], ["get", "render_min_height"], 0];
const K  = ["coalesce", ["feature-state", "k"], 1];

function addBuildingLayers(map, source, sourceLayer, before) {
  const common = { type: "fill-extrusion", source, minzoom: 13 };
  if (sourceLayer) {
    common["source-layer"] = sourceLayer;
    common.filter = ["all", ["!=", ["get", "extrude"], "false"], ["!=", ["get", "underground"], "true"]];
  }
  map.addLayer({
    ...common, id: "bld-walls",
    paint: {
      "fill-extrusion-pattern": ["case",
        [">=", H, 45], "fac-glass",
        [">=", H, 14], "fac-light",
        "fac-brick"],
      "fill-extrusion-height": ["*", H, K],
      "fill-extrusion-base":   ["*", MH, K],
      "fill-extrusion-opacity": 1,
    },
  }, before);
  // a thin second extrusion on top gives the roof its own texture
  map.addLayer({
    ...common, id: "bld-roof",
    paint: {
      "fill-extrusion-pattern": "fac-roof",
      "fill-extrusion-height": ["+", ["*", H, K], 0.05],
      "fill-extrusion-base":   ["max", ["*", MH, K], ["-", ["*", H, K], 0.45]],
      "fill-extrusion-opacity": 1,
    },
  }, before);
}

before should be the ID of the first symbol layer in the style, so the buildings go under the labels. Find it with a loop over map.getStyle().layers.

A pattern can't be rotated or scaled per feature, so windows don't line up with floors exactly. At the camera heights this runs at, nobody can tell.

Step 10: Decide What Falls

Now the simulation part, which is short. For each building, you need the shaking intensity at its location and the probability that a building of that type collapses at that intensity.

Shaking is expressed on the Modified Mercalli scale, I to XII. A standard attenuation relation gives intensity from magnitude M and distance r in kilometres (depth is the hypocentre depth, 10 km for a shallow crustal quake):

const DEPTH_KM = 10;

function mmiAt(M, rKm) {
  const R = Math.sqrt(rKm * rKm + DEPTH_KM * DEPTH_KM);
  const v = 1.5 * M - 3.0 * Math.log10(R) - 0.0018 * R + 1.6;
  return Math.max(0, Math.min(12, v));
}

Collapse probability as a function of intensity is a fragility curve. The published ones are per building class and far more detailed than a simulation needs. This piecewise-linear version is tuned to look right and to give sane casualty numbers. Multiply it by a factor for construction quality and another for height. Mid-rise buildings fare worst, very tall ones are engineered and fare better:

const FRAG = [[6.5, 0], [7, 0.004], [8, 0.025], [9, 0.08], [10, 0.2], [11, 0.38], [12, 0.55]];
const QUALITY = { strict: 0.2, avg: 1, poor: 3 };

function collapseProbability(mmi) {
  if (mmi <= FRAG[0][0]) return 0;
  for (let i = 1; i < FRAG.length; i++) {
    if (mmi <= FRAG[i][0]) {
      const [m0, p0] = FRAG[i - 1], [m1, p1] = FRAG[i];
      return p0 + (p1 - p0) * (mmi - m0) / (m1 - m0);
    }
  }
  return FRAG[FRAG.length - 1][1];
}

function assignDamage(buildings, M, epicentre, quality) {
  const kx = 111320 * Math.cos(epicentre.lat * Math.PI / 180), ky = 110574;
  const q = QUALITY[quality] || 1;

  for (const b of buildings) {
    const rKm = Math.hypot((b.lng - epicentre.lng) * kx, (b.lat - epicentre.lat) * ky) / 1000;
    const mmi = mmiAt(M, rKm);
    const hf = b.h < 5 ? 0.7 : b.h < 35 ? 1.15 : b.h < 70 ? 0.8 : 0.45;
    const pCollapse = Math.min(0.9, collapseProbability(mmi) * q * hf);
    const pDamage = mmi >= 6 ? Math.min(0.6, pCollapse * 2.5 + 0.004) : 0;

    const r = Math.random();
    b.cat = r < pCollapse ? 2 : r < pCollapse + pDamage ? 1 : 0;   // 2 collapse, 1 damaged, 0 fine
    b.start = 0.8 + Math.random() * 2.8;                            // seconds after the shock
  }
}

The per-building randomness is what sells it. Two identical towers next to each other, one falls and one stands. That's also how real earthquakes look.

Step 11: Animate Collapse Without Rebuilding Layers

Here's the mechanism from Step 9 paying off. Each frame, for every building that's falling, set its feature-state k to a number between 1 and its final rubble height. The extrusion shrinks. No layer is rebuilt, no GeoJSON is re-set, and no tiles are re-parsed.

map.setFeatureState({ source: "bld", id: b.id }, { k: 0.62 });

There's one catch, and it's the one that cost me the most time in this whole project: setFeatureState needs a feature ID, and Mapbox Streets buildings don't have one. Calling it against the vector source silently does nothing.

The fix is to copy the buildings you collected into your own GeoJSON source, with an id you control, and tell the source to promote that property to the feature ID:

const feats = buildings.map((b, i) => {
  b.id = i;
  return { type: "Feature", properties: { i, height: b.h, min_height: b.mh }, geometry: b.geometry };
});

map.addSource("bld", {
  type: "geojson",
  data: { type: "FeatureCollection", features: feats },
  promoteId: "i",
  maxzoom: 16,
});
addBuildingLayers(map, "bld", null, firstSymbolLayer);

Now setFeatureState({ source: "bld", id: 17 }, ...) reaches building 17. Keep the original tile-backed layers around until the GeoJSON source reports isSourceLoaded, then remove them, so there's no frame where the city blinks.

The shape of k over time is what makes a collapse look like a collapse rather than a building sliding into the ground. I ended up with five modes, chosen at random per building and constrained by height:

// k0 = starting height multiplier (1), kf = final rubble height, u = progress 0..1
function collapseK(b, u) {
  const k0 = b.k0, sp = k0 - b.kf;
  switch (b.mode) {
    case 1: {   // floor by floor: ns steps, each with a fast drop then a pause
      const s = u * b.ns, i = Math.min(b.ns - 1, Math.floor(s));
      let f = s - i;
      f = f < 0.45 ? f * f / 0.45 * 1.1 : 0.55 + (f - 0.45) * 0.45 / 0.55;
      return k0 - sp * Math.min(1, (i + Math.min(1, f)) / b.ns);
    }
    case 2: {   // soft storey: ground floor gives, building drops one floor, pauses, then goes
      const sd = Math.min(sp * 0.5, 3.2 / b.h);
      if (u < 0.12) return k0 - sd * (u / 0.12) ** 2;
      if (u < 0.5)  return k0 - sd;
      const v = (u - 0.5) / 0.5;
      return k0 - sd - (sp - sd) * v * v;
    }
    case 3: {   // crushed top: upper part fails first
      const pt = sp * 0.45;
      if (u < 0.3)  return k0 - pt * (u / 0.3) ** 2;
      if (u < 0.55) return k0 - pt;
      const v = (u - 0.55) / 0.45;
      return k0 - pt - (sp - pt) * v * v;
    }
    case 4:     // sideways: steady, slightly accelerating
      return k0 - sp * Math.pow(u, 1.6);
    default:    // pancake: accelerating free fall
      return k0 - sp * u * u;
  }
}

The frame loop is then a few lines per building:

function frame(now) {
  const el = (now - T0) / 1000, dt = Math.min(0.04, (now - last) / 1000);
  last = now;

  for (const b of active) {
    if (b.done || el < b.start) continue;
    const u = Math.min(1, (el - b.start) / b.dur);
    const k = collapseK(b, u);
    map.setFeatureState({ source: "bld", id: b.id }, { k });
    if (u >= 1) { b.done = true; onLanded(b); }
  }
  requestAnimationFrame(frame);
}

For a damaged-but-standing building (cat === 1), kf is close to 1 and the mode is a short settle: the building visibly drops half a metre and stops. It's a small detail, and it's the one that makes the whole street look hit rather than just the buildings that fell.

Clamp dt to 40 ms. If the tab stutters and a frame takes a quarter of a second, a building that was mid-fall would jump instead of fall, and the eye catches that instantly.

Step 12: A Three.js Scene Inside the Map

Shrinking extrusions handle the buildings in the distance. For the ones near the camera, you want to see the structure break: a tower tipping over and snapping into blocks, debris arcing out from the base, or rubble left on the street. That needs objects, so this is where three.js comes in.

The custom layer gets two callbacks: onAdd once, with the map and its GL context, and render every frame, with the context and the map's projection matrix. The non-obvious part is coordinates. You want to work in metres from an origin, not in Mercator units, so build a matrix that converts, and multiply it into the map's projection each frame:

function makeScene(map, origin, before) {
  const mc = mapboxgl.MercatorCoordinate.fromLngLat([origin.lng, origin.lat], 0);
  const s = mc.meterInMercatorCoordinateUnits();
  // metres east / north / up at the origin  ->  Mercator.  Mercator y grows southward, hence -s.
  const toMercator = new THREE.Matrix4()
    .makeTranslation(mc.x, mc.y, mc.z)
    .scale(new THREE.Vector3(s, -s, s));

  const scene = new THREE.Scene();
  const camera = new THREE.Camera();
  scene.add(new THREE.HemisphereLight(0xfff4e4, 0x3c342d, 0.55));
  scene.add(new THREE.AmbientLight(0xffffff, 0.1));
  const sun = new THREE.DirectionalLight(0xfff8ee, 0.8);
  sun.position.set(-0.45, -0.6, 1);
  scene.add(sun);

  let renderer = null;
  const layer = {
    id: "debris3d",
    type: "custom",
    renderingMode: "3d",
    onAdd(m, gl) {
      renderer = new THREE.WebGLRenderer({ canvas: m.getCanvas(), context: gl, antialias: true });
      renderer.autoClear = false;             // the map has already drawn this frame
      renderer.localClippingEnabled = true;   // used to sink buildings below ground
    },
    render(gl, matrix) {
      camera.projectionMatrix.fromArray(matrix).multiply(toMercator);
      renderer.resetState();                  // Mapbox changed GL state since our last draw
      renderer.render(scene, camera);
    },
  };
  map.addLayer(layer, before);
  return { scene, camera, layer };
}

On MapLibre, the second argument of render is an object, and the matrix is at args.defaultProjectionData.mainMatrix. Everything else is the same.

renderingMode: "3d" is what makes the map share its depth buffer with you. A block of concrete that lands behind a standing extrusion is hidden by it, which is the entire reason to render into the map's context rather than in a separate canvas on top.

Thousands of debris pieces means instancing. HazardSim keeps a pool of InstancedMesh objects, one per kind of piece (slabs, wall fragments, rebar, rocks, roof prisms), each with a fixed capacity, and hands out instance indices as pieces are created:

function makePool(scene, geometry, material, capacity) {
  const mesh = new THREE.InstancedMesh(geometry, material, capacity);
  // allocate the colour buffer for the full capacity up front (see Gotchas)
  mesh.instanceColor = new THREE.InstancedBufferAttribute(new Float32Array(capacity * 3).fill(1), 3);
  mesh.count = 0;
  mesh.frustumCulled = false;                 // the map camera is not a three.js camera
  mesh.instanceMatrix.setUsage(THREE.DynamicDrawUsage);
  scene.add(mesh);

  const dummy = new THREE.Object3D(), color = new THREE.Color();
  dummy.rotation.order = "ZXY";
  return {
    n: 0, capacity,
    add(p) {
      if (this.n >= capacity) return -1;
      const i = this.n++;
      this.set(i, p);
      color.setRGB(p.rgb[0], p.rgb[1], p.rgb[2]);
      mesh.setColorAt(i, color);
      mesh.instanceColor.needsUpdate = true;
      mesh.count = this.n;
      return i;
    },
    set(i, p) {
      dummy.position.set(p.x, p.y, p.z);
      dummy.rotation.set(p.tx, p.ty, p.yaw);
      dummy.scale.set(p.sx, p.sy, p.sz);
      dummy.updateMatrix();
      mesh.setMatrixAt(i, dummy.matrix);
      mesh.instanceMatrix.needsUpdate = true;
    },
  };
}

const concrete = new THREE.MeshStandardMaterial({ map: concreteTexture, roughness: 1 });
const pools = {
  slab: makePool(scene, new THREE.BoxGeometry(1, 1, 1), concrete, 30000),
  wall: makePool(scene, new THREE.BoxGeometry(1, 1, 1), facadeMaterial, 1500),
  bar:  makePool(scene, new THREE.BoxGeometry(1, 1, 1), new THREE.MeshStandardMaterial({ color: 0x6e4c38, metalness: 0.25 }), 20000),
  rock: makePool(scene, rockGeometry(), concrete, 9000),
};

Thirty thousand slabs sounds like a lot, but it's one draw call.

One more thing the map needs from you: it only redraws when something changes. Your scene changing is not, from the map's point of view, something changing. Call map.triggerRepaint() at the end of every frame in which you moved anything, or your meshes will update in memory and never on screen.

Step 13: From Footprint to a Box That Tips Over

The vector tile gives you a footprint polygon. To tip a building over, you need a box: a centre, an orientation, and two side lengths. The oriented bounding box is a few lines:

function orientedBox(ring, kx, ky, origin) {
  // ring in lng/lat -> local metres
  const loc = ring.map(([lng, lat]) => [(lng - origin.lng) * kx, (lat - origin.lat) * ky]);

  // the longest edge sets the orientation
  let ang = 0, best = 0;
  for (let i = 1; i < loc.length; i++) {
    const ex = loc[i][0] - loc[i - 1][0], ey = loc[i][1] - loc[i - 1][1];
    if (ex * ex + ey * ey > best) { best = ex * ex + ey * ey; ang = Math.atan2(ey, ex); }
  }

  // project every vertex onto that axis and its normal, take the extents
  const ca = Math.cos(ang), sa = Math.sin(ang);
  let u0 = Infinity, u1 = -Infinity, v0 = Infinity, v1 = -Infinity;
  for (const [x, y] of loc) {
    const u = x * ca + y * sa, v = -x * sa + y * ca;
    if (u < u0) u0 = u; if (u > u1) u1 = u;
    if (v < v0) v0 = v; if (v > v1) v1 = v;
  }
  const cu = (u0 + u1) / 2, cv = (v0 + v1) / 2;
  return {
    x: cu * ca - cv * sa, y: cu * sa + cv * ca,    // centre, metres from origin
    ang,
    L1: Math.max(2, u1 - u0), L2: Math.max(2, v1 - v0),
  };
}

The box becomes a stack of two to four blocks, one Mesh each, in a Group whose origin is on the bottom edge the building will rotate around. The physics is a rigid body falling over: angular acceleration proportional to g / h · sin θ, with a small initial nudge so it starts moving. When the lean passes a threshold, the upper blocks detach and fly ballistically with the velocity they had at that instant:

function startTip(scene, box, h, onLand) {
  const n = Math.max(2, Math.min(4, Math.round(h / 9))), hs = h / n;
  const geo = new THREE.BoxGeometry(box.L1, box.L2, hs);
  const group = new THREE.Group();
  group.position.set(box.x + Math.cos(box.ang) * box.L1 / 2, box.y + Math.sin(box.ang) * box.L1 / 2, 0);
  group.rotation.z = box.ang;
  scene.add(group);

  const segs = [];
  for (let i = 0; i < n; i++) {
    const m = new THREE.Mesh(geo, wallMaterial);
    m.frustumCulled = false;
    group.add(m);
    segs.push({ m, free: false, gone: false, x: -box.L1 / 2, z: hs * (i + 0.5), th: 0, px: 0, pz: 0 });
  }

  const tip = { th: 0.03, w: 0.22, broke: false, bend: 0.1 + Math.random() * 0.14, brk: 0.4 + Math.random() * 0.3 };

  function pose() {
    let bx = -box.L1 / 2, bz = 0;
    for (let j = 0; j < n; j++) {
      const s = segs[j];
      if (s.free) continue;
      const th = tip.th * (1 + tip.bend * j);        // upper blocks bend a little further
      const sn = Math.sin(th), cs = Math.cos(th);
      if (j === 0) { bx = -box.L1 / 2 * cs; bz = box.L1 / 2 * sn; }
      s.px = s.x; s.pz = s.z;
      s.x = bx + sn * hs / 2; s.z = bz + cs * hs / 2; s.th = th;
      bx += sn * hs; bz += cs * hs;
      s.m.position.set(s.x, 0, s.z);
      s.m.rotation.y = th;
    }
  }

  return function tick(dt) {
    tip.w = Math.min(2.2, tip.w + 1.5 * 9.8 / h * Math.sin(tip.th) * dt);
    tip.th += Math.min(0.09, tip.w * dt);          // never more than ~5 degrees per frame
    pose();

    if (!tip.broke && tip.th * (1 + tip.bend * (n - 1)) > tip.brk && n > 1) {
      tip.broke = true;
      for (let j = 1; j < n; j++) {
        const s = segs[j];
        s.free = true;
        s.vx = Math.max(-18, Math.min(18, (s.x - s.px) / Math.max(dt, 0.008)));
        s.vz = Math.max(-18, Math.min(18, (s.z - s.pz) / Math.max(dt, 0.008)));
        s.w = tip.w * (1 + tip.bend * j) * (1.05 + Math.random() * 0.25);
      }
    }

    let left = 0;
    for (const s of segs) {
      if (s.gone) continue;
      if (s.free) {
        s.vz -= 9.8 * dt; s.x += s.vx * dt; s.z += s.vz * dt; s.th += s.w * dt;
        s.m.position.set(s.x, 0, s.z);
        s.m.rotation.y = s.th;
      }
      const landed = s.free ? s.z < Math.min(hs, box.L1) * 0.5 + 0.3 : tip.th >= 1.25;
      if (landed) {
        s.gone = true;
        group.remove(s.m);
        onLand(group.position.x + Math.cos(box.ang) * s.x, group.position.y + Math.sin(box.ang) * s.x);
      } else left++;
    }
    if (!left) { scene.remove(group); geo.dispose(); return true; }   // finished
    return false;
  };
}

The clamps aren't decoration. Velocity at detachment is limited to 18 m/s and the frame step dt to a minimum of 8 ms, because the velocity is computed from a position difference, and a hiccup frame would otherwise launch a block into orbit. Angular velocity is limited to 2.5 rad/s and the lean to 0.09 rad per frame for the same reason. Every one of those numbers was added after watching a building do something a building can't do.

onLand is where a collapse starts to look like one: each block that hits the ground spawns a burst of debris and dust at that point, and leaves a pile.

Step 14: Debris and Dust

Debris is the instanced pools from Step 12 plus gravity. When a collapsing building passes a milestone (the first drop, each floor in floor-by-floor mode, a block landing), throw a handful of pieces out from the edges of the footprint, with velocity pointing away from the centre:

function throwDebris(pools, b, origin, kx, ky, top, n) {
  const ring = b.polys[0][0];
  const cx = (b.lng - origin.lng) * kx, cy = (b.lat - origin.lat) * ky;
  for (let i = 0; i < n; i++) {
    // a random point on a random edge of the footprint
    const e = 1 + Math.floor(Math.random() * (ring.length - 1)), t = Math.random();
    const a = ring[e - 1], z = ring[e];
    const x = ((a[0] + (z[0] - a[0]) * t) - origin.lng) * kx;
    const y = ((a[1] + (z[1] - a[1]) * t) - origin.lat) * ky;
    // outward, with a little spread
    const ox = x - cx, oy = y - cy, ol = Math.hypot(ox, oy) || 1, v = 2 + Math.random() * 6;
    const size = 0.3 + Math.random() * 1.1;
    const idx = pools.rock.add({
      x, y, z: top * (0.35 + Math.random() * 0.65),
      yaw: Math.random() * 6, tx: Math.random() * 6, ty: Math.random() * 6,
      sx: size, sy: size * 0.8, sz: size * 0.6, rgb: concreteColour(),
    });
    if (idx >= 0) flying.push({ idx, x, y, z: top, vx: ox / ol * v, vy: oy / ol * v, vz: Math.random() * 3, spin: (Math.random() - 0.5) * 8 });
  }
}

function tickDebris(pools, dt) {
  for (let i = flying.length - 1; i >= 0; i--) {
    const p = flying[i];
    p.vz -= 9.8 * dt;
    p.x += p.vx * dt; p.y += p.vy * dt; p.z += p.vz * dt;
    p.tx += p.spin * dt;
    if (p.z <= p.sz * 0.35) { p.z = p.sz * 0.35; flying.splice(i, 1); }   // landed: stays as rubble
    pools.rock.set(p.idx, p);
  }
}

Pieces that land simply stop being updated and stay in the pool. That's your rubble, for free.

HazardSim adds a few more things on top: stacked floor slabs on a 9-metre grid where a building stood, wall fragments with windows, bent rebar, a low fill-extrusion mound under the pile so the ground isn't flat, and for the closest buildings, a set of photogrammetry-scanned rubble models. Those are polish, and the pool plus gravity gets you most of the way.

Dust is the one place where I draw in screen space, and it's worth saying why that's fine here when it was fatal for water in Part 1. Dust particles are small, there are hundreds of them, and each lives three to eight seconds. They're also drawn as soft sprites with per-particle position, size, and rotation projected from a map point each frame. There are no edges to see and nothing to line up with. The screen-space rule is about large, persistent things with shape.

The important decision is where dust comes from. Early versions had a constant haze over the city and burning fires, and it looked like a screensaver. What reads as real is dust that comes only from events: a puff when a floor drops, a surge along the ground when a building lands, a jet from the ground floor of a soft-storey collapse. Each event emits a burst with a size proportional to the building's footprint and a vertical speed proportional to the drop, then nothing until the next event.

function emitDust(b, level, n, surge) {
  for (let i = 0; i < n && dust.length < MAX_DUST; i++) {
    const a = Math.random() * Math.PI * 2;
    const speed = surge ? 5 + Math.random() * 8 : 1 + Math.random() * 3;
    dust.push({
      lng: b.lng, lat: b.lat,                                        // anchor on the map
      x: Math.cos(a) * b.rf * 0.6, y: Math.sin(a) * b.rf * 0.6,     // offset, metres
      z: surge ? Math.random() * 5 : level * (0.2 + Math.random() * 0.8),
      vx: Math.cos(a) * speed, vy: Math.sin(a) * speed,
      vz: surge ? 0.3 + Math.random() * 1.2 : 1.5 + Math.random() * 3,
      size: (surge ? b.rf * 0.5 + 6 : b.rf * 0.6 + 4) * (0.7 + Math.random() * 0.6),
      grow: 3 + Math.random() * 5,
      t: 0, life: (surge ? 5 : 3.5) + Math.random() * 2.5,
      sprite: sprites[Math.floor(Math.random() * sprites.length)],
      alpha: 0.5 + Math.random() * 0.3,
    });
  }
}

Per frame, each particle decelerates, drifts with the wind, grows, and is projected: map.project([lng + x / kx, lat + y / ky]) gives the screen position of its ground point, and the height is applied as y -= z * pixelsPerMetre * sin(pitch). Sort by screen depth, draw back to front with drawImage, and fade with age. It's a few hundred drawImage calls per frame on a 2D canvas over the map, and it costs almost nothing.

Step 15: The Camera Has to Shake

Everything so far could be running silently in a corner of the screen. What makes a viewer feel the earthquake is the camera.

Two things. First, the dive: from the overview, ease to zoom 14.6 at 45 degrees, hold until tiles are loaded, then ease to zoom 16.2 at 62 degrees over the epicentre. The hold isn't optional. If you dive straight to zoom 16, the vector tiles for that zoom haven't arrived, querySourceFeatures returns almost nothing, and the earthquake hits an empty city.

function diveToCity(map, center, then) {
  const bearing = map.getBearing();
  map.easeTo({ center, zoom: 14.6, pitch: 45, bearing: bearing - 10, duration: 1100, essential: true });
  const t1 = Date.now();
  (function hold() {
    const loaded = !map.isMoving() && map.areTilesLoaded();
    if (Date.now() - t1 > 1000 && (loaded || Date.now() - t1 > 2000)) {
      map.easeTo({ center, zoom: 16.2, pitch: 62, bearing: bearing - 25, duration: 1500, essential: true });
      map.once("moveend", then);
    } else setTimeout(hold, 150);
  })();
}

Second, the shake. Don't move the map camera for this. A camera that jitters through easeTo fights the animation system and loads tiles for every jitter. Shake the container with a CSS transform. The map doesn't know it moved:

function shake(durationMs, magnitude) {
  const el = map.getContainer();
  const w = window.innerWidth, h = window.innerHeight;
  const amp = Math.min(28, w * 0.02, Math.max(6, (magnitude - 4) * 4.2));
  const dir = Math.random() * Math.PI;
  const t0 = performance.now();
  let last = t0, phase = 0;

  function tick(now) {
    const k = (now - t0) / durationMs;
    if (k >= 1) { el.style.transform = ""; return; }
    const dt = Math.min(0.05, (now - last) / 1000); last = now;

    // builds up over the first third, then decays; frequency rises with it
    let env, omega;
    if (k < 0.35) { const u = k / 0.35; env = 0.15 + 0.85 * u * u * (3 - 2 * u); omega = 7 + 39 * u * u; }
    else          { const v = (k - 0.35) / 0.65; env = Math.pow(1 - v, 1.4); omega = 46 - 18 * v; }
    phase += omega * dt;

    const a = amp * env;
    const osc = Math.sin(phase) * 0.78 + (Math.random() - 0.5) * 0.44 * Math.min(1, omega / 30);
    const x = Math.cos(dir) * osc * a, y = Math.sin(dir) * osc * a * 0.7;
    el.style.transform = `translate(${x.toFixed(1)}px, ${y.toFixed(1)}px) rotate(${(osc * a * 0.012).toFixed(3)}deg) scale(${(1 + 2.4 * amp / Math.min(w, h) * env).toFixed(4)})`;
    requestAnimationFrame(tick);
  }
  requestAnimationFrame(tick);
}

The envelope matters more than the amplitude. Real shaking builds for a second or two, peaks, and dies off over several more, with the frequency rising as it builds. A constant jitter reads as a broken video. An aftershock at about half the amplitude, three or four seconds after the main shock, finishes the effect.

When the shaking stops, ease the camera down to the centre of the nearest collapses and hold for a few seconds so the rubble gets its moment, then pull back up to the intensity rings and the casualty figures.

Keeping It Fast

A few decisions kept this at 60 fps on a laptop and usable on phones.

1. Pools with Fixed Capacity

Every instanced pool has a cap, and add returns -1 when it's full. The caps scale down for low-power devices: 9,000 slabs instead of 30,000, 3,000 rocks instead of 9,000. The scene degrades to fewer pieces rather than to a slideshow.

2. A Device Tier, Decided Once

Mobile user agent, navigator.deviceMemory < 6, hardwareConcurrency <= 4, or a short screen side under 700 px puts the device in the low tier. The low tier skips the three.js tipping entirely and uses only shrinking extrusions, caps dust at a third, and drops the hero models.

3. Rebuild the Rubble Mound at Most Every 1.2 Seconds

The low fill-extrusion mound under each pile is a GeoJSON source, and setData on a GeoJSON source re-parses it. Doing that once per collapsed building was the main cause of stutter in early versions. Now landed buildings go into a pending list that's flushed on a timer.

4. Never Remove a Collapsed Building from the Source

It stays in bld with k set to 0.001. Removing features means setData, and setData means a stall.

5. Load Assets on Intent

The textures and models for the full-detail mode add up to 24 MB. They start downloading when the visitor clicks the earthquake tab, not on page load, and only in the full tier.

What Went Wrong, and Why

Each of these cost me real time, and the first two are why Part 1 exists at all.

The Sticker Problem

My first version of a "wall of water" was a video clip drawn on a screen-space canvas, positioned by projecting a map point each frame. On a 13-inch laptop it almost worked. On a 4K monitor it was a rectangle with soft edges hovering over the city, obviously not part of the scene.

No amount of tuning opacity, feathering, or size fixes this, because the problem isn't the sprite. A sprite has no place in a 3D world. Everything in Steps 3 to 6 is the alternative.

If you do need screen-space elements, say a splash sprite anchored to a point, keep them small, short-lived, and numerous, and accept that they'll never carry the shot.

I Couldn't Find My Layer, Twice

The 3D water wall from the end of Part 1 rendered without an error, the mesh moved exactly where my debug probe said it should, and nothing appeared on screen. I'd added it with map.addLayer(layer, "flood"), which inserts it below the flood raster, and the opaque water was hiding my wall. Check map.getStyle().layers before you check anything else.

The second time I lost a layer, it was the opposite problem: it wasn't there at all. map.getStyle().layers doesn't list custom layers. Check with map.getLayer("debris3d") instead. I spent a while convinced the layer wasn't being added.

24 Levels of Transparency

I built sprite sheets with soft alpha edges and ran them through PIL.Image.quantize() to shrink the PNG. The palette compression quantized the alpha channel along with the colours. 256 levels became 24, the smooth falloff became a staircase, and every step was a hard straight edge that looked exactly like a tile boundary. I spent two rounds hunting for a tiling bug that didn't exist.

The fix was to stop quantizing and make the RGB channels constant (the sprites were white anyway), so PNG compression had nothing left to do. The file got three times smaller. I now check alpha with a histogram instead of trusting my eyes.

Same Filename, Different File

I re-encoded flood.webm with a different grade and redeployed. Visitors kept getting the old one for hours. The CDN cached by URL, and a ?v=2 query string didn't bust it for a file that already existed.

Renaming the file to flood2.webm fixed it in one deploy. Since then I rename every binary asset I update instead of versioning the URL.

Headless Screenshots Lied, Twice

I tested under headless Chrome with a software GL renderer. Each screenshot took a second or two to capture, so a "sequence" of frames was nothing of the sort, and I spent an afternoon tuning timings against noise.

Two things fixed it: record the browser session as video and cut frames with ffmpeg, and, for colour, read the simulation canvas directly with getImageData instead of measuring the composited screen. The second one sounds obvious written down, but it took me an afternoon to arrive at it.

The same headless setup fooled me again on the earthquake. Satellite tiles load slowly there, and for a long time I thought my building layer was broken because screenshots showed a black city. It was the base map that hadn't arrived. If a layer looks wrong in headless tests, wait for map.areTilesLoaded() before trusting the screenshot.

Nothing Happened When I Set Feature State

setFeatureState against the tile source did nothing and reported nothing, because the features have no IDs. The fix is the GeoJSON copy with promoteId from Step 11. It took me an embarrassingly long time to find, because the API doesn't fail. It just doesn't do anything.

Every Instance Was Black

InstancedMesh.setColorAt in three r128 lazily creates the colour buffer the first time you call it, sized to the mesh's current count. My pools started with count = 0, so the buffer was created with zero entries, and every piece rendered black. Allocate instanceColor yourself, for the full capacity, before the first setColorAt. That's the line in makePool.

Half the Debris Vanished When I Panned

three.js frustum-culls meshes against its own camera. Our camera is a bare THREE.Camera with a projection matrix stuffed into it each frame and no idea where the map is looking, so its idea of the frustum is nonsense. Set frustumCulled = false on every mesh in the custom layer.

The Scene Updated But the Screen Didn't

The map only repaints when it thinks something changed. Call map.triggerRepaint() after you move your objects. Without it the animation runs at whatever rate the map happens to repaint for other reasons, which in a still scene is zero.

A Sky Made Everything Disappear

On a different project with the same setup, map.setSky(...) hid the whole three.js scene. The sky writes to the depth buffer, and everything the custom layer draws afterwards fails the depth test. If you render 3D into the map, don't enable the sky layer.

The idle Event Never Fired in a Background Tab

I preloaded models on map.on("idle"). In a background tab the map can go a very long time without becoming idle, and visitors who opened the site in a tab and came back later got an earthquake with no models. The preload moved to the moment the hazard type is selected.

Conclusion

The whole guide reduces to two structures.

On the ground:

1. Align a grid to the map's tile pyramid, so every tiled dataset drops in without resampling.

2. Decode free elevation tiles into that grid.

3. Register a canvas as a map source over the grid's rectangle, with animate: true.

4. Run the simulation on the grid once. Animate by writing pixels every frame.

5. Use source-atop and destination-in to drape textures exactly inside the simulated shape.

6. Move the camera. The drawing comes along.

With height:

1. Pull buildings from the vector tiles the map already loaded. Deduplicate, as they have no IDs.

2. Draw them all as fill-extrusion with a per-feature height multiplier, through a GeoJSON copy with promoteId so feature state works.

3. Decide damage from an attenuation relation and a fragility curve, with randomness per building.

4. Animate collapse by writing the multiplier each frame.

5. For the buildings near the camera, render three.js into the map's own GL context with renderingMode: "3d", so the depth buffer is shared.

6. Turn footprints into oriented boxes, tip them over, snap them into blocks, and throw instanced debris. Emit dust only from events. Shake the container, not the camera.

The idea under all of it is the one I kept relearning: anything that belongs to the world has to be drawn in the world's coordinates. A canvas on top of the map is a sheet of glass. A canvas inside the map is the ground, and a three.js scene inside the map's GL context is the air above it.

Every improvement that made either simulation look real came from moving something out of the first category and into the second.

The other thing I'd tell myself at the start: the map engine and three.js are not competitors. The map is better at ten thousand static buildings than anything you'll write, and three.js is better at ten moving ones than any expression language.

Give each the job it's good at, and spend your effort on the seam between them, which is almost entirely the coordinate transform in Step 12 and the feature-state trick in Step 11.

You can see both scenarios at hazardsim.com. Pick a coastal city and launch a tsunami, or pick any city, set a magnitude, and press the button. The code in this guide is a cleaned-up version of what runs there. Thanks to Tim for breaking it in ways I never would have.

The first deadline of the 2026-27 tax year, 7 August, has already passed, and the second lands on 7 November. Each of those summaries is a single API call from software to HMRC, and getting that call right is what this tutorial is about.

You can follow it without any earlier reading. The one-time setup every MTD integration needs (a sandbox application, an OAuth 2.0 access token, and fraud prevention headers) is summarised in the quick recap below, together with the small request helper every snippet uses.

If you want that setup in depth, I walked through it step by step in a previous freeCodeCamp tutorial.

By the end, you'll know how to find the business you're filing for, work out which period is due, build a cumulative summary HMRC accepts, submit it, and read the tax calculation that follows. The code is Node and TypeScript, trimmed from the HMRC integration I built for a Making Tax Digital app.

Table of Contents

• Quick Recap: The Setup This Tutorial Assumes

• How Quarterly Updates Work in MTD

• Step 1: How to Find the Business ID

• Step 2: How to Find Out What's Due

• Step 3: How to Build the Cumulative Summary

• Step 4: How to Submit the Update

• Step 5: How to Trigger and Read a Tax Calculation

• Common Gotchas

• Where to Go Next

Quick Recap: The Setup This Tutorial Assumes

Before you can file anything, your application needs the same one-time setup as every MTD integration. If you already have it, skip to the next section. Otherwise, here is the short version:

1. Register a sandbox application on the HMRC Developer Hub and subscribe it to the four APIs used here: Business Details, Obligations, Self Employment Business, and Individual Calculations. An API you haven't subscribed to returns 403 Forbidden, which looks like an authorisation problem but is not.

2. Create a sandbox test user with HMRC's Create Test User API. It gives you a fake taxpayer with a National Insurance number (NINO) and Government Gateway credentials.

3. Get an access token through HMRC's OAuth 2.0 authorization code flow, asking for the read:self-assessment and write:self-assessment scopes and signing in as your test user on HMRC's consent screen.

4. Send fraud prevention headers on every call. HMRC makes them mandatory and the exact set depends on how your application connects, so build them once in a getFraudHeaders(req) function.

Every snippet in this tutorial calls one small helper. It sets the bearer token, pins the API version through the Accept header (each HMRC API has its own version, and the wrong one returns 406 Not Acceptable), and attaches your fraud prevention headers:

import axios from 'axios';

const HMRC_BASE_URL = 'https://test-api.service.hmrc.gov.uk'; // sandbox

async function request(method, path, accessToken, req, data = null, apiVersion = '2.0') {
  const headers = {
    Authorization: 'Bearer ' + accessToken,
    Accept: 'application/vnd.hmrc.' + apiVersion + '+json',
    ...getFraudHeaders(req),
  };
  // Only set a JSON Content-Type when there is a body. HMRC's edge rejects
  // a bodyless GET that carries one with a 403.
  if (data !== null && data !== undefined) {
    headers['Content-Type'] = 'application/json';
  }
  const res = await axios({ baseURL: HMRC_BASE_URL, method, url: path, headers, data });
  return res.data;
}

The req argument is your user's incoming request (an Express Request in my code), which is where the client details in the fraud prevention headers come from. With a token and this helper in place, you're ready to file.

How Quarterly Updates Work in MTD

A quarterly update isn't a tax return. It's a set of running totals: how much the business has earned and spent so far this tax year, grouped into the same categories Self Assessment already uses.

The word that matters is "so far". Each update is cumulative. It covers everything from the start of the tax year to the end of the current update period, not just the last three months. GOV.UK's guidance on sending quarterly updates sets out the standard periods and their deadlines:

That design has a pleasant side effect: if a user spots a mistake in an earlier quarter, the next update simply carries the correction. HMRC's end-to-end service guide puts it plainly: each update invalidates the previous one, because every period starts on 6 April.

Customers whose accounting year runs from 1 April can opt into calendar periods instead (1 April to 30 June, and so on), with the same deadlines. You don't need to hardcode either set, because the Obligations API returns the exact dates.

HMRC calls each required update an "obligation": four per tax year for each self-employment or property business, plus the annual tax return.

Here is the whole journey you're about to build:

Step 1: How to Find the Business ID

Every self-employment endpoint needs a businessId, HMRC's identifier for one income source. A sole trader who also lets a property has two. You get them from the Business Details API with the customer's National Insurance number (NINO):

// GET /individuals/business/details/{nino}/list  (Business Details API v2.0)
const result = await request(
  'GET',
  '/individuals/business/details/' + nino + '/list',
  accessToken,
  req,
  null,
  '2.0',
);

const businesses = result.listOfBusinesses ?? [];
const soleTrade = businesses.find((b) => b.typeOfBusiness === 'self-employment');
const businessId = soleTrade?.businessId;

The array lives under listOfBusinesses, and each entry carries a typeOfBusiness (self-employment, uk-property, foreign-property, or property-unspecified), the businessId, and optionally a tradingName. Sandbox self-employment IDs look like XBIS12345678901.

HMRC's guide recommends storing the ID rather than looking it up before every call, and I do. The list changes rarely, typically when a customer adds or ceases a business, so refresh it when they connect and when they ask.

Step 2: How to Find Out What's Due

Next, ask the Obligations API which periods are still open. This endpoint is on version 3.0:

// GET /obligations/details/{nino}/income-and-expenditure  (Obligations API v3.0)
const raw = await request(
  'GET',
  '/obligations/details/' + nino + '/income-and-expenditure?status=open',
  accessToken,
  req,
  null,
  '3.0',
);

The response groups obligations by business, with the dates nested one level down:

{
  "obligations": [
    {
      "typeOfBusiness": "self-employment",
      "businessId": "XBIS12345678901",
      "obligationDetails": [
        {
          "periodStartDate": "2026-04-06",
          "periodEndDate": "2026-10-05",
          "dueDate": "2026-11-07",
          "status": "open"
        }
      ]
    }
  ]
}

That nesting gets awkward quickly in a UI, so I flatten it into one row per obligation, lift the business fields onto each row, and pick the open one with the earliest due date:

function flattenObligations(raw, businessId) {
  return (raw.obligations ?? [])
    .filter((group) => group.businessId === businessId)
    .flatMap((group) =>
      (group.obligationDetails ?? []).map((d) => ({
        businessId: group.businessId,
        periodStartDate: d.periodStartDate,
        periodEndDate: d.periodEndDate,
        dueDate: d.dueDate,
        status: (d.status ?? '').toLowerCase() === 'fulfilled' ? 'fulfilled' : 'open',
      })),
    );
}

const next = flattenObligations(raw, businessId)
  .filter((o) => o.status === 'open')
  .sort((a, b) => a.dueDate.localeCompare(b.dueDate))[0];

Three details are worth knowing before you build a screen on this.

First, these obligations have no period key. The start and end dates identify a period, and they're exactly what you send back in step 3. If you need a stable key, derive one from the two dates.

Second, read dueDate from the response instead of calculating it. HMRC sets these dates and has changed them before, so let the API be the source of truth rather than a date rule in your code.

Third, the filters have rules. fromDate and toDate must be sent together and at most 366 days apart, and a businessId filter also requires typeOfBusiness. Fetching everything open and filtering in your own code, as above, sidesteps both.

Step 3: How to Build the Cumulative Summary

Now for the payload. The Self Employment Business API calls it a cumulative period summary, and it has four parts:

• periodDates: required. The start and end of the period you're reporting, copied from the obligation.

• periodIncome: turnover (takings, fees, and sales), other business income, and taxTakenOffTradingIncome.

• periodExpenses: either one consolidatedExpenses figure or an itemised breakdown.

• periodDisallowableExpenses: the part of each itemised expense that can't be claimed for tax.

Here's a complete, valid body for the second quarter of 2026-27, using a single consolidated expenses figure:

{
  "periodDates": {
    "periodStartDate": "2026-04-06",
    "periodEndDate": "2026-10-05"
  },
  "periodIncome": {
    "turnover": 28450,
    "other": 0
  },
  "periodExpenses": {
    "consolidatedExpenses": 4310.45
  }
}

The itemised form swaps that figure for categories that mirror the Self Assessment boxes, such as costOfGoods, carVanTravelExpenses, adminCosts, and professionalFees. If part of an expense was personal, the disallowable portion goes in the matching field:

"periodExpenses": {
  "costOfGoods": 2100,
  "carVanTravelExpenses": 1640.2,
  "adminCosts": 185.99
},
"periodDisallowableExpenses": {
  "carVanTravelExpensesDisallowable": 410.05
}

You can't mix the two forms. Sending consolidatedExpenses alongside any itemised field returns RULE_BOTH_EXPENSES_SUPPLIED. And because each disallowable field pairs with an itemised category, disallowable expenses belong with the itemised form.

So when is the consolidated form allowed? HMRC's service guide says customers with annual turnover under £90,000 can report a single expenses total. Because every update is cumulative, the practical check is to compare year-to-date turnover with the threshold on each submission. Once it reaches £90,000, the update has to be itemised. I enforce that on the server before anything goes to HMRC:

const CONSOLIDATED_EXPENSES_TURNOVER_LIMIT = 90_000;

function checkConsolidatedExpensesLimit({ turnover, usesConsolidatedExpenses }) {
  if (!usesConsolidatedExpenses) return null;
  if ((turnover ?? 0) >= CONSOLIDATED_EXPENSES_TURNOVER_LIMIT) {
    return (
      'Consolidated expenses are not permitted once turnover reaches £90,000. ' +
      'Please enter your expenses itemised instead.'
    );
  }
  return null;
}

Finally, every amount allows at most two decimal places, so round your totals (I use Number(total.toFixed(2))) before they leave your code.

Step 4: How to Submit the Update

With the body built, the submission itself is one PUT. The tax year goes in the path in YYYY-YY form, and the period dates travel in the body, not the URL:

// PUT /individuals/business/self-employment/{nino}/{businessId}/cumulative/{taxYear}
// Self Employment Business API v5.0
await request(
  'PUT',
  '/individuals/business/self-employment/' + nino + '/' + businessId + '/cumulative/' + taxYear,
  accessToken,
  req,
  summary,
  '5.0',
);

Success is 204 No Content, with no receipt number, so keep your own audit record of what you sent, when, and the status HMRC returned. You can read HMRC's copy back at any time with a GET on the same path.

Because it's a PUT, the same call creates and amends. Resubmitting for the same tax year replaces what HMRC holds. That's the cumulative model working as designed, and also the source of the most expensive bug in this flow (see the gotchas).

This endpoint only accepts tax years from 2025-26 onwards. Older tutorials that POST to a /period endpoint describe the earlier, per-period model.

Step 5: How to Trigger and Read a Tax Calculation

After an update, customers want to know roughly how much tax they're heading for. The Individual Calculations API answers that asynchronously: you trigger a calculation, get back an ID, and fetch the result a little later. The calculation type goes at the end of the path, and in-year is the one you want after a quarterly update:

// POST /individuals/calculations/{nino}/self-assessment/{taxYear}/trigger/in-year
// Individual Calculations API v8.0. Returns 202 Accepted with a calculationId.
const { calculationId } = await request(
  'POST',
  '/individuals/calculations/' + nino + '/self-assessment/' + taxYear + '/trigger/in-year',
  accessToken,
  req,
  {}, // an empty object, not null (see the gotchas)
  '8.0',
);

HMRC's endpoint documentation recommends waiting at least five seconds before you try to retrieve the result. Until the calculation is ready, the retrieve endpoint returns 404 Not Found, so a short, bounded retry loop handles it:

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function waitForCalculation(nino, taxYear, calculationId, accessToken, req) {
  const path =
    '/individuals/calculations/' + nino + '/self-assessment/' + taxYear + '/' + calculationId;
  await sleep(5000); // HMRC recommends waiting at least five seconds after the trigger

  for (let attempt = 0; attempt < 5; attempt += 1) {
    try {
      return await request('GET', path, accessToken, req, null, '8.0');
    } catch (err) {
      // Only a 404 means "not ready yet". Anything else is a real error, so rethrow it.
      if (err.response?.status !== 404 || attempt === 4) throw err;
    }
    await sleep(1000 * (attempt + 1));
  }
}

The full result covers every kind of income HMRC knows about, but a quarterly update screen needs only a few fields. Here's an abridged response with illustrative figures:

{
  "metadata": {
    "calculationId": "f2fb30e5-4ab6-4a29-b3c1-c7264259ff1c",
    "taxYear": "2026-27",
    "calculationType": "in-year",
    "periodFrom": "2026-04-06",
    "periodTo": "2026-10-05"
  },
  "calculation": {
    "taxCalculation": {
      "totalIncomeTaxAndNicsDue": 3902.6
    }
  }
}

calculation.taxCalculation.totalIncomeTaxAndNicsDue is the headline number. metadata.periodTo shows how far into the year the figures run: the end of the latest submission, not the date you asked.

If the submitted data fails HMRC's checks, there's no calculation at all. Instead, messages.errors holds a list of { id, text } objects, and you should show that text to the customer so they can fix their records.

Before you show any in-year figure, HMRC's minimum functionality standards require a disclaimer. The tax calculations section of HMRC's service guide offers this wording, or you can write your own:

"This calculation is only based on information HMRC have received about your income and expenses to 20XX-XX-XX. This may change as we receive further information about you during the tax year."

Fill the date from periodTo. It also helps to say that this is an estimate: HMRC's guide is explicit that the customer doesn't need to pay anything at this point.

Common Gotchas

1. The Default Sandbox Data Is Years Out of Date

Call the Obligations API in the sandbox with no test scenario and you get static data from an old tax year, which the cumulative endpoint rejects with RULE_TAX_YEAR_NOT_SUPPORTED. Send Gov-Test-Scenario: DYNAMIC instead to get open obligations for the current year.

The cumulative endpoint isn't stateful by default either: a GET after your PUT returns canned figures. For a true round trip, use the STATEFUL scenario with a business created through the Self Assessment Test Support API.

2. A New Update Replaces the Old One

Since each PUT replaces the year's figures, a form that opens blank and submits only what the user typed can wipe out earlier quarters. Prefill it from the retrieve endpoint or your own records, so every resubmission starts from the full year-to-date totals.

3. Send Zeros, Don't Leave Fields Out

HMRC's endpoint documentation says submissions must include income and expense values even when they're zero, so a quarter with no income still sends turnover and other as 0. GOV.UK also says customers must send an update when nothing happened in the period, so don't block a submission because every figure is zero.

4. You Can't File Too Early

The cumulative endpoint rejects a submission made more than 10 days before the period ends (RULE_EARLY_DATA_SUBMISSION_NOT_ACCEPTED), and an end date earlier than one already submitted (RULE_SUBMISSION_END_DATE_CANNOT_MOVE_BACKWARDS). Show the period end date clearly and enable the submit button only when it applies.

5. "Fulfilled" Can Take an Hour

After a successful 204, HMRC can take up to an hour to mark the obligation as fulfilled, so an immediate re-read still shows it as open. Trust your own record of the submission and tell the customer the status will update shortly.

6. Trigger the Calculation With an Empty Object

In my testing, posting the trigger with a null body produced a 500 from HMRC's calculation backend, while an empty JSON object ({}) was accepted. Watch the version too: Individual Calculations 9.0 is in the sandbox, but 8.0 is the version available in production at the time of writing.

Where to Go Next

You now have the complete quarterly loop: find the business, read its open obligations, build a cumulative summary that respects the consolidated-expenses rule, submit it, then trigger, wait for, and read the calculation with the right disclaimer.

After the fourth update, the year ends in the same Individual Calculations API: you trigger an intent-to-finalise calculation instead of in-year, show the customer the result, and submit their final declaration against that calculation ID.

That deserves a tutorial of its own, and it's the one I plan to write next.

One practical note before you plan a launch: HMRC's API pages currently carry a notice that it's no longer accepting production credential requests for new 2026-27 quarterly update products. The sandbox remains open, so you can build and test everything above today, but read that notice on the Self Employment Business API page before you commit to a go-live date.

I work on TapTax, a Making Tax Digital app for UK sole traders, which is where the code in this article comes from.

About the author: Solomon Amos is the founder of TapTax and built its HMRC Making Tax Digital integration. He has spent the last three years as a lead technical architect on HMRC digital modernisation programmes, and holds a PhD in engineering, with research in machine learning. You can find him on LinkedIn.

But learning Node.js can feel overwhelming. With so many APIs, packages, tools, and concepts to understand, it’s easy to feel lost.

That’s why this book focuses on the fundamental Node.js concepts you need to build practical applications without unnecessary distractions. You’ll learn how Node.js works, how to use its built-in APIs, and how to create applications that interact with files, URLs, events, HTTP requests, and more.

You’ll also learn how Express.js simplifies many common server-side tasks. We’ll explore routes, routers, controllers, views, and other concepts that help you organize and build web applications more efficiently.

Whether you’re learning backend development for the first time or expanding your JavaScript skills beyond the browser, this guide is designed to give you a clear and practical foundation for building with Node.js and Express.js.

Table of Contents

• What You Should Already Know

• The Tools You'll Need

• What Exactly Is Node.js?

  • Create a New Directory for Your Project

  • Create apackage.jsonFile

• Why Node.js?

  • 1. Create a JavaScript File

  • 2. Write your JavaScript Program

  • 3. Run your JavaScript Program

  • 4. Automatically Rerun Your JavaScript Program

• Node's Module Support

• Create HTTP Servers with Node.js

  • Initialize a New Instance of the HTTPServerObject

  • Configure the System's Port Where the Server Should Run

  • Configure the Server to Respond to Client Requests

• What Exactly Is Express.js?

  • 1. Install Express

  • 2. Use Express.js to Configure the Project's Node.js Web Server

  • 3. Run the Express.js Web Server

• Raw Node.js Code vs. Express.js Code

  • Example 1: Use raw Node.js to Handle Web Requests

  • Example 2: Use Express.js to Handle Web Requests

• Important Stuff to Know About Creating Express.js Web Servers in Node.js

  • It's Common to Callapp.listen()Last

  • You Can Set the Port Number Using Environment Variables

  • Common Express.js Response Methods

  • Express's Response Methods Do Not Terminate the HTTP Request Handler's Execution

• What Is a Route in Express.js?

  • Syntax of a Route in Express

  • Middleware vs. Route Handler

  • Route Examples in Express.js

  • Important Things to Know About thenextParameter

  • Categories of Middleware in Express.js

• What Is the express.Router() Method?

  • Create a Directory for the Routers

  • Create the Route Files

  • Create a Mini-App for the Book-Related Routes and Middleware

  • Time to Practice with Routers

  • Update the Main Express Application

  • Run your Express.js Application

  • Important Things to Know About Express Routers

• What Is a View in Express.js?

  • What Are Static Views in Express.js?

  • What Are Dynamic Views in Express.js?

• Overview

  • Dive Deeper into Node.js and Express.js

What You Should Already Know

This guide is for readers who know basic JavaScript and want to start building web applications with Node.js and Express.js. You'll get the most value from it if you're familiar with:

• JavaScript fundamentals, including variables, functions, arrays, and objects.

• Basic asynchronous JavaScript, including callbacks, promises, and async/await.

• Basic HTML, including links, forms, and input elements.

• Using a terminal to navigate folders and run commands.

You don't need previous experience with Node.js or Express.js. Familiarity with installing npm packages is helpful, but the examples will guide you through the commands you need.

The Tools You'll Need

Before you begin, have a code editor, a web browser, and the following installed:

• Node.js 24.15.0 or later.

• npm 11.14.0 or later.

Check your installed versions by running:

node --version && npm --version

Node.js includes npm, but the bundled version may be older than the one listed here. Check both versions and update npm separately if needed. The CodeSweetly package manager guide explains how to install, update, and check these tools.

The file-creation commands assume an environment with mkdir and touch available, such as Bash or zsh on macOS or Linux, Git Bash on Windows, or a Linux shell through Windows Subsystem for Linux (WSL). You can also create the files and folders directly in your editor.

Let's get started with Node.js.

What Exactly Is Node.js?

Node.js is not a framework or language. Instead, it is an asynchronous, event-driven environment that includes tools such as:

• JavaScript engine (Google's V8)

• Module system (CommonJS and ES modules)

• Operating system APIs (os)

• Filesystem access (fs)

• Network servers (http, net, https)

• Memory management tools

• Event loop

These components make up Node.js's core infrastructure, enabling JavaScript to run outside of browsers.

The image above illustrates Node.js as a well-equipped runtime environment providing the infrastructure to build and run any scale of JavaScript applications outside the browser, including servers, CLIs, scripts, and full-stack applications.

Let's discuss the points highlighted in the image by building applications with Node.js and Express.js. To begin, create the project directory.

Create a New Directory for Your Project

Use the mkdir CLI command to create a new project directory as follows:

mkdir codesweetly-nodejs-app-001

Note: You can use any name you prefer. In this guide, we'll use codesweetly-nodejs-app-001 for demonstration.

Afterward, navigate to your project directory using the command line.

cd codesweetly-nodejs-app-001

Create a package.json File

Use npm to initialize a package.json file after navigating into the project directory.

npm init -y

Next, use the following command to delete the optional main field from package.json:

npm pkg delete main

Tip: The main field in package.json identifies a package's entry point when another program loads the package. We do not need it for this project because we'll run our scripts directly. Leaving it in place is also harmless.

Why Node.js?

JavaScript was originally created for web browsers, but other environments also provide the infrastructure to execute it.

Node's 2009 release provided an environment for running JavaScript outside the web browser. For example, let's create a script to run from our system's terminal.

1. Create a JavaScript File

Create the JavaScript file you want Node to run.

In a shell that supports touch, such as Bash or zsh, use the following command. Otherwise, create the file in your code editor:

touch console.js

2. Write your JavaScript Program

Open the newly created JavaScript file and write your program:

console.log("=== Hello from the CodeSweetly Team! ===");
console.log("We hope you have fun Coding Sweetly with Node.js.");
console.log("Thank you for being part of the CodeSweetly community.");
console.log("=== Keep coding. Keep creating. Keep shipping. ===");

3. Run your JavaScript Program

Installing Node.js makes the node command available for running a Node application from your command line.

node console.js

• node: The command for running Node.js scripts or the REPL.

• console.js: The JavaScript file you want Node to run.

You can also add the command to the "scripts" field of your project's package.json file:

{
  "scripts": {
    "start": "node console.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  }
}

With this script in place, you can run your JavaScript program from your terminal like this:

npm run start

Once you execute your script, Node will print the file's output to your terminal. It will look like this:

$ npm run start

> codesweetly-nodejs-app-001@1.0.0 start
> node console.js

=== Hello from the CodeSweetly Team! ===
We hope you have fun Coding Sweetly with Node.js.
Thank you for being part of the CodeSweetly community.
=== Keep coding. Keep creating. Keep shipping. ===

As you can see, we've successfully executed JavaScript outside a web browser. That's precisely what Node.js helps us with. It is an asynchronous, event-driven runtime environment for running JavaScript code outside the web browser. You can also automate rerunning your program. Let's discuss how.

4. Automatically Rerun Your JavaScript Program

By default, Node.js requires you to manually rerun your JavaScript file each time you make changes.

Repeating the manual process of executing your application as you make changes can be burdensome. Luckily, Node provides the --watch flag for automating the process:

node --watch filename.extension

The snippet above uses the --watch flag to start Node.js in watch mode. This causes Node to re-execute the specified script whenever you update any of the files in its dependency graph.

• node: The command for running Node.js scripts or the REPL.

• --watch: The flag for activating Node's watch mode.

• filename.extension: The script you want Node to execute.

Tip: Stop the running process using Ctrl + C on Windows, macOS, or Linux.

The --watch flag, by default, watches the entry point and all the modules it depends on. In other words, the watch command causes Node to monitor:

• The entry file (filename.extension)

• All files imported by the entry file

• All files imported in those imports

• And so on through the whole dependency graph

Node's default watch mode does not watch unrelated files. If you want Node to watch other files, use the --watch-path flag:

node --watch-path=./src --watch-path=./tests filename.extension

The --watch-path flag, in the snippet above, tells Node to watch all files in the src and tests directories, even if they are not in the entry point's dependency graph.

• node: The command for running Node.js scripts or the REPL.

• --watch-path: The flag for specifying the path you want Node to watch for changes.

• filename.extension: The script you want Node to execute.

Tip:

• The --watch-path flag causes Node to ignore the dependency graph's modules unless they are part of --watch-path. Instead, Node.js will watch only the exact paths you specified, and nothing else.

• --watch-path enables watch mode itself. If you also pass --watch, Node still watches only the specified paths.

• --watch-path is supported on macOS and Windows, but not Linux. On Linux, use --watch to monitor the entry file and its dependencies.

• Watch commands do not reload the browser automatically. They simply detect file changes, so Node reruns the program you specified. As such, if your program sends data to browsers, you will need to refresh the browser yourself or use a third-party tool for auto-reloading.

Node.js supports two module systems. Let's learn about them.

Node's Module Support

Node.js supports both CommonJS (.cjs) and ECMAScript (.mjs) modules. This allows you to use your preferred module type to create Node.js applications.

For example, below is a script.mjs JavaScript file. Node will treat it as an ECMAScript module because it has a .mjs file extension.

script.mjs

import http from "node:http";

On the other hand, Node will regard the script.cjs JavaScript file below as a CommonJS module because it has a .cjs file extension.

script.cjs

const http = require("node:http");

Suppose you want to specify the module type for the .js files in your project. In that case, specify a type field in your package.json file like so:

package.json

{
  "scripts": {
    "start": "node console.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "type": "module",
  "license": "ISC"
}

The "type": "module" field in the snippet above makes Node treat .js files governed by this package.json as ES modules. A nested package.json establishes its own scope. .mjs and .cjs files retain their respective module types.

Set the "type" field to "commonjs" to make Node treat .js files in that scope as CommonJS.

Some notes:

• The ES module system is the official standard for JavaScript.

• Kevin Dangoor started the project that became CommonJS in January 2009, before JavaScript had an official standard module system.

• ES modules became part of the ECMAScript standard in 2015. CommonJS remains supported in Node.js.

In this guide, we'll mainly use Node.js with ES modules, since ES modules are JavaScript's standard module system. To follow the examples, make sure the "type" field in your package.json file is set to "module". You can do this by running the following command from your project directory:

npm pkg set type=module

When relevant, we'll compare CommonJS syntax.

Let's now discuss using Node.js to create web servers that receive and respond to requests from browsers.

Create HTTP Servers with Node.js

An HTTP server lets your JavaScript application handle requests from clients, such as browsers, and send responses.

There are three main steps to configuring an app's HTTP server:

1. Create a new instance of the HTTP Server object.

2. Specify the system's port where you want the server to run.

3. Define how the server should respond to client requests.

Let's discuss the three steps in detail. To start, create an ES module for your project.

touch server.js

Afterward, open the newly created module and initialize a new instance of the HTTP Server object.

Initialize a New Instance of the HTTP Server Object

Node provides the createServer method for creating an instance of the HTTP Server object (http.Server).

You can use it in your project by importing it from Node's http module as follows:

server.js

import { createServer } from "node:http";

const server = createServer();

Here's what's going on:

• import statement: Imports the createServer API from Node's http library to the server.js ES module.

• server: A variable for storing the HTTP Server object that the createServer() function outputs.

Here's the CommonJS alternative:

server.cjs

const http = require("node:http");

const server = http.createServer();

Once you have created the local HTTP Server object, specify the server's port.

Configure the System's Port Where the Server Should Run

The HTTP Server object provides a listen() method to configure the port on which the server should run and accept client requests.

Syntax

import { createServer } from "node:http";

const server = createServer();

server.listen(port, hostname, backlog, callback);

This form of the listen() method accepts the following arguments:

• port: (number) The port number where the server should run and listen for client requests. If omitted, the operating system will assign any unused port. You can use server.address().port to retrieve the port the server is listening on after it starts listening for client connections.

• hostname: (string) The hostname or IP address on which the server should accept client connections. If omitted, the server will default to either an unspecified IPv6 (::) address or an IPv4 (0.0.0.0) address. These unspecified addresses bind to all network interfaces for the applicable address family. You can use server.address().address to retrieve the bound IP address once it starts listening for client connections.

• backlog: (number) Maximum length of the pending connections' queue. 511 is the default value.

• callback: (function) The function to execute once the server starts listening for client requests.

Example

server.js

import { createServer } from "node:http";

const server = createServer();

server.listen(3000, "127.0.0.1", 511, () => {
  const info = server.address();
  console.log(`Server running at http://${info.address}:
${info.port}/`);
});

Here's what's going on:

• import statement: Imports the createServer API from Node's http library to the server.js ES module.

• server: A variable for storing the HTTP Server object that the createServer() function outputs.

• server.listen(): Starts the server to listen for client connections. (Tip: The method emits a listening event once the server starts successfully.)

Here's the CommonJS alternative:

server.cjs

const http = require("node:http");

const server = http.createServer();

server.listen(3000, "127.0.0.1", 511, () => {
  const info = server.address();
  console.log(`Server running at http://${info.address}:
${info.port}/`);
});

What is the 127.0.0.1 address?

The 127.0.0.1 address is an IPv4 loopback address that refers to your local computer. The hostname localhost also refers to your local computer, but it may resolve to 127.0.0.1 or the IPv6 loopback address, ::1. You can use it as follows:

server.js

import { createServer } from "node:http";

const server = createServer();

server.listen(3000, "localhost", 511, () => {
  console.log("Server running at http://localhost:3000/");
});

If you run this server and visit its web address, the browser will wait for a response because we have not yet set up a request handler. Let's configure that now.

Configure the Server to Respond to Client Requests

The createServer() method accepts a requestListener callback that is invoked automatically whenever the server receives an HTTP request. Node allows you to use this callback to respond to requests.

Syntax

The createServer() method accepts two optional arguments. Here's the syntax:

import { createServer } from "node:http";

const server = createServer(options, callback);

• options: An object for customizing the server's behavior.

• callback: The requestListener function for handling and responding to client requests. It accepts two parameters:

import { createServer } from "node:http";

const server = createServer(options, function (request, response) {
  // the requestListener function's body
});

• request parameter: An http.IncomingMessage object that provides details about the client request.

• response parameter: An http.ServerResponse object for responding to the client requests.

Providing the requestListener callback function as createServer's second argument causes Node.js to automatically register it as a listener for the "request" event. So, the syntax above is equivalent to:

import { createServer } from "node:http";

const server = createServer(options);

server.on("request", function (request, response) {
  // the requestListener function's body
});

server.on("request", callback) tells the server to listen for a request event and execute the callback on such an event.

Example

server.js

// Add the Node.js HTTP module
import { createServer } from "node:http";

// Specify the hostname and port to run the server
const hostname = "localhost";
const port = 3000;

// Create a new Server instance with a requestListener callback
const server = createServer((req, res) => {
  res.statusCode = 200; // Set an OK success (200) response status code
  res.setHeader("Content-Type", "text/plain"); // Define the media type of the response data
  res.end("Hello World!"); // Specify the response data and close the response stream
});

// Run the server on the specified port and hostname
server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:
${port}/`);
});

The snippet above used the requestListener callback function's response parameter to configure the server to respond to client requests. Here's the server.on("request", callback) alternative:

server.js

// Add the Node.js HTTP module
import { createServer } from "node:http";

// Specify the hostname and port to run the server
const hostname = "localhost";
const port = 3000;

// Create a new Server instance
const server = createServer();

// Listen for a request event and execute the requestListener callback
server.on("request", (req, res) => {
  res.statusCode = 200; // Set an OK success (200) response status code
  res.setHeader("Content-Type", "text/plain"); // Define the media type of the response data
  res.end("Hello World!"); // Specify the response data and close the response stream
});

// Run the server on the specified port and hostname
server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:
${port}/`);
});

Now that your server is set up and the port configured, you can run the application.

node server.js

• node: The command for running Node.js scripts or the REPL.

• server.js: The JavaScript file you want Node to run.

While the server is running, if users request the app's resource at the server's web address, they will see a Hello World! response as illustrated in the following image.

Tip: Stop the running process using Ctrl + C on Windows, macOS, or Linux.

Although the node:http module is Node.js's native API for handling HTTP requests, developers typically use frameworks to simplify the process. Some popular Node.js web frameworks include Express.js, Fastify, and Koa.js. Let's use Express.js as an example to see how a framework can simplify HTTP request handling in Node.

What Exactly Is Express.js?

Express.js is a web framework that simplifies server-side HTTP request handling in Node.js. It extends Node's HTTP module with built-in features such as routing and middleware, helping developers to build web applications and APIs efficiently without repetitive code. Here's how to use it in your Node.js project.

1. Install Express

npm install express@5.2.1

2. Use Express.js to Configure the Project's Node.js Web Server

Open the JavaScript server file and use Express.js to set up a web server:

server.js

// Add the Express module
import express from "express";

// Create a new Express application instance
const app = express();

// Set the port number where the server will run and listen for requests
const port = 3000;

// Create a route handler for GET requests to the "/" path
app.get("/", (req, res) => res.send("Hello, world!"));

// Run the server on the specified port
app.listen(port, () => {
  console.log(`Server running at http://localhost:${port}`);
});

Here's what the code above does:

• Creates an Express application instance to access Express APIs, such as routers and middleware, simplifying HTTP request handling in Node.js.

• Uses Express's app.get() method to define how the server handles HTTP GET requests to the / route.

• Uses Express's res.send() method to send an HTTP response and automatically set the appropriate headers based on the data type provided.

• Uses Express's app.listen() method to start the Node.js HTTP server on a specific port and execute a callback when the server begins listening for client requests.

Tip:

• In the statement app.get("/", (req, res) => res.send("Hello, world!")),

  • app.get() is the Express routing method.

  • "/" is the route path.

  • (req, res) => ... is the route handler function.

  • The entire statement is the route (or route definition).

• The objects passed to an Express route handler's parameters (req and res) are enhanced versions of Node's http.IncomingMessage and http.ServerResponse objects. Express adds properties and helper methods to simplify handling HTTP requests and responses.

• The order of route definitions matters. Express processes matching routes in the order they are defined. A handler can end the response or pass control to another handler using next().

3. Run the Express.js Web Server

Use the node command followed by the script's filename to run a Node.js script, including one that uses Express.js to handle HTTP requests.

node server.js

Open http://localhost:3000 in your browser to see Hello, world!.

Tip: Stop the server using Ctrl + C on Windows or Control + C on macOS.

Raw Node.js Code vs. Express.js Code

While you can use Node's native HTTP API to handle web requests, it requires manual configuration, such as inspecting each request's method and URL, setting response headers, and converting objects to JSON.

Express abstracts repetitive request-handling logic into declarative methods, simplifying Node.js web server development.

Below are two examples comparing Node.js web server code written with the native HTTP module and with Express methods.

Example 1: Use raw Node.js to Handle Web Requests

The following web server uses Node's built-in HTTP module to handle GET requests to two endpoints—home (/) and books (/books)—and return a 404 response for unmatched requests.

server.js

import { createServer } from "node:http";
const port = 3000;

const server = createServer((req, res) => {
  if (req.method === "GET" && req.url === "/") {
    res.writeHead(200, { "Content-Type": "text/plain" });
    res.end("Welcome to the homepage!");
  } else if (req.method === "GET" && req.url === "/books") {
    const books = [
      { id: 1, name: "Code React Sweetly" },
      { id: 2, name: "Creating NPM Package" },
    ];
    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify(books));
  } else {
    res.writeHead(404, { "Content-Type": "text/plain" });
    res.end("Page not found");
  }
});

server.listen(port, () => {
  console.log(`Raw Node server running at http://localhost:${port}`);
});

The snippet above uses Node's native http module to build a web server that accepts and responds to browser connections. In this raw Node.js example, you manually handle the following tasks:

• Routing to the correct endpoint: if (req.method === "..." && req.url === "...")

• Response header configuration: res.writeHead(...)

• JSON formatting: JSON.stringify(...)

Now, let's see how to build a similar web server using Express. Run these examples one at a time, replacing the contents of server.js and restarting the server.

Example 2: Use Express.js to Handle Web Requests

The following web server uses Express routes to handle GET requests to two endpoints—home (/) and books (/books)—and return a 404 response for unmatched requests.

server.js

import express from "express";
const app = express();
const port = 3000;

app.get("/", (req, res) => {
  res.send("Welcome to the homepage!");
});

app.get("/books", (req, res) => {
  const books = [
    { id: 1, name: "Code React Sweetly" },
    { id: 2, name: "Creating NPM Package" },
  ];
  res.json(books);
});

app.use((req, res) => {
  res.status(404).send("Page not found");
});

app.listen(port, () => {
  console.log(`Express server running at http://localhost:${port}`);
});

The snippet above uses Express.js to build a web server that accepts and responds to browser connections. Express simplifies the following tasks:

• Routing logic: Express.js provides simple route declarations like app.get(), app.post(), and app.delete(), eliminating the need for complex conditional ogic to check req.url and req.method for each route.

• Response header configuration: Express's res.send() method automatically sets appropriate response headers based on the response body. Use res.status() when you need to set a status code, as in the 404 handler above.

• JSON formatting: Express's res.json() method automatically converts JavaScript objects into valid JSON responses, handling serialization and headers for you.

Important Stuff to Know About Creating Express.js Web Servers in Node.js

Keep the following key points in mind when creating Express.js servers in your Node.js project.

It's Common to Call app.listen() Last

It's common to call app.listen() last in an Express.js app. This way, all your routes, middleware, and error handlers are set up before the server starts listening for requests.

You Can Set the Port Number Using Environment Variables

To read the port number from an environment variable, replace the existing port declaration with the following:

const port = process.env.PORT || 3000;

The above snippet instructs Node to use the PORT environment variable's value, or default to 3000 if it's unset or empty. In a shell such as Bash, the following command sets PORT to 8000 for this run of the server:

PORT=8000 node server.js

Common Express.js Response Methods

Here are some common response methods you'll use in Express.js:

• res.send(): Send an HTTP response to the client and automatically set the appropriate response headers based on the data type provided as the method's argument.

• res.json(): Send JSON responses to the client and automatically set the response's Content-Type header to application/json.

• res.redirect(): Redirect the client's request to a different URL.

• res.render(): Render a template engine's view and send the resulting HTML string to the client.

• res.status(): Set the response's HTTP status code without ending the request-response cycle. You can chain other response methods to it, for example, res.status(404).send("404 Error: Page not found"). You do not need to call this method to use the default status code of 200.

• res.end(): End the response. When called without a data argument, it sends no additional response body data. It can also accept data to send before ending the response.

Express's Response Methods Do Not Terminate the HTTP Request Handler's Execution

While response methods like res.send() close the HTTP request-response cycle, they do not end the execution of the route handler function.

Here's an example:

app.get("/", (req, res) => {
  // This sends a text response to the client and ends the request-response cycle, but does not end the route handler's execution
  res.send("Hello, client!");

  // This logs the string to the console because the function is still running
  console.log("Hello, devs!");

  // This will cause an error because the request-response cycle is closed, so you cannot send additional responses to the client during the callback's execution
  res.send("Hello, client again!");
});

The console.log() statement in the example above works because the request handler is still running. The res.send("Hello, client!") line ends the response to that request. It does not stop the callback's execution.

Note that the app.get(...) code in the snippet above is called an Express route. But what exactly is a route in Express.js? Let's discuss it now.

What Is a Route in Express.js?

A route in Express.js consists of an HTTP method, a URL path, and a chain of one or more middleware functions for processing client requests to an endpoint.

Tip:

• An endpoint consists of an HTTP method and a URL path. (Endpoint = HTTP method + URL path)

• The route includes an HTTP method, a URL path, and the server-side logic that handles client requests. (Route = Server HTTP method + Server URL path + handler function)

Syntax of a Route in Express

app.METHOD(PATH, HANDLER);

• app: An Express application instance that provides access to Express APIs, including methods and middleware, to simplify HTTP request handling in Node.js.

• METHOD: A lowercase HTTP request method, such as get, post, or delete.

• PATH: The URL path the route will handle. It can be a string or a regular expression. Express 4 also supports string patterns whose syntax differs in Express 5.

• HANDLER: The callback (or middleware) function that Express executes when the request and route endpoints correspond. The handler may be:

  • A single function

  • More than one function

  • An array of functions

  • A combination of an array of functions and individual function arguments.

Tip:

• You can use the app.all() method to set up handler functions that run for every supported HTTP request method on a specific PATH.

• The app.use() method lets you set up middleware functions that can run for all HTTP request methods and paths when a request reaches them. If you specify a path argument, app.use() matches that path and its subpaths. For example, /book matches /book, /book/dashboard, and /book/dashboard/author/202605.

Middleware vs. Route Handler

Developers often use the term "route handler" to describe the final function responsible for resolving the request and sending the response, whereas preceding reusable functions are referred to as "middleware".

app.METHOD(PATH, MIDDLEWARE1, MIDDLEWARE2, HANDLER);

Note: Express does not mandate distinct names or categories for functions attached to a route. From Express's perspective, they are all handler functions, with next() passing control from one to the next. However, developers often differentiate between middleware and route handlers to enhance code organization, readability, and communication.

• Middleware functions, often stored in a /middlewares directory, serve as utility components that perform cross-cutting or preparatory tasks such as authentication, validation, logging, and data loading. These functions are often reusable and typically invoke next() to pass control to the next function.

• Route handlers, sometimes organized as controllers in a /controllers directory, contain the business logic that generates the endpoint's primary response, such as getUserProfile, createBlogPost, or deleteAccount. These functions may be specific to a route and usually conclude the request-response cycle by invoking methods such as res.send() or res.json().

Route Examples in Express.js

Here are some examples of routes in Express.js.

Express route with a single handler function

app.get("/single/handler", (req, res) => {
  res.send("Request handled with a single handler function!");
});

The example above uses a single callback function to handle GET requests to the GET /single/handler endpoint. The handler function, in this case, is an inline controller. A common approach is to keep controllers in a controllers directory. This makes the code easier to test and maintain while keeping the route definition readable.

Here's an example:

The following example shows a GET /single/handler route with a modular controller that has been moved to its own module (controllers/single-handler.js). This approach separates route definitions from the request-handling logic that generates the endpoint's response.

import express from "express";
import * as controller from "./controllers/single-handler.js";

const app = express();
const port = 3000;

app.get("/single/handler", controller.singleHandler);

app.listen(port, () => {
  console.log(`Express server running at http://localhost:${port}`);
});

Below is an example of a controller file that contains only the singleHandler handler function.

controllers/single-handler.js

function singleHandler(req, res) {
  res.send("Request handled with a single handler function!");
}

export { singleHandler };

Tip: A controller is a route handler or group of handlers that manages requests for specific endpoints. Controllers are often placed in separate modules to improve code organization and maintain a clear separation of concerns.

Express route with multiple middleware functions

app.get(
  "/multiple/middleware",
  (req, res, next) => {
    console.log("First callback: passing control to the next middleware >");
    next();
  },
  (req, res, next) => {
    console.log("Second callback: passing control to the next middleware >");
    next();
  },
  (req, res) => {
    res.send("Request handled with multiple middleware functions!");
  },
);

The example above uses multiple middleware functions to handle GET requests to the /multiple/middleware path.

Express route with an array of middleware functions

const arrayOfCallbacks = [
  function callback1(req, res, next) {
    console.log("First callback: passing control to the next middleware >");
    next();
  },
  function callback2(req, res, next) {
    console.log("Second callback: passing control to the next middleware >");
    next();
  },
  function callback3(req, res) {
    res.send("Request handled with an array of middleware functions!");
  },
];

app.get("/array/middleware", arrayOfCallbacks);

The example above uses an array of callback functions to handle GET requests to the GET /array/middleware endpoint.

Express route with a combination of an array of middleware functions and individual callbacks

const arrayOfCallbacks = [
  function callback1(req, res, next) {
    console.log("First callback: passing control to the next middleware >");
    next();
  },
  function callback2(req, res, next) {
    console.log("Second callback: passing control to the next middleware >");
    next();
  },
];

app.get(
  "/mix/middleware",
  arrayOfCallbacks,
  (req, res, next) => {
    console.log("Third callback: passing control to the next middleware >");
    next();
  },
  (req, res, next) => {
    console.log("Fourth callback: passing control to the next middleware >");
    next();
  },
  (req, res) => {
    res.send(
      "Request handled with a combination of an array of middleware functions and independent callback arguments!",
    );
  },
);

The example above uses both an array of callbacks and individual function arguments to handle GET requests to the /mix/middleware path.

Important Things to Know About the next Parameter

• next() passes control to the next middleware function in the stack.

• next("route") skips the remaining handlers for the current route and continues looking for a matching route. Matching considers both the request method and path. It is only effective within middleware functions defined for app.METHOD() or router.METHOD().

• next(new Error(value)) passes control to the next error-handling middleware.

• If the currently running middleware does not end the request-response cycle, include a next parameter and invoke it within the callback to pass control to the next middleware. Otherwise, the client's request will remain pending.

Categories of Middleware in Express.js

Express middleware can be classified into five primary categories:

• Application-level

• Router-level

• Error-handling

• Built-in

• Third-party

These categories describe where middleware is mounted, how it behaves, or its source.

Application-level middleware

This type of middleware is attached to the Express application instance (app).

Example:

import express from "express";
const app = express();

app.get("/profile", (req, res, next) => {
  console.log("Hi, there!");
  next();
});

Router-level middleware

This middleware is attached to an Express Router instance (router). It operates similarly to application-level middleware but is mounted on a router instead of the application.

Example:

import express from "express";
const router = express.Router();

router.get("/profile", (req, res, next) => {
  console.log("Hi, there!");
  next();
});

Error-handling middleware

This middleware handles errors during request processing and is identified by its four-parameter signature.

import express from "express";
const app = express();

app.use((err, req, res, next) => {
  console.error(err);
  res.status(500).send("Error 500: Encountered an unexpected error");
});

Note:

• All four parameters must be present, even if some are unused. Otherwise, Express won't recognize the function as error-handling middleware. Express uses the number of declared parameters, not their names, to identify such middleware.

• Define error-handling middleware after the routes and middleware whose errors it should handle. This placement ensures it processes errors passed from preceding middleware functions.

Built-in middleware

These middleware functions are included with Express and available by default, without requiring additional package installations.

Examples:

import express from "express";

express.static("public");
express.json();
express.urlencoded();

Third-party middleware

These middleware functions are provided by external packages developed and shared by the broader Express community. Examples include cors, morgan, and helmet.

import express from "express";
import cors from "cors";

const app = express();
app.use(cors());

Note: Middleware categories in Express are not mutually exclusive

Example 1:

import express from "express";

const app = express();
app.use(express.json());

The express.json() middleware in the snippet above is both:

• Application-level, because it is attached to the app instance

• Built-in, because Express provides it

Example 2:

import express from "express";
import cors from "cors";

const router = express.Router();
router.use(cors());

The cors() middleware in the snippet above is both:

• Router-level, because it is attached to the router instance

• Third-party, because a package outside of Express provided it

Are you asking what the express.Router() method in the example above is? Let's discuss it.

What Is the express.Router() Method?

The express.Router() method creates a router object that groups related routes and middleware. You can attach this router to an Express application or another router using app.use() or router.use().

As illustrated in the image above, a router is often called a mini-app because it provides a complete middleware and routing system. It helps organize applications by allowing you to define related routes in one place and mount them onto the main Express application. For instance, suppose your Express application needs to handle the following categories of routes and middleware:

• Index: For index-related routes and middleware.

• Books: For book-related routes and middleware.

• Videos: For video-related routes and middleware.

In this situation, you can use the express.Router() method to group related routes and middleware. This keeps things organized and makes each group easier to manage and reuse.

While you can create all router and app instances in one file, developers usually place routers in separate files for better organization and maintainability. Let's follow this approach by creating separate files for each route and middleware category.

Create a Directory for the Routers

Create a routes directory at your project's root to store all the app's routers.

mkdir routes

Note: You can name this folder whatever you like, but routes is a common name since it holds the files that define your app's routes (URL endpoints). The express.Router() method is just a tool that Express provides for managing those routes.

Create the Route Files

A route file is a module where you define related routes and export an Express router object. Each file usually represents a resource or feature, such as users, books, or authentication.

Create the route files in the routes directory.

touch routes/index.js routes/books.js routes/videos.js

Create a Mini-App for the Book-Related Routes and Middleware

Open the books.js file and use an express.Router() instance to group all the book-related routes and middleware.

routes/books.js

// Import the Router function from Express
import { Router } from "express";

// Create a new Express router instance
const bookRouter = Router();

// Define middleware specific to this router instance
bookRouter.use((req, res, next) => {
  console.log("Book route accessed at: ", Date.now());
  next();
});

// Define routes specific to this router instance:

bookRouter.get("/", (req, res) => {
  res.send("<h1>List of All Books</h1>");
});

bookRouter.get("/:id", (req, res) => {
  res.type("text/plain").send(`Details for book ${req.params.id}`);
});

bookRouter.get("/:id/draft", (req, res) => {
  // Escape the ID before inserting it into HTML text
  const bookId = req.params.id
    .replaceAll("&", "&amp;")
    .replaceAll("<", "&lt;")
    .replaceAll(">", "&gt;");
  res.send(`<h1>New Draft for Book ${bookId}</h1>`);
});

export { bookRouter };

Here's what the code above does:

• Creates a bookRouter instance to access Express APIs, such as .get(), .post(), .use(), and .route().

• Uses Express's router.use() method to mount middleware onto the bookRouter instance.

• Uses Express's router.get() method to define how the server handles HTTP GET requests.

• Uses Express's res.send() method to send an HTTP response and automatically set the appropriate headers based on the data type provided. The /:id handler explicitly sets the response type to plain text, while the /:id/draft handler escapes the ID before inserting it into HTML so it is displayed as text.

• Uses the export statement to make the router object available for import wherever it is needed in the application.

Time to Practice with Routers

This is your moment to try out what you've learned about Express routers.

For this exercise, create mini-apps for the project's index and video-related routes and middleware. Export them as indexRouter from routes/index.js and videoRouter from routes/videos.js to match the imports in the next section.

Take a moment to try this on your own before moving on. If you need help, review the books.js file. Practicing will help you understand better.

After you finish the exercise, move on to setting up the main Express app.

Update the Main Express Application

Open your project's server.js file and attach the routers to the main Express application.

server.js

// Add the Express module
import express from "express";

// Import the routers from their separate files
import { indexRouter } from "./routes/index.js";
import { bookRouter } from "./routes/books.js";
import { videoRouter } from "./routes/videos.js";

// Create a new Express application instance
const app = express();

// Specify the host and port to receive requests
const host = "localhost";
const port = 3000;

// Mount the routers to specific paths:

// Mount indexRouter at /
app.use("/", indexRouter);

// Mount bookRouter at /books and its subpaths
app.use("/books", bookRouter);

// Mount videoRouter at /videos and its subpaths
app.use("/videos", videoRouter);

// Run the server on the specified port and host
app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}!`);
});

Here's what the code above does:

• Creates an Express application instance to access Express APIs, such as routers and middleware, which simplifies HTTP request handling in Node.js.

• Uses Express's app.use() method to add the mini-apps to the main Express application.

• Uses Express's app.listen() method to start the Node.js HTTP server on a specific port and execute a callback when the server begins listening for client requests.

Note: The app.use() method's path serves as the router's base path or mount path. Routes defined in the router are matched relative to this mount path. For example:

• bookRouter's / route matches requests to /books.

• bookRouter's /:id route matches requests to /books/:id.

• bookRouter's /:id/draft route matches requests to /books/:id/draft.

Run your Express.js Application

Once you've set up the routers, run your app by starting the server file with Node.js.

node --watch server.js

Then, check your app running live at the specified endpoints. For example, http://localhost:3000/books.

Tip: Stop the running process using Ctrl + C on Windows, macOS, or Linux.

Important Things to Know About Express Routers

Here are some important things to remember when using Express routers.

Routers do not inherit parent route parameters by default

By default, Express doesn't pass parameters from a router's mount path to routes inside that router.

Example:

// Add the Express module
import express from "express";

// Create a new Express application instance
const app = express();

// Create a new Express router instance
const bookRouter = express.Router();

// Define a sub-route
bookRouter.get("/books/:bookId", (req, res) => {
  const topic = req.params.topic; // undefined by default
  const bookId = req.params.bookId;
  res.json({ topic, bookId });
});

// Define a parent route
app.use("/codesweetly/:topic", bookRouter);

// Run the server on the specified port and host
const server = app.listen(3000, "127.0.0.1", () => {
  const { address, port } = server.address();
  console.log(`Server running live at http://${address}:
${port}!`);
});

If you run the web server in the example above and enter the following URL in the browser:

http://127.0.0.1:3000/codesweetly/CSS/books/91827

The req.params.topic parameter will be undefined because sub-routes can't access the parent route's path parameter by default. So, the server will only send a {"bookId":"91827"} response to the client.

To let a child route access its parent's path parameter, set mergeParams to true when you create the router.

Example:

// Add the Express module
import express from "express";

// Create a new Express application instance
const app = express();

// Create a new Express router instance
const bookRouter = express.Router({ mergeParams: true });

// Define a sub-route
bookRouter.get("/books/:bookId", (req, res) => {
  const topic = req.params.topic; // "CSS" for the URL below
  const bookId = req.params.bookId;
  res.json({ topic, bookId });
});

// Define a parent route
app.use("/codesweetly/:topic", bookRouter);

// Run the server on the specified port and host
const server = app.listen(3000, "127.0.0.1", () => {
  const { address, port } = server.address();
  console.log(`Server running live at http://${address}:
${port}!`);
});

If you run the web server in the example above and enter the following URL in your browser:

http://127.0.0.1:3000/codesweetly/CSS/books/91827

The req.params.topic parameter will be CSS because mergeParams: true lets the sub-route access the parent's path parameter. So, the server will send a {"topic":"CSS","bookId":"91827"} response to the client.

Tip:

• The line const { address, port } = server.address() uses object destructuring to get the address and port values from the server.address() object.

• The app.listen() method returns a native Node.js http.Server instance, allowing access to underlying server APIs such as .address(), .on(), and .maxHeadersCount.

• 127.0.0.1 is an IPv4 loopback address. localhost may resolve to 127.0.0.1 or the IPv6 loopback address ::1. Use http://127.0.0.1:3000 for these examples because the server explicitly listens on that IPv4 address.

Routers can be mounted on other routers

You can attach one Express router to another.

Example:

// Add the Express module
import express from "express";

// Create a new Express application instance
const app = express();

// Create two Express router instances
const htmlRouter = express.Router();
const bookRouter = express.Router();

// Define a sub-route
bookRouter.get("/:bookId", (req, res) => {
  res
    .type("text/plain")
    .send(`Details for the HTML book with ID ${req.params.bookId}`);
});

// Mount bookRouter on htmlRouter's /books path
htmlRouter.use("/books", bookRouter);

// Mount htmlRouter at /html and its subpaths
app.use("/html", htmlRouter);

// Run the server on the specified port and host
const server = app.listen(3000, "127.0.0.1", () => {
  const { address, port } = server.address();
  console.log(`Server running live at http://${address}:
${port}!`);
});

The bookRouter's GET /:bookId route matches requests to GET /html/books/:bookId because, in this example, the book router is attached to htmlRouter.

Therefore, if you run the web server in the example above and enter the following URL in the browser:

http://127.0.0.1:3000/html/books/91827

The server will send a "Details for the HTML book with ID 91827" response to the client.

Now that you know what a router is, let's discuss how to use views to define the content and structure of the webpage your Express.js app returns to users.

What Is a View in Express.js?

A view is a template or document that defines the content and structure of a webpage returned to users by an Express.js application. It can be a static HTML file or a dynamic template rendered into HTML at runtime by a template engine.

In this guide, we'll work with two types of views:

• Static views

• Dynamic views

What Are Static Views in Express.js?

Static views contain the final HTML that the server sends to the browser as-is. For example, let's configure your Express.js application to serve four static views.

Create static views

Create index.html, about.html, contact.html, and 404.html files in your project's root directory.

touch index.html about.html contact.html 404.html

Open each view and add the following HTML content:

index.html (homepage)

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Static Views in Express.js | CodeSweetly Tutorial</title>
  </head>
  <body>
    <h1>Welcome to the Static View Guide</h1>
    <p>Site's pages:</p>
    <ul>
      <li><a href="/about">About</a></li>
      <li><a href="/contact">Contact</a></li>
    </ul>
  </body>
</html>

about.html (about page)

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>About Us | Static Views in Express.js</title>
  </head>
  <body>
    <h1>About Us</h1>
    <p>
      Lorem ipsum dolor sit amet consectetur adipisicing elit. Libero soluta
      voluptas reprehenderit minus veniam! Corrupti a esse quidem nostrum harum,
      explicabo tempore tempora aut, sint et voluptatem magni ea vel?
    </p>
    <div><a href="/">Return to the home page</a></div>
  </body>
</html>

contact.html (contact page)

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Contact | Static Views in Express.js</title>
  </head>
  <body>
    <h1>Contact Us</h1>
    <ul>
      <li><a href="https://codesweetly.com">Website</a></li>
      <li><a href="https://x.com/oluwatobiss">X (Twitter)</a></li>
    </ul>
    <div><a href="/">Return to the home page</a></div>
  </body>
</html>

404.html (error page)

<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>404 - Page not found | Static Views in Express.js</title>
  </head>
  <body>
    <h1>Page not found</h1>
    <div>Sorry, we couldn't find that page</div>
    <div><a href="/">Go back to the home page</a></div>
  </body>
</html>

Next, configure your project to handle browser connections.

Configure the project's web server to serve static views

Open the project's server.js module and configure the web server to serve the appropriate static HTML view in response to requests.

// Add the required modules
import express from "express";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";

// Create a new Express application instance
const app = express();

// Specify the hostname and port to receive requests
const host = "localhost";
const port = 3000;

// Get the directory name of the current file's path
const __dirname = dirname(fileURLToPath(import.meta.url));

// Use the index.html static view as a response to 'GET /' requests
app.get("/", (req, res) => res.sendFile(join(__dirname, "index.html")));

// Use the about.html static view as a response to 'GET /about' requests
app.get("/about", (req, res) => {
  res.sendFile(join(__dirname, "about.html"));
});

// Use the contact.html static view as a response to 'GET /contact' requests
app.get("/contact", (req, res) => {
  res.sendFile(join(__dirname, "contact.html"));
});

// Use the 404.html static view as a response to unmatched requests
app.use((req, res) => {
  res.status(404).sendFile(join(__dirname, "404.html"));
});

// Run the server on the specified port and hostname
app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}`);
});

Here are the main things the snippet above does:

• Create an Express application instance to access Express APIs, such as routers and middleware, which simplify HTTP request handling in Node.js.

• Get the directory name of the current file's path.

  • dirname() returns a path's directory name.

  • fileURLToPath() converts a URL to a valid path string.

  • import.meta.url returns the absolute URL of the current file.

• Create routes to receive and respond to clients' requests.

  • app is an Express application instance.

  • The app.get() method defines how the server handles HTTP GET requests to the specified path (the method's first argument).

  • The res.sendFile() method sends a file from the server to the client.

  • join() joins multiple path segments into a single, normalized path string using the current operating system’s path separator.

  • The app.use() method lets you set up handler functions that Express runs for all HTTP request methods and paths. If you specify a path argument, app.use() treats it as a prefix for matching routes. For example, /book matches /book, /book/dashboard, and /book/dashboard/author/202605.

  • res.status(404) sets the HTTP status code to 404 for a request that reaches this final handler without receiving a response.

• Use Express's app.listen() method to start the Node.js HTTP server on a specific port and execute a callback when the server begins listening for client requests.

Run your Express.js application

Once you've set up the server, run it with Node.js.

node --watch server.js

Afterward, check your app running live at http://localhost:3000.

Tip: Stop the running process using Ctrl + C on Windows, macOS, or Linux.

To keep things organized, put all your static views in a public directory. This is a common practice.

Create a directory for static views

Create a public directory at your project's root to store all static views.

mkdir public

Note: You can name this folder whatever you like, but public is a common choice to indicate that its files (HTML, CSS, client-side JavaScript, images) are intended to be served to clients. The folder's name alone does not make its contents accessible. The server must be configured to serve them. Don't put sensitive information in it.

Move all static views into the public folder

mv index.html about.html contact.html 404.html public/

Update the server to retrieve static views from the public folder

server.js

import express from "express";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";

const app = express();
const host = "localhost";
const port = 3000;
const __dirname = dirname(fileURLToPath(import.meta.url));

// Serve static assets (HTML, CSS, images) from the 'public' folder
app.use(express.static(join(__dirname, "public")));

// Use the index.html static view as a response to 'GET /' requests
app.get("/", (req, res) => {
  res.sendFile(join(__dirname, "/public/index.html"));
});

// Use the about.html static view as a response to 'GET /about' requests
app.get("/about", (req, res) => {
  res.sendFile(join(__dirname, "/public/about.html"));
});

// Use the contact.html static view as a response to 'GET /contact' requests
app.get("/contact", (req, res) => {
  res.sendFile(join(__dirname, "/public/contact.html"));
});

// Use the 404.html static view as a response to unmatched requests
app.use((req, res) => {
  res.status(404).sendFile(join(__dirname, "/public/404.html"));
});

// Run the server on the specified port and hostname:
app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}`);
});

Note The app.use(express.static(join(__dirname, 'public'))) line tells Express to check the public folder first for static asset requests. If it finds the file there, it immediately serves it to the browser and stops the request-response cycle.

Example:

• If a user requests localhost:3000/about.html, express.static checks the public folder for the about.html file. If found, Express automatically sends it without running the route handlers that follow the static middleware.

• If a user asks for localhost:3000/about, express.static checks the public folder for an about file (no extension). If it finds an about directory instead, it redirects to /about/, where it looks for index.html by default. With the files created in this guide, neither an about file nor an about directory exists, so Express uses the GET /about custom route to handle the request.

Let's now discuss dynamic views.

What Are Dynamic Views in Express.js?

Dynamic views contain placeholders that template engines, such as EJS, replace with data at runtime.

Express.js supports template engines such as EJS, Pug, and Handlebars. Let's use EJS as an example to discuss how dynamic views work.

What is EJS?

EJS (Embedded JavaScript templating) is a templating language that lets you create dynamic HTML views by embedding JavaScript code within HTML templates using template tags.

When the server renders a template, the EJS engine executes the embedded JavaScript and generates the final HTML sent to the client.

For instance, consider the following code. Install EJS with npm install ejs before running this example:

// Add the Express and EJS modules
import express from "express";
import ejs from "ejs";

// Create a new Express application instance
const app = express();

// Specify the host and port to receive requests
const host = "localhost";
const port = 8000;

// Define the data object
const friendsArray = ["Sarah", "Abraham", "Mary"];
const replacementObject = { friends: friendsArray };

// Use backticks to define a template string with embedded JavaScript
const templateString = `
  <h1>Hello, <%= friends.join(", "); %>!</h1>
  <p>Welcome to CodeSweetly.</p>
`;

// Compile and render the template string to HTML (the rendered view)
const html = ejs.render(templateString, replacementObject);

// Use the dynamic view as a response to 'GET /' requests
app.get("/", (req, res) => res.send(html));

// Run the server on the specified port and hostname
app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}!`);
});

The above snippet tells Express to send the rendered view to the client when users request the / path.

At runtime, EJS evaluates friends.join(", ") using the array passed as friends and inserts the result, Sarah, Abraham, Mary, into the HTML. Here, rendering happens once when the script starts. Each request receives that same rendered HTML.

What are EJS template tags?

EJS template tags are special tags written using the <% ... %> syntax. They allow JavaScript code and values to be embedded into EJS view templates.

When EJS renders the template, scriptlet tags execute JavaScript, output tags insert values, and comment tags are ignored.

For example, the <%= %> tags below allow you to embed a JavaScript friend variable into the view template.

<h1>About <%= friend %>, my special pal</h1>

Note: Output tags such as <%= %> must contain a valid JavaScript expression. Scriptlet tags can contain JavaScript statements and parts of control-flow structures, while comment tags contain comments.

Example:

• <%= <p>I love CodeSweetly</p> %> will throw an error because <p>I love CodeSweetly</p> is HTML, not JavaScript.

• <%= "<p>I love CodeSweetly</p>" %> is valid because "<p>I love CodeSweetly</p>" is embedded as a JavaScript string data type.

Types of EJS template tags

The nine EJS tag forms covered here are as follows:

Tip: Wrap JavaScript code in EJS tags to distinguish it from the surrounding template content. A single pair of tags can contain multiple lines of JavaScript.

Closing tag (%>)

Use the closing tag to end an EJS tag. It has no special behavior on its own.

Scriptlet tag (<%)

Use the scriptlet tag to run JavaScript code without outputting anything into the generated HTML. This tag is commonly used for control flow, such as if, for, and forEach logic.

<ul>
  <% friends.forEach(function () { %>
  <li>My friend</li>
  <% }) %>
</ul>

The snippet above wraps <% %> around the control-flow syntax to indicate to EJS that the JavaScript code is only for logic and should not be rendered to the generated HTML.

Escaped output tag (<%=)

Use the escaped output tag to evaluate JavaScript expressions and output the result into the generated HTML while escaping HTML characters. This tag helps prevent HTML injection.

<%= 100 + 200 %>

The above snippet wraps <%= %> around the arithmetic expression to tell EJS to render the code's value into the generated HTML.

Note: If the expression evaluates to a string containing HTML, <%= %> escapes the HTML syntax so the browser displays it as text. For example, <%= "<h1>About CodeSweetly</h1>" %> will render <h1>About CodeSweetly</h1> as text, not an actual <h1> element.

Unescaped output tag (<%-)

Use the unescaped tag to evaluate JavaScript expressions and output the result into the generated HTML without escaping HTML characters.

<%- "<h1>About CodeSweetly</h1>" %>

The snippet above wraps <%- %> around a string containing an <h1> element so the browser renders it as HTML. Use unescaped output only for trusted HTML.

Literal opening tag (<%%)

Use the literal tag to output a literal <% sequence instead of treating it as an EJS tag. This is useful when you want to show EJS syntax in the generated output.

<%% "<p><strong>Name:</strong> <em>Oluwatobi</em></p>" %>

The snippet above outputs a literal <% followed by the remaining text, without evaluating it as JavaScript. The generated HTML source is:

<% "
<p><strong>Name:</strong> <em>Oluwatobi</em></p>
" %>

The HTML tags remain unescaped, so this shows the generated source, not how the browser displays the page.

Newline-trimmed ending tag (-%>)

Use the newline-trimmed ending tag to remove the newline immediately after the closing tag.

<ul>
  <% friends.forEach(function (friend) { %>
  <li><%= friend -%></li>
  <% }) %>
</ul>

The snippet above wraps <%= -%> around the friend variable to tell EJS to trim any newline following the closing EJS tag.

Tip: The newline-trimmed ending tag is also called the newline slurp tag.

Whitespace slurp opening tag (<%_)

Use the whitespace slurping tag to remove spaces and tabs immediately before the opening EJS tag on the same line.

<ul>
  <%_ friends.forEach(function (friend) { %>
  <li><%= friend %></li>
  <%_ }) %>
</ul>

The snippet above wraps <%_ %> around the control-flow syntax to tell EJS to trim the indentation before the template tags.

Whitespace slurp closing tag (_%>)

Use the whitespace slurping ending tag to remove spaces and tabs immediately after the closing tag, followed by one newline if present.

<ul>
  <% friends.forEach(function (friend) { _%>
  <li><%= friend %></li>
  <% }) _%>
</ul>

The snippet above wraps <% _%> around the control-flow syntax to tell EJS to trim the newline after each closing tag. The indentation on the following line remains.

Comment tag (<%#)

Use the comment tag to add EJS comments that are ignored during rendering and do not appear in the generated HTML.

<ul>
  <%# Loop through the friends array %>
  <% friends.forEach(function (friend) { _%>
  <li><%= friend %></li>
  <% }) _%>
</ul>

The snippet above wraps the comment in <%# %>.

Tip: EJS allows you to combine some tags. For example, the following combinations are possible:

• <%_ _%>: Combine the whitespace slurping scriptlet and ending tag to remove adjacent spaces and tabs on the same line, plus one following newline if present.

• <% -%>: Combine the control-flow with the newline slurp.

• <%= -%>: Combine the escaped tag with the newline slurp.

How to use EJS in an Express project

The following sections will guide you through the process of using EJS in your Express.js project.

Install EJS

First, install EJS in your Express project.

npm install ejs@6.0.1

Next, open your project's server.js module and configure the web server to serve the appropriate dynamically generated HTML view in response to requests.

server.js

// Add the Express and EJS modules
import express from "express";
import ejs from "ejs";

// Create a new Express application instance
const app = express();

// Specify the host and port to receive requests
const host = "localhost";
const port = 8000;

// Define the data object
const friendsData = {
  bestFriend: "Sarah",
  codingFriend: "Abraham",
  dreamFriend: "Mary",
};
const replacementObject = { friends: friendsData };

// Use backticks to define a template string with embedded JavaScript
const templateString = `
<html>
  <body>
    <h1>List of Friends</h1>
    <ul>
    <%# Loop through the friends object %>
    <% for (const eachFriend in friends) { -%>
      <li><%= friends[eachFriend] %> is my <%= eachFriend %></li>
    <% } -%>
    </ul>
  </body>
</html>
`;

// Compile and render the template string to HTML (the rendered view)
const htmlData = ejs.render(templateString, replacementObject);

// Use the dynamic view as a response to 'GET /' requests
app.get("/", (req, res) => res.send(htmlData));

// Run the server on the specified port and hostname
app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}!`);
});

Here are the main things the snippet above does:

• Create an Express application instance to access Express APIs, such as routers and middleware, which simplify HTTP request handling in Node.js.

• Render the template string once when the script starts to generate the HTML that the server sends to each client.

• Create routes to receive and respond to clients' requests.

• Use Express's app.listen() method to start the Node.js HTTP server on a specific port and execute a callback when the server begins listening for client requests.

Run the Express app

After setting up the server, run it with Node.js.

node --watch server.js

Afterward, check your app running live at http://localhost:8000.

Tip: Stop the running process using Ctrl + C on Windows, macOS, or Linux.

To keep things organized, put all your view templates in a views directory. This is a common practice.

Create a directory for view templates

Create a views directory at your project's root to store all view templates.

mkdir views

Create an index view template

touch views/index.ejs

Tip: EJS template files use the .ejs extension.

Open the file and move the HTML template from server.js into it.

views/index.ejs

<html>
  <body>
    <h1>List of Friends</h1>
    <ul>
      <%# Loop through the friends object %>
      <% for (const eachFriend in friends) { -%>
      <li><%= friends[eachFriend] %> is my <%= eachFriend %></li>
      <% } -%>
    </ul>
  </body>
</html>

Update the server to retrieve view templates from the views folder

server.js

import express from "express";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";

const app = express();
const host = "localhost";
const port = 8000;
const __dirname = dirname(fileURLToPath(import.meta.url));

// Specify the app's views directory
app.set("views", join(__dirname, "views"));

// Specify the app's view engine
app.set("view engine", "ejs");

// Define the data object
const friendsData = {
  bestFriend: "Sarah",
  codingFriend: "Abraham",
  dreamFriend: "Mary",
};
const replacementObject = { friends: friendsData };

// Compile and render the index view template to HTML and use it as a response to 'GET /' requests
app.get("/", (req, res) => {
  res.render("index", replacementObject);
});

// Run the server on the specified port and hostname
app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}`);
});

The snippet above uses Express's render() method to generate HTML from the index.ejs template file.

Note:

• If you omit the app.set("view engine", "ejs") line, then the render() method's view template argument must include a file extension like this:

app.get("/", (req, res) => {
  res.render("index.ejs", replacementObject);
});

• Express has no default template engine. Express also supports engines such as Pug (formerly called Jade), EJS, Mustache, and Handlebars.

• app.set() is the method for adding a key-value pair to the application's settings. You can retrieve the setting's value with app.get() as follows:

app.set("my name", "Oluwatobi");

const bio = app.get("my name");

console.log(bio); // Outputs: "Oluwatobi"

How to create reusable EJS templates (partials)

EJS provides the include() method for including (nesting) one view template into another.

Tip: In EJS, partials are reusable templates.

Syntax of the include() method

The include() method accepts two arguments. Here's the syntax:

<%- include("path/to/partial", data) %>

• "path/to/partial": (required) The path to the reusable template, relative to the current file (the parent template). For example, if the current file is at "./views/index.ejs" and the partial is at "./views/partials/footer.ejs", the include() path argument would be "partials/footer".

• data: (optional) An object containing the properties to pass to the partial.

How to include a partial in a view template

Consider the following server file:

server.js

import express from "express";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";

const app = express();
const host = "localhost";
const port = 8080;
const __dirname = dirname(fileURLToPath(import.meta.url));

app.set("views", join(__dirname, "views"));
app.set("view engine", "ejs");

const friendsData = {
  bestFriend: "Sarah",
  codingFriend: "Abraham",
  dreamFriend: "Mary",
};
const replacementObject = { friends: friendsData };

app.get("/", (req, res) => {
  res.render("index", replacementObject);
});

app.listen(port, host, () => {
  console.log(`Server running live at http://${host}:
${port}`);
});

The snippet above will render an index template when users send a GET request to the "/" path. Below is the index.ejs template file.

views/index.ejs

<html>
  <body>
    <h1>List of Friends</h1>
    <ul>
    <%# Loop through the friends object %>
    <% for (const eachFriend in friends) { -%>
      <li><%= friends[eachFriend] %> is my <%= eachFriend %></li>
    <% } -%>
    </ul>
  </body>
</html>

If you need the <li> element to be reusable in multiple templates, extract it into a separate file and use the include() method to add it to any template as needed. Here's how:

Create a partials directory

Create a partials directory in your views folder to store all reusable view templates.

mkdir views/partials

Create a partial view template

Create a partial view template for the <li> element.

touch views/partials/friendLi.ejs

Open the file and add the <li> element.

views/partials/friendLi.ejs

<li><%= friend %> is my <%= friendType %></li>

Update the main view template

Open the index.ejs file and use the include() method to add the friendLi.ejs partial view template.

views/index.ejs

<html>
  <body>
    <h1>List of Friends</h1>
    <ul>
    <%# Loop through the friends object %>
    <% for (const eachFriend in friends) { -%>
      <%- include("partials/friendLi", { friend: friends[eachFriend], friendType: eachFriend }) %>
    <% } -%>
    </ul>
  </body>
</html>

Using include() to add the <li> element makes it reusable in multiple templates, rather than restricting it to the index.ejs view template.

Run your server from the root directory

node --watch server.js

Then, check your app running live at http://localhost:8080.

Note: Although the steps above demonstrate EJS partials with a simple <li> element, you can use the same concept to reuse your application's navbar, footer, and other components across multiple view templates.

Overview

In this handbook, we explored the core concepts you need to start building applications with Node.js and Express.js. We discussed how Node.js works, its module support, and how to create HTTP servers. We also explored how Express.js simplifies server-side development through routes, routers, and views.

Whether you’re considering a small personal project or a full-stack application for a larger user base, you now have a solid starting point for building with Node.js and Express.js.

Thanks for reading!

Dive Deeper into Node.js and Express.js

This handbook has given you a peek inside my Node.js and Express.js Simplified book.

Whether you’re learning backend development for the first time, expanding your JavaScript skills beyond frontend development, or looking for a practical introduction to Node.js and Express.js, the book will help you build on what you’ve learned here and develop the skills to create, test, and deploy complete applications.

You’ll dive deeper into Node.js and Express.js through practical explanations and hands-on examples covering topics such as file management, file uploads, forms, sessions, frontend-to-backend communication, testing, TypeScript, and deployment.

In this tutorial, you'll build a fully functional, real-time Word Counter tool from scratch. You'll use HTML for the structure, CSS for a clean design, and vanilla JavaScript to handle the counting logic.

Because this tool runs completely in the browser, it works instantly as you type and keeps your text data completely private.

Let's get started.

Prerequisites

To follow this guide easily, you should have:

• A basic understanding of HTML tags and CSS styling.

• Familiarity with JavaScript concepts like functions, event listeners, and basic Regular Expressions (Regex).

• A code editor (like VS Code) and a web browser.

Table of Contents

• Step 1: Set Up Your Project

• Step 2: Build the HTML Structure

  • Understanding the HTML:

• Step 3: Style the Tool with CSS

  • Understanding the CSS:

• Step 4: Add the JavaScript Logic

  • Understanding the JavaScript:

• Step 5: Test Your Application

Step 1: Set Up Your Project

First, you need to set up your workspace. Create a new folder on your computer and name it word-counter-app.

Inside this folder, create three empty files:

• index.html

• style.css

• script.js

Step 2: Build the HTML Structure

Open your index.html file. You need to create a layout that includes a header, a grid to display the real-time statistics, a text area for the user to type in, and buttons to copy or clear the text.

Add the following code into your HTML file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Word Counter Tool</title>
    <link rel="stylesheet" href="style.css">
</head>
<body>

    <header class="header-section">
        <h1>Word Counter</h1>
        <p class="description">Paste or type your text below to get a real-time count of words, characters, sentences, and paragraphs.</p>
    </header>

    <main class="tool-container">
        
        <!-- Statistics Panel -->
        <div class="stats-grid">
            <div class="stat-box">
                <div class="stat-value" id="wordCount">0</div>
                <div class="stat-label">Words</div>
            </div>
            <div class="stat-box">
                <div class="stat-value" id="charCount">0</div>
                <div class="stat-label">Characters</div>
            </div>
            <div class="stat-box">
                <div class="stat-value" id="sentenceCount">0</div>
                <div class="stat-label">Sentences</div>
            </div>
            <div class="stat-box">
                <div class="stat-value" id="paragraphCount">0</div>
                <div class="stat-label">Paragraphs</div>
            </div>
        </div>

        <!-- User Input Area -->
        <textarea id="textInput" placeholder="Start typing or paste your text here..."></textarea>

        <!-- Action Buttons -->
        <div class="controls">
            <button class="btn-primary" onclick="copyText()" id="copyBtn">Copy Text</button>
            <button class="btn-secondary" onclick="clearText()">Clear</button>
        </div>

    </main>

    <script src="script.js"></script>
</body>
</html>

Understanding the HTML:

• The .stats-grid: This holds four separate boxes to show the counts. Each number has a unique id (like wordCount) so JavaScript can find and update it easily.

• The <textarea>: This is the main input box where users will type or paste their content.

• The <script> tag: This connects your HTML to the logic you will write in the next steps.

Step 3: Style the Tool with CSS

Next, you'll give your tool a modern, professional look. You'll use CSS Grid to align the statistics boxes perfectly.

Open your style.css file and add this code:

:root {
    --primary-color: #007bff;
    --bg-color: #f8f9fa;
    --text-dark: #202124;
    --text-muted: #5f6368;
    --border-color: #dadce0;
    --panel-bg: #ffffff;
}

body {
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
    background-color: var(--bg-color);
    color: var(--text-dark);
    margin: 0;
    padding: 40px 20px;
    display: flex;
    flex-direction: column;
    align-items: center;
}

.header-section {
    text-align: center;
    margin-bottom: 30px;
}

h1 {
    font-size: 2.5rem;
    margin: 0 0 10px 0;
    font-weight: 800;
}

p.description {
    color: var(--text-muted);
    max-width: 600px;
    margin: 0 auto;
    line-height: 1.6;
    font-size: 1.1rem;
}

.tool-container {
    background-color: var(--panel-bg);
    border: 1px solid var(--border-color);
    border-radius: 12px;
    padding: 24px;
    width: 100%;
    max-width: 900px;
    box-shadow: 0 4px 12px rgba(0,0,0,0.05);
}

/* Stats Grid */
.stats-grid {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
    gap: 16px;
    margin-bottom: 24px;
}

.stat-box {
    background-color: var(--bg-color);
    border: 1px solid var(--border-color);
    border-radius: 8px;
    padding: 16px;
    text-align: center;
}

.stat-value {
    font-size: 2rem;
    font-weight: 700;
    color: var(--primary-color);
    margin-bottom: 4px;
}

.stat-label {
    font-size: 0.9rem;
    color: var(--text-muted);
    text-transform: uppercase;
    letter-spacing: 0.5px;
    font-weight: 600;
}

/* Text Area */
textarea {
    width: 100%;
    height: 250px;
    padding: 16px;
    border: 1px solid var(--border-color);
    border-radius: 8px;
    font-size: 1rem;
    line-height: 1.6;
    resize: vertical;
    box-sizing: border-box;
    font-family: inherit;
    margin-bottom: 20px;
    transition: border-color 0.2s ease;
}

textarea:focus {
    outline: none;
    border-color: var(--primary-color);
}

/* Buttons */
.controls {
    display: flex;
    gap: 12px;
}

button {
    padding: 10px 20px;
    font-size: 1rem;
    font-weight: 600;
    border-radius: 6px;
    cursor: pointer;
    border: none;
    transition: background-color 0.2s ease;
}

.btn-primary {
    background-color: var(--primary-color);
    color: #ffffff;
}

.btn-primary:hover {
    background-color: #0056b3;
}

.btn-secondary {
    background-color: transparent;
    color: var(--text-dark);
    border: 1px solid var(--border-color);
}

.btn-secondary:hover {
    background-color: var(--bg-color);
}

Understanding the CSS:

• CSS Grid: The .stats-grid class uses grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));. This makes the four stat boxes automatically stack neatly on top of each other if a user views the tool on a small mobile screen.

• CSS Variables: Using :root at the top allows you to quickly change the brand color later if you want to use something other than the default blue (#007bff).

At this point, the visual design of your tool is complete. Here's what the final Word Counter will look like in your browser:

Step 4: Add the JavaScript Logic

Now you need to make the tool count the text. You'll use an event listener that watches every keystroke. Every time the user types, it recalculates the words, characters, sentences, and paragraphs.

Open your script.js file and paste this code:

const textInput = document.getElementById('textInput');
const wordCountDisplay = document.getElementById('wordCount');
const charCountDisplay = document.getElementById('charCount');
const sentenceCountDisplay = document.getElementById('sentenceCount');
const paragraphCountDisplay = document.getElementById('paragraphCount');
const copyBtn = document.getElementById('copyBtn');

// 1. Listen for user input in real-time
textInput.addEventListener('input', updateStatistics);

// 2. The Core Counting Logic
function updateStatistics() {
    const text = textInput.value;

    // Character Count (includes spaces)
    charCountDisplay.textContent = text.length;

    // Word Count
    const words = text.match(/\S+/g) || [];
    wordCountDisplay.textContent = words.length;

    // Sentence Count
    const sentences = text.split(/[.!?]+(?=\s|$)/).filter(sentence => sentence.trim().length > 0);
    sentenceCountDisplay.textContent = sentences.length;

    // Paragraph Count
    const paragraphs = text.split(/\n+/).filter(paragraph => paragraph.trim().length > 0);
    paragraphCountDisplay.textContent = paragraphs.length;
}

// 3. Copy functionality
function copyText() {
    if (!textInput.value) return;
    
    textInput.select();
    document.execCommand('copy');
    
    // Provide visual feedback
    copyBtn.textContent = 'Copied!';
    setTimeout(() => {
        copyBtn.textContent = 'Copy Text';
    }, 1500);
}

// 4. Clear functionality
function clearText() {
    textInput.value = '';
    updateStatistics(); // Reset counts to zero
}

Understanding the JavaScript:

• The input Event: textInput.addEventListener('input', ...) is the secret to real-time updates. It triggers the counting function the exact moment a key is pressed or text is pasted.

• Counting Words: text.match(/\S+/g) is a Regular Expression that looks for unbroken strings of non-whitespace characters. This is much more accurate than just splitting the text by spaces, because it ignores extra empty spaces left by mistake.

• Counting Sentences: The .split(/[.!?]+(?=\s|$)/) logic cuts the text into pieces every time it sees a period, exclamation mark, or question mark followed by a space.

Step 5: Test Your Application

You're completely done coding! Now it's time to verify that your logic works correctly.

1. Open your word-counter-app folder.

2. Double-click the index.html file to open it in your web browser.

3. Type a few sentences into the text area. Watch the numbers at the top update instantly.

4. Try adding double spaces or hitting the "Enter" key to make new paragraphs, and ensure the logic counts them accurately.

5. Click Copy Text to test the clipboard feature, and Clear to reset the board.

By completing this project, you've built a fast, client-side utility tool using pure vanilla JavaScript. You learned how to manipulate strings, use Regular Expressions for text analysis, and create responsive UI grids.

If you want to see this exact codebase running in a live production environment, you can try out this Online Word Counter. Keep building, and happy coding!

A slide deck, a video player, kiosk dashboard, game, drawing canvas, or timer on a classroom projector all feel half-finished while the browser's tabs and address bar are still hanging around the edges.

The good news is that the browser has a built-in answer: the Fullscreen API. The less good news is that it comes with a handful of quirks that will bite you the first time you ship it: user gestures, Safari prefixes, iPhones that simply refuse, and a screen that goes to sleep two minutes into your beautiful fullscreen experience.

In this tutorial, you'll learn how to:

• Put any element (or the whole page) into fullscreen and back out again

• Keep your UI in sync when the user presses Esc

• Style fullscreen content with the :fullscreen pseudo-class

• Handle the cross-browser gotchas, including the one platform that still doesn't support it

• Stop the screen from dimming with the Screen Wake Lock API

• Combine all of it into one small, reusable pattern

Everything here is vanilla JavaScript with no libraries or build step. You should be comfortable with DOM events and async/await. If you need a refresher on the latter, freeCodeCamp has a solid async/await tutorial that covers everything we'll rely on.

Table of Contents

• How the Fullscreen API Works

• Entering and Exiting Fullscreen

• The User Gesture Rule

• Keeping Your UI in Sync with fullscreenchange

• Styling Fullscreen Content

• Cross-Browser Gotchas

• Keeping the Screen Awake with the Wake Lock API

• Putting It All Together

• A Quick Gotcha Checklist

• Conclusion

How the Fullscreen API Works

The Fullscreen API is small. It gives you three things you'll use every day:

• element.requestFullscreen(): asks the browser to display that element (and its descendants) using the entire screen. It returns a Promise.

• document.exitFullscreen(): leaves fullscreen. Also returns a Promise.

• document.fullscreenElement: the element currently in fullscreen, or null if there isn't one. This is your single source of truth for "are we fullscreen right now?"

Plus two events on document: fullscreenchange (fires when fullscreen is entered or exited) and fullscreenerror (fires if a request fails).

One thing that surprises people: you can make any element fullscreen, not just the page. If you fullscreen a <div>, only that <div> fills the screen. Everything else in the document is hidden behind it.

To fullscreen the whole page, you call the method on document.documentElement (the <html> element).

The full reference lives on MDN's Fullscreen API page, but you won't need much beyond what's above.

Entering and Exiting Fullscreen

Let's start with the simplest useful thing: a button that toggles the page in and out of fullscreen.

<button id="fs-toggle">Fullscreen</button>

const toggleBtn = document.getElementById('fs-toggle');

async function toggleFullscreen() {
  if (!document.fullscreenElement) {
    // Nothing is fullscreen yet – enter it.
    await document.documentElement.requestFullscreen();
  } else {
    // Something is fullscreen – leave it.
    await document.exitFullscreen();
  }
}

toggleBtn.addEventListener('click', toggleFullscreen);

That's genuinely it for the happy path. Because both methods return Promises, you can await them and know that the transition is finished before running any follow-up code.

Both calls can reject, though. For example, if the document isn't allowed to go fullscreen (we'll get to why below), or if you call exitFullscreen() when nothing is fullscreen. So in real code, wrap them in try/catch rather than letting an unhandled rejection land in the console:

async function toggleFullscreen() {
  try {
    if (!document.fullscreenElement) {
      await document.documentElement.requestFullscreen();
    } else {
      await document.exitFullscreen();
    }
  } catch (err) {
    console.warn(`Fullscreen failed: ${err.name} – ${err.message}`);
  }
}

The User Gesture Rule

Here's the first gotcha, and it's the one that generates the most confused questions on Stack Overflow: you can't enter fullscreen automatically. Not on page load, after a timer, or in response to a network event.

requestFullscreen() only works while the page has what the spec calls transient activation: a short window right after the user genuinely interacts with the page (like with a click, tap, or key press). Outside that window the Promise rejects, and in most browsers you'll see a message like "API can only be initiated by a user gesture."

This is deliberate. Without it, any page could hijack your whole screen the moment it loaded, which is exactly the kind of thing phishing pages would love to do.

In practice, this means two things for your code:

1. Always trigger fullscreen from an event handler tied to user input: click, keydown, pointerup, and so on.

2. Don't put an await for something slow before the requestFullscreen() call. If you await fetch(...) first, the activation window may have expired by the time you ask for fullscreen.

Call requestFullscreen() first and do the slow work after, or the activation window closes before you ask.

A common and very natural pattern is to let a keyboard shortcut do the same job as the button:

document.addEventListener('keydown', (e) => {
  // Ignore shortcuts while the user is typing in a field.
  if (e.target.matches('input, textarea, [contenteditable]')) return;

  if (e.key === 'f' || e.key === 'F') {
    e.preventDefault();
    toggleFullscreen();
  }
});

Two small things I got wrong the first time around are worth passing on.

First, make the fullscreen surface itself keyboard-reachable: give it tabindex="0" and treat Enter and Space on it exactly like a click, otherwise keyboard users have a button nobody told them about.

Second, think about whether your shortcut key collides with what the page actually does. I have a "hacker typer" page where any key spits out fake terminal output, including F. For the first few days, users typing furiously would drop out of fullscreen mid-"hack" every time they hit that letter. The fix was a one-liner: on that page F only ever enters fullscreen, and leaving is Esc's job alone.

Which brings up Esc: you don't handle it yourself. Every browser exits fullscreen on Esc, and you can't prevent that. It's a safety escape hatch. What you can do is react to it, which is the next section.

Keeping Your UI in Sync with fullscreenchange

Because the user can leave fullscreen in ways your code doesn't control (like by pressing Esc, using the browser's own exit button, or switching apps on mobile), you should never track fullscreen state in your own variable. It will drift out of sync.

Instead, treat document.fullscreenElement as the truth and update your UI whenever fullscreenchange fires:

function syncFullscreenUI() {
  const isFullscreen = Boolean(document.fullscreenElement);
  toggleBtn.textContent = isFullscreen ? 'Exit fullscreen' : 'Fullscreen';
  toggleBtn.setAttribute('aria-pressed', String(isFullscreen));
  document.body.classList.toggle('is-fullscreen', isFullscreen);
}

document.addEventListener('fullscreenchange', syncFullscreenUI);
document.addEventListener('fullscreenerror', () => {
  console.warn('Could not enter fullscreen.');
});

This single listener covers every path in and out of fullscreen, including the ones you didn't initiate. It's also the right place to pause an animation, resume a game loop, or (as you'll see later) release a wake lock.

Styling Fullscreen Content

CSS gives you two hooks for fullscreen state.

The :fullscreen pseudo-class matches the element that's currently fullscreen. Browsers apply a default stylesheet that stretches the element to the full viewport, but you'll usually want to control things like background and overflow yourself:

:fullscreen {
  background: #000;
  overflow: hidden;
  cursor: none; /* hide the pointer on an idle fullscreen surface */
}

That cursor: none line looks cosmetic until you put a pure black page on an OLED display: every pixel is off, and the mouse pointer is quite literally the only thing lit on the panel. Hiding it once the page is fullscreen is the difference between "the screen is off" and "the screen has a tiny white arrow in the middle of it."

The ::backdrop pseudo-element is the layer painted behind the fullscreen element. It only matters when your fullscreen element doesn't cover the whole screen (for example, an element with a fixed aspect ratio), and it's how you control the letterboxing colour:

:fullscreen::backdrop {
  background: #000;
}

One practical tip: if you're fullscreening the whole page, set your background colour on html, not body. In fullscreen, html is the element being displayed, and on some browsers a body-only background leaves a thin strip of the wrong colour at the edges during the transition.

Cross-Browser Gotchas

The Fullscreen API has been standard in Chrome, Edge and Firefox for years. Safari is where the work is.

Older Safari Needs the webkit Prefix

Safari only shipped the unprefixed API in version 16.4 (spring 2023) on macOS and iPadOS. Before that it used webkitRequestFullscreen(), webkitExitFullscreen(), webkitFullscreenElement, and a webkitfullscreenchange event. Unless you can ignore three-year-old Safari installs, a tiny compatibility layer is worth having:

const fs = {
  get element() {
    return document.fullscreenElement ?? document.webkitFullscreenElement ?? null;
  },
  get enabled() {
    return Boolean(document.fullscreenEnabled ?? document.webkitFullscreenEnabled);
  },
  request(el = document.documentElement) {
    if (el.requestFullscreen) return el.requestFullscreen();
    if (el.webkitRequestFullscreen) return Promise.resolve(el.webkitRequestFullscreen());
    return Promise.reject(new Error('Fullscreen not supported'));
  },
  exit() {
    if (document.exitFullscreen) return document.exitFullscreen();
    if (document.webkitExitFullscreen) return Promise.resolve(document.webkitExitFullscreen());
    return Promise.reject(new Error('Fullscreen not supported'));
  },
};

// Listen to both event names so old Safari stays in sync too.
['fullscreenchange', 'webkitfullscreenchange'].forEach((evt) =>
  document.addEventListener(evt, syncFullscreenUI)
);

Now fs.request() and fs.exit() both return a Promise in every browser, and the rest of your code doesn't care which one it's running in. (You'll see screenfull.js recommended for this. It's a fine library, but it's been declared feature-complete and frozen, and the fifteen lines above are all it was ever doing for you.)

iPhone Safari Doesn't Support Element Fullscreen at All

This is the gotcha that catches everyone. As of the time of writing, Safari on iPhone doesn't support the Fullscreen API for arbitrary elements – only for <video>. It works on iPad and on the Mac, but on the phone document.fullscreenEnabled is false and requestFullscreen doesn't exist on a <div>. Chrome and Firefox on iOS inherit the same limitation because they're required to use Safari's engine.

You have two realistic options:

1. A "pseudo-fullscreen" fallback: Position your element with position: fixed; inset: 0 and hide your own chrome. You won't get rid of Safari's address bar, but it collapses on scroll, and for most tools this is good enough.

2. Suggest "Add to Home Screen": A web app launched from the home screen with "display": "standalone" in its manifest runs without any browser UI. This is the only way to get truly edge-to-edge content on an iPhone today.

Whichever you choose, don't rely on feature detection alone. document.fullscreenEnabled tells you whether the API exists, but the request can still be rejected at runtime (inside an iframe without the allowfullscreen attribute, on a page a browser policy has locked down, or simply because a browser you haven't tested has its own opinion).

The pattern that has held up for me is to try the real API and treat a rejection as the trigger for the CSS fallback:

function enterFullscreen(el) {
  if (fs.enabled) {
    return fs.request(el).catch(() => {
      // The API exists but refused – degrade to a fixed overlay.
      document.body.classList.add('pseudo-fullscreen');
    });
  }
  document.body.classList.add('pseudo-fullscreen');
  return Promise.resolve();
}

On phones, I go one step further and don't call the API at all, even on Android where it technically works. Between the address bar reappearing on scroll and the system's swipe gestures, a fixed overlay behaves more predictably than real fullscreen on a small touch screen. It's also the same code path that iPhone Safari forces you into anyway. One less branch to test.

You can check the current support table on Can I use before deciding how much of this you need.

Keeping the Screen Awake with the Wake Lock API

You've built a beautiful fullscreen experience. The user leans back to watch it… and ninety seconds later the screen dims and locks, because the operating system saw no input and assumed nobody was there.

For a video element, the browser handles this for you. For anything else (like a countdown, slideshow, canvas animation, recipe, or sheet-music page), you need the Screen Wake Lock API. It landed in every major browser in 2024 (Chrome 84+, Safari 16.4+, Firefox 126+) and became part of Baseline in 2025, so you can use it today without a polyfill.

The API is a single call, and it's Promise-based:

let wakeLock = null;

async function requestWakeLock() {
  if (!('wakeLock' in navigator)) return; // unsupported – fail silently

  try {
    wakeLock = await navigator.wakeLock.request('screen');
    wakeLock.addEventListener('release', () => {
      // The OS or browser released it (tab hidden, battery saver, etc.)
      wakeLock = null;
    });
  } catch (err) {
    // Rejected – most often because the device is in a power-saving mode.
    console.warn(`Wake lock failed: ${err.name} – ${err.message}`);
  }
}

async function releaseWakeLock() {
  if (wakeLock) {
    await wakeLock.release();
    wakeLock = null;
  }
}

Three rules to know:

1. It requires a secure context. The API is only exposed over HTTPS (and localhost).

2. The browser releases the lock automatically when the page is hidden. If the user switches tabs, minimises the window, or locks the phone, when the page becomes visible again, the lock does not come back on its own. You have to re-request it:

document.addEventListener('visibilitychange', () => {
  if (document.visibilityState === 'visible' && shouldStayAwake()) {
    requestWakeLock();
  }
});

1. Be a good citizen. A wake lock drains batteries. Only hold it while it's actually useful, like while something is playing, running, or being displayed. Release it the moment that stops.

The nice part is that the Fullscreen API already gives you a perfect signal for "something is being displayed": fullscreenchange. When the user enters fullscreen, then request the lock. When they leave fullscreen, release it.

Fullscreen isn't the only good signal, though, and it helps to think of the wake lock as attached to an activity rather than to a display mode. On a countdown timer, I request the lock when the countdown starts and release it when it reaches zero or is paused (whether or not the page is fullscreen), because nobody wants a timer that goes dark at the 90-second mark. On a rain-sounds player, the lock follows the audio: it's acquired on play and released on pause or when the sleep timer fades the sound out.

The mechanics are identical: only the "should the screen stay awake right now?" question changes.

Putting It All Together

Here's the whole pattern in one place: a fullscreen toggle with a keyboard shortcut, a wake lock that follows fullscreen state, UI that stays in sync no matter how the user leaves, and a fallback for browsers that can't do it.

const toggleBtn = document.getElementById('fs-toggle');
let wakeLock = null;

/* ---------- Fullscreen compatibility layer ---------- */
const fs = {
  get element() {
    return document.fullscreenElement ?? document.webkitFullscreenElement ?? null;
  },
  get enabled() {
    return Boolean(document.fullscreenEnabled ?? document.webkitFullscreenEnabled);
  },
  request(el = document.documentElement) {
    if (el.requestFullscreen) return el.requestFullscreen();
    if (el.webkitRequestFullscreen) return Promise.resolve(el.webkitRequestFullscreen());
    return Promise.reject(new Error('Fullscreen not supported'));
  },
  exit() {
    if (document.exitFullscreen) return document.exitFullscreen();
    if (document.webkitExitFullscreen) return Promise.resolve(document.webkitExitFullscreen());
    return Promise.reject(new Error('Fullscreen not supported'));
  },
};

/* ---------- Wake lock ---------- */
async function requestWakeLock() {
  if (!('wakeLock' in navigator) || wakeLock) return;
  try {
    wakeLock = await navigator.wakeLock.request('screen');
    wakeLock.addEventListener('release', () => { wakeLock = null; });
  } catch (err) {
    console.warn(`Wake lock failed: ${err.name}`);
  }
}

async function releaseWakeLock() {
  if (!wakeLock) return;
  await wakeLock.release();
  wakeLock = null;
}

/* ---------- Toggle ---------- */
async function toggleFullscreen() {
  const pseudo = document.body.classList.contains('pseudo-fullscreen');
  try {
    if (pseudo) {
      document.body.classList.remove('pseudo-fullscreen');
      onFullscreenChange();
    } else if (!fs.element) {
      await fs.request();
    } else {
      await fs.exit();
    }
  } catch (err) {
    // The API refused (iframe policy, unsupported platform, etc.) – degrade gracefully.
    document.body.classList.add('pseudo-fullscreen');
    onFullscreenChange();
  }
}

/* ---------- Keep everything in sync ---------- */
function onFullscreenChange() {
  const active = Boolean(fs.element) || document.body.classList.contains('pseudo-fullscreen');
  toggleBtn.textContent = active ? 'Exit fullscreen' : 'Fullscreen';
  toggleBtn.setAttribute('aria-pressed', String(active));
  document.body.classList.toggle('is-fullscreen', active);

  // The screen should stay awake exactly as long as we're fullscreen.
  if (active) requestWakeLock(); else releaseWakeLock();
}

['fullscreenchange', 'webkitfullscreenchange'].forEach((evt) =>
  document.addEventListener(evt, onFullscreenChange)
);

// Re-acquire the lock if the tab was hidden and came back while fullscreen.
document.addEventListener('visibilitychange', () => {
  if (document.visibilityState === 'visible' && fs.element) requestWakeLock();
});

/* ---------- Wiring ---------- */
toggleBtn.addEventListener('click', toggleFullscreen);
document.addEventListener('keydown', (e) => {
  if (e.target.matches('input, textarea, [contenteditable]')) return;
  if (e.key === 'f' || e.key === 'F') { e.preventDefault(); toggleFullscreen(); }
});
// Unsupported platforms (iPhone Safari) simply take the catch branch above
// on the first click and land in pseudo-fullscreen – no user-agent sniffing needed.
// One more mobile quirk: rotating the device changes the viewport *after*
// the event fires, so re-measure your canvas or layout a beat later.
const resizeStage = () => { /* re-measure whatever fills the screen */ };
window.addEventListener('orientationchange', () => setTimeout(resizeStage, 100));

And the matching CSS:

html { background: #000; }

:fullscreen { overflow: hidden; cursor: none; }
:fullscreen::backdrop { background: #000; }

/* Fallback for browsers without element fullscreen */
body.pseudo-fullscreen .app { position: fixed; inset: 0; }
body.pseudo-fullscreen .chrome { display: none; }

Roughly sixty lines, and it's the same core I run in production: I log every fullscreen entry as an analytics event, because on a screen-tool site that transition is the conversion, and this exact code is what fires it.

It powers, for instance, the black screen page on blankscreen.io – a page whose entire job is to turn a display pure black, go fullscreen on a click or an F key, hide the pointer, stay awake for as long as it's on screen, and get out of the way cleanly on Esc. If you open it on an iPhone, you'll see the pseudo-fullscreen fallback from above doing its thing instead.

The same pattern in the wild: one click or the F key, the whole display goes black, and the screen stays awake for as long as it's showing. Esc brings the browser back.

Once you have this pattern, adding it to a timer, a slideshow, or a canvas experiment is a matter of dropping in the element you want to fill the screen.

A Quick Gotcha Checklist

Before you ship, run through this list. Every item is something I've been bitten by at least once:

• User gesture required: No fullscreen on load, on timer or after an await that takes a while.

• Don't track state yourself: Read document.fullscreenElement and listen for fullscreenchange.

• You can't blockEsc: Design for the user leaving at any moment.

• Old Safari wants webkit prefixes for the methods and the event name.

• iPhone Safari has no element fullscreen: Feature-detect with document.fullscreenEnabled, and treat a runtime rejection as a fallback trigger, not just a console warning.

• Check your shortcut against the page's own keys: If the app consumes letters, let the shortcut only enter fullscreen and leave exiting to Esc.

• Make the fullscreen surface keyboard-reachable (tabindex="0", Enter/Space).

• Wake locks need HTTPS: Can be rejected in battery-saver mode, and they're dropped when the page is hidden. Re-request on visibilitychange.

• Release the wake lock as soon as the reason for it is gone.

• Put the background on html, not just body, when fullscreening the page.

Conclusion

The Fullscreen API is one of those browser features that looks like a one-liner and turns out to have a personality.

The core (requestFullscreen(), exitFullscreen(), fullscreenElement, and the fullscreenchange event) really is simple. The craft is in respecting the user-gesture rule, keeping your UI honest about the real state, handling Safari's history, and remembering that "fullscreen" is only half of the job if the screen goes dark two minutes later. The Wake Lock API closes that gap, and pairing the two through fullscreenchange keeps both of them tidy.

Take the sixty-line pattern above, drop your own element into it, and you have a production-ready fullscreen experience for anything from a presentation to a game.

Happy coding!

ORMs like Sequelize, Prisma, TypeORM, and Knex parameterize the queries they build for you automatically. That part works.

The problem shows up in the parts of your app where you step outside that safe path: a raw query the ORM's query builder can't express cleanly, a dynamic sort column, or a single helper function reached for without a second thought.

Those are exactly the places attackers go looking first, and they're easy for developers to miss precisely because the rest of the codebase feels protected.

Four of those patterns are the focus here. Each one gets a working example that triggers the bug, a breakdown of why it's exploitable, and a rewritten version that closes it. I also include notes on exactly what moved and why.

You don't need a security background to follow along. You just need enough comfort with Node.js, SQL, and at least one ORM to read the code.

Prerequisites

The examples assume you're comfortable with:

• The basics of Node.js and Express

• Writing basic SQL queries

• Basic use of an ORM (all code examples use Sequelize, but the patterns apply to Prisma, TypeORM, Knex, and others too)

• How an HTTP request/response cycle works

Note on code examples: Throughout this guide, sequelize is assumed to already be an initialized Sequelize instance, with QueryTypes and Op imported from 'sequelize' alongside it, and User/Product are Sequelize models defined elsewhere in the app. Examples use Sequelize v6+ syntax (the current major version) since some of the APIs referenced here changed between major versions. You can adapt the syntax to the version and ORM you use.

Table of Contents

• 1. Raw Query Escape Hatches

• 2. Identifiers That Can't Be Parameterized

• 3. Second-Order Injection Through Stored Data

• 4. Raw SQL Smuggled Into Normal ORM Calls

• Summary

1. Raw Query Escape Hatches

Every major ORM ships an "escape hatch" for queries the query builder can't express cleanly: sequelize.query(), Prisma's $queryRawUnsafe, or TypeORM's query(). They exist for good reasons, like complex joins, window functions, and vendor-specific SQL.

The trouble starts when developers treat that escape hatch like the rest of the ORM, and interpolate user input directly into the string it builds.

How to Identify This Vulnerability in Your Code

Here's a typical filtered report endpoint:

// Express.js - Vulnerable
app.get('/reports', async (req, res) => {
  const { region } = req.query;
  const results = await sequelize.query(
    `SELECT * FROM sales WHERE region = '${region}'`,
    { type: QueryTypes.SELECT }
  );
  res.json(results);
});

The query builder never sees this string. It's handed straight to the database driver exactly as written.

Why This Matters

A request like ?region=' OR '1'='1 turns the query into SELECT * FROM sales WHERE region = '' OR '1'='1', which is always true. Every row in the table comes back, regardless of region.

Worse, because this code lives inside an ORM-based project, it often doesn't get the same scrutiny a raw mysql.query() call would in a non-ORM codebase. Reviewers assume the ORM already handled it.

How to Fix This Vulnerability

Pass values through the replacement/binding mechanism the raw-query API already provides, instead of building the string yourself:

// Express.js - Secure
app.get('/reports', async (req, res) => {
  const { region } = req.query;
  const results = await sequelize.query(
    'SELECT * FROM sales WHERE region = :region',
    {
      replacements: { region },
      type: QueryTypes.SELECT
    }
  );
  res.json(results);
});

The fix isn't avoiding raw queries entirely, because sometimes you genuinely need them. It's never building the SQL string by hand when a binding mechanism is sitting right there.

Never concatenate user input into a raw query string. Use the replacement/binding API your raw-query method already provides.

2. Identifiers That Can't Be Parameterized

Parameterized queries protect values. They don't protect identifiers like table names, column names, or ORDER BY direction. Those have to be part of the SQL string itself.

That's why a dynamic sort feature is one of the most common places injection sneaks back into otherwise well-written ORM code.

How to Identify This Vulnerability in Your Code

Here's a typical sortable list endpoint:

// Express.js - Vulnerable
app.get('/users', async (req, res) => {
  const { sortBy = 'created_at' } = req.query;
  const users = await sequelize.query(
    `SELECT * FROM users ORDER BY ${sortBy}`,
    { type: QueryTypes.SELECT }
  );
  res.json(users);
});

sortBy goes straight from the query string into the ORDER BY clause, with nothing in between.

Why This Matters

sortBy looks like a harmless UI convenience until someone sends created_at; DROP TABLE users; --. Whether that exact payload runs depends on your database driver: Postgres's pg driver executes stacked statements like this by default, while MySQL's mysql2 blocks them unless multipleStatements: true is explicitly set.

Either way, the underlying problem is the same: you have arbitrary SQL sitting in a position that was only ever meant to hold a column name.

How to Fix This Vulnerability

Since identifiers can't be bound as parameters, the only safe option is an allowlist:

// Express.js - Secure
const ALLOWED_SORT_COLUMNS = ['created_at', 'name', 'email'];

app.get('/users', async (req, res) => {
  const { sortBy = 'created_at' } = req.query;
  const column = ALLOWED_SORT_COLUMNS.includes(sortBy) ? sortBy : 'created_at';

  const users = await sequelize.query(
    `SELECT * FROM users ORDER BY ${column}`,
    { type: QueryTypes.SELECT }
  );
  res.json(users);
});

Never pass user input through to an identifier position, even after "sanitizing" it first. Sanitization for identifiers is much easier to get wrong than for values.

Identifiers can't be parameterized. If user input decides a column or table name, allowlist it, don't sanitize it.

3. Second-Order Injection Through Stored Data

This one catches teams off guard because the input was parameterized...the first time it was written to the database.

The injection happens later, when that already-stored value gets reused inside a different, unparameterized query.

How to Identify This Vulnerability in Your Code

Here's a registration flow, and elsewhere in the codebase, an admin search feature:

// Express.js - Vulnerable

// Step 1: user registration, correctly parameterized
app.post('/register', async (req, res) => {
  await User.create({ username: req.body.username });
  res.json({ success: true });
});

// Step 2: somewhere else in the codebase, an admin search feature
app.get('/admin/search', async (req, res) => {
  const user = await User.findByPk(req.params.id);
  const results = await sequelize.query(
    `SELECT * FROM audit_log WHERE actor = '${user.username}'`,
    { type: QueryTypes.SELECT }
  );
  res.json(results);
});

Step 1 is completely safe on its own. Step 2 is where things go wrong.

Why This Matters

A username like admin' OR '1'='1 passes through step 1 without any issue. Sequelize's create() parameterizes it, so it's stored exactly as typed. It sits in the database looking completely normal.

The payload only fires when step 2 pulls that value back out and drops it straight into a raw query string. "It's already in our own database" is not the same thing as "it's safe". It's still attacker-controlled if it originated from user input anywhere upstream.

How to Fix This Vulnerability

Same fix as pattern #1. You bind the value instead of concatenating it:

// Express.js - Secure

// Step 1 (registration) is unchanged — it was already parameterized

app.get('/admin/search', async (req, res) => {
  const user = await User.findByPk(req.params.id);
  const results = await sequelize.query(
    'SELECT * FROM audit_log WHERE actor = :actor',
    {
      replacements: { actor: user.username },
      type: QueryTypes.SELECT
    }
  );
  res.json(results);
});

The bug here is harder to spot than pattern #1, because the tainted data crosses a database round-trip before it becomes dangerous.

Data from your own database isn't automatically safe. If it originated from user input anywhere upstream, it still needs to be parameterized everywhere it's used.

4. Raw SQL Smuggled Into Normal ORM Calls

Pattern #1 covered the obvious raw-query escape hatch, a method you reach for on purpose when you need real SQL. This one is sneakier: injection hiding inside a call that looks completely ORM-managed. It's not raw SQL, right? It's just one helper function among many.

How to Identify This Vulnerability in Your Code

Here's a typical filtered product listing:

// Express.js - Vulnerable
app.get('/products', async (req, res) => {
  const { minPrice } = req.query;
  const products = await Product.findAll({
    where: sequelize.literal(`price > ${minPrice}`)
  });
  res.json(products);
});

Product.findAll() looks like a fully safe, parameterized ORM call...right up until sequelize.literal() shows up inside it.

Why This Matters

literal() tells Sequelize "don't touch this, insert it into the SQL exactly as written." Anything interpolated into it is exactly as injectable as pattern #1, just wearing a normal-looking ORM method as a disguise.

Send ?minPrice=0 OR 1=1 and the WHERE clause becomes unconditionally true: every row comes back, price filter or not. No stacked statement needed this time. It's a single boolean expression, so it works the same regardless of database driver.

How to Fix This Vulnerability

Stop reaching for literal(). Sequelize's own operator API already covers this case:

// Express.js - Secure
app.get('/products', async (req, res) => {
  const minPrice = Number(req.query.minPrice);

  if (!Number.isFinite(minPrice)) {
    return res.status(400).json({ error: 'minPrice must be a number' });
  }

  const products = await Product.findAll({
    where: { price: { [Op.gt]: minPrice } }
  });
  res.json(products);
});

Op.gt, Op.between, Op.in, and the rest of Sequelize's operator API already cover the vast majority of cases people reach for literal() for, and they parameterize correctly by default. Validating that minPrice is actually a number, before it ever reaches the query, closes the gap even if a future refactor reintroduces literal() somewhere else.

Grep for literal(, fn(, and your ORM's other raw escape hatches everywhere they appear in the codebase, not just inside the queries that are clearly "raw."

Summary

For a quick recap, here's how the four patterns line up:

None of these four bugs takes a skilled attacker to find. Each one is just a value that never made it into the ORM's parameterization path.

What's worth making part of your workflow: parameterize values, allowlist identifiers, treat data pulled from your own database as no safer than a fresh request, and treat literal()/raw() calls as exactly as dangerous as a dedicated raw-query method (no matter how safe the surrounding code looks).

Fixing these gaps is only half the picture. Knowing how someone actually goes looking for them (and chains them together during a real assessment) is the half most developers never see.

If you're curious about that offensive perspective, this deep dive on SQL injection through ORMs walks through exactly how these four patterns get exploited step by step in a live penetration test.

While backend compression tools are common, building a client-side image compressor offers a massive advantage: privacy. When you process images directly in the browser, no user data ever touches a server.

In this tutorial, you'll learn how to build a fully functional, browser-based bulk image compressor. You'll use the HTML5 Canvas API to reduce image file sizes and integrate the JSZip library to package multiple compressed files into a single, convenient ZIP download.

To make this project highly practical, you'll structure the code as an embeddable widget. By omitting standard HTML boilerplate tags, you can easily drop this snippet directly into a WordPress Custom HTML block or any other CMS without causing layout conflicts.

Prerequisites

To follow along, you should have a basic understanding of:

• HTML & CSS: Structuring a UI and creating interactive hover/drag states.

• JavaScript Promises: Handling asynchronous operations like file reading and ZIP generation.

• The Canvas API: Understanding how browsers can draw and manipulate image data natively.

Table of Contents

• Step 1: Build the HTML Structure

  • Understanding the HTML

• Step 2: Style the Interface with CSS

  • Understanding the CSS

• Step 3: Add the JavaScript Logic

  • Understanding the JavaScript

• Conclusion

Step 1: Build the HTML Structure

First, you need to create the visual interface. You'll build a drag-and-drop zone, an invisible file input, and a hidden action panel that appears once the user selects their images.

Add the following code to your file. Notice that you're including the JSZip CDN link right at the top so your script can access it later.

<!-- JSZip Library for Bulk Downloading -->
<script src="https://cdnjs.cloudflare.com/ajax/libs/jszip/3.10.1/jszip.min.js"></script>

<div class="ic-main-wrapper">
    <div class="ic-title">Image Compressor</div>
    
    <!-- The Drag and Drop Zone -->
    <div class="ic-dropzone-area" id="icDropzone">
        <input type="file" id="icFileInput" multiple accept="image/jpeg, image/png, image/webp, image/gif, image/bmp, image/tiff, image/avif" style="display: none;">
        
        <div class="ic-icon-box">
            <svg viewBox="0 0 24 24">
                <path d="M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zm0 16H5V5h14v14zm-5-7l-3 3.72L9 13l-3 4h14l-4-5z"/>
            </svg>
        </div>
        
        <div class="ic-primary-text">
            Drop images here or <span class="ic-browse-link" id="icBrowseBtn">click to browse</span>
        </div>
        <div class="ic-secondary-text">
            Multiple images supported — bulk compress & download as ZIP
        </div>
        
        <div class="ic-format-tags">
            <span>JPEG</span><span>PNG</span><span>WebP</span><span>GIF</span><span>BMP</span><span>TIFF</span><span>AVIF</span>
        </div>
    </div>
    
    <!-- Action Panel (Hidden by default) -->
    <div class="ic-action-panel" id="icActionPanel">
        <div id="icStatusText" class="ic-status-msg">0 images selected</div>
        <button class="ic-btn-primary" id="icCompressBtn">Compress & Download ZIP</button>
    </div>
</div>

Understanding the HTML:

• accept="..." attribute: This restricts the hidden <input type="file"> to only accept image formats, preventing users from accidentally uploading PDFs or text documents.

• Embeddable Structure: Because this markup relies on a single .ic-main-wrapper container rather than full <html> and <body> tags, you can safely embed it into existing web pages without breaking the parent theme.

Step 2: Style the Interface with CSS

Next, you'll apply styling to make the tool look professional and responsive. You'll use a clean, modern aesthetic with a specific brand accent color (#1A73E8) to highlight interactive elements.

Add this <style> block right above your HTML:

<style>
.ic-main-wrapper {
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
    max-width: 850px;
    margin: 20px auto;
    background: #ffffff;
    border-radius: 12px;
    padding: 30px;
    box-shadow: 0 4px 20px rgba(0,0,0,0.04);
}

.ic-title {
    font-size: 2.2rem;
    font-weight: 700;
    color: #0f172a;
    text-align: center;
    margin-bottom: 30px;
}

.ic-dropzone-area {
    border: 1.5px dashed #cbd5e1;
    border-radius: 12px;
    padding: 60px 20px;
    text-align: center;
    background-color: #f8fafc;
    transition: all 0.3s ease;
}

/* Active Drag State */
.ic-dropzone-area.dragover {
    border-color: #1A73E8;
    background-color: #f1f5f9;
}

.ic-icon-box {
    width: 64px;
    height: 64px;
    background-color: #e8f0fe;
    border-radius: 50%;
    display: flex;
    align-items: center;
    justify-content: center;
    margin: 0 auto 24px;
    color: #1A73E8;
}

.ic-icon-box svg {
    width: 28px;
    height: 28px;
    fill: currentColor;
}

.ic-primary-text {
    font-size: 1.15rem;
    color: #0f172a;
    font-weight: 500;
    margin-bottom: 12px;
}

.ic-browse-link {
    color: #1A73E8;
    cursor: pointer;
    text-decoration: none;
    font-weight: 500;
}

.ic-browse-link:hover {
    text-decoration: underline;
}

.ic-secondary-text {
    font-size: 0.95rem;
    color: #64748b;
    margin-bottom: 24px;
}

.ic-format-tags {
    display: flex;
    flex-wrap: wrap;
    gap: 12px;
    justify-content: center;
}

.ic-format-tags span {
    border: 1px solid #e2e8f0;
    background: #ffffff;
    color: #64748b;
    font-size: 0.8rem;
    padding: 6px 18px;
    border-radius: 20px;
}

.ic-action-panel {
    margin-top: 25px;
    display: none;
    text-align: center;
    background: #f8fafc;
    padding: 20px;
    border-radius: 8px;
    border: 1px solid #e2e8f0;
}

.ic-btn-primary {
    background: #1A73E8;
    color: #ffffff;
    border: none;
    padding: 12px 28px;
    border-radius: 6px;
    font-size: 1rem;
    cursor: pointer;
    font-weight: 500;
    transition: background 0.3s;
}

.ic-btn-primary:hover {
    background: #1557b0;
}

.ic-btn-primary:disabled {
    background: #94a3b8;
    cursor: not-allowed;
}

.ic-status-msg {
    margin-bottom: 15px;
    font-size: 0.95rem;
    color: #334155;
    font-weight: 500;
}
</style>

Understanding the CSS:

• The .dragover Class: This class alters the border and background color of the dropzone. You'll use JavaScript to apply this class dynamically when a user hovers a file over the area, providing crucial visual feedback.

• Unique Prefixes: Notice how every class starts with ic- (Image Compressor). This acts as a CSS namespace, ensuring your tool's styles won't accidentally override or be overridden by other styles on your website.

At this stage, your user interface is fully structured and styled. It will look like this:

Step 3: Add the JavaScript Logic

Now comes the engine of the tool. The JavaScript will handle file selection, convert the images using the Canvas API to reduce their size, and package them into a ZIP file.

Add this <script> block below your HTML:

<script>
document.addEventListener('DOMContentLoaded', () => {
    const dropzone = document.getElementById('icDropzone');
    const fileInput = document.getElementById('icFileInput');
    const browseBtn = document.getElementById('icBrowseBtn');
    const actionPanel = document.getElementById('icActionPanel');
    const statusText = document.getElementById('icStatusText');
    const compressBtn = document.getElementById('icCompressBtn');
    
    let filesArray = [];

    // 1. Handle File Input and Drag & Drop
    browseBtn.addEventListener('click', (e) => {
        e.stopPropagation();
        fileInput.click();
    });
    
    dropzone.addEventListener('dragover', (e) => {
        e.preventDefault();
        dropzone.classList.add('dragover');
    });
    
    dropzone.addEventListener('dragleave', () => {
        dropzone.classList.remove('dragover');
    });
    
    dropzone.addEventListener('drop', (e) => {
        e.preventDefault();
        dropzone.classList.remove('dragover');
        if (e.dataTransfer.files.length) {
            processSelectedFiles(e.dataTransfer.files);
        }
    });

    fileInput.addEventListener('change', () => {
        if (fileInput.files.length) {
            processSelectedFiles(fileInput.files);
        }
    });

    // 2. Validate and Display the Action Panel
    function processSelectedFiles(files) {
        filesArray = Array.from(files).filter(file => file.type.startsWith('image/'));
        if (filesArray.length > 0) {
            actionPanel.style.display = 'block';
            statusText.innerText = `${filesArray.length} image(s) selected. Ready to compress.`;
        }
    }

    // 3. The Core Compression Engine
    function compressImage(file) {
        return new Promise((resolve) => {
            const reader = new FileReader();
            reader.onload = (event) => {
                const img = new Image();
                img.onload = () => {
                    const canvas = document.createElement('canvas');
                    canvas.width = img.width;
                    canvas.height = img.height;
                    const ctx = canvas.getContext('2d');
                    
                    // Draw the image onto the canvas
                    ctx.drawImage(img, 0, 0);
                    
                    // Force compatibility: Convert unusual formats to JPEG or WebP
                    let mimeType = file.type;
                    if (mimeType !== 'image/jpeg' && mimeType !== 'image/webp') {
                        mimeType = 'image/jpeg';
                    }
                    
                    // Compress using the canvas.toBlob method at 70% quality (0.7)
                    canvas.toBlob((blob) => {
                        let finalName = file.name;
                        
                        // Ensure the file extension matches the new mimeType
                        if (mimeType === 'image/jpeg' && !finalName.match(/\.(jpg|jpeg)$/i)) {
                            finalName = finalName.substring(0, finalName.lastIndexOf('.')) + '.jpg';
                        }
                        
                        resolve({ name: finalName, blob: blob });
                    }, mimeType, 0.7); 
                };
                img.src = event.target.result;
            };
            reader.readAsDataURL(file);
        });
    }

    // 4. Handle the Bulk ZIP Process
    compressBtn.addEventListener('click', async () => {
        if (filesArray.length === 0) return;
        
        compressBtn.disabled = true;
        
        try {
            const zip = new JSZip();
            const folder = zip.folder("compressed_images");
            
            // Loop through each file and await its compression
            for (let i = 0; i < filesArray.length; i++) {
                statusText.innerText = `Compressing ${i + 1} of ${filesArray.length}...`;
                const compressedData = await compressImage(filesArray[i]);
                folder.file(compressedData.name, compressedData.blob);
            }
            
            statusText.innerText = 'Creating ZIP file...';
            
            // Generate and trigger the ZIP download
            const zipContent = await zip.generateAsync({ type: "blob" });
            const downloadLink = document.createElement('a');
            downloadLink.href = URL.createObjectURL(zipContent);
            downloadLink.download = "compressed_images.zip";
            downloadLink.click();
            
            statusText.innerText = 'Download successful!';
        } catch (error) {
            statusText.innerText = 'An error occurred during compression.';
            console.error(error);
        } finally {
            compressBtn.disabled = false;
        }
    });
});
</script>

Understanding the JavaScript:

• Drag and drop events: The dragover, dragleave, and drop event listeners work together to capture files dragged from a user's desktop directly into the browser window.

• The canvas trick: Browsers can't magically compress files on their own. Instead, the compressImage function reads the image, draws it onto an invisible HTML5 <canvas>, and then uses the canvas.toBlob() method to export a new, optimized version of that image. The 0.7 argument sets the compression quality to 70%.

• Extension handling: Because the canvas exports images as either JPEG or WebP, the script actively checks and renames the file extensions (for example, changing a .png filename to .jpg) before packaging them.

• Asynchronous zipping: A for loop uses await to compress each image sequentially. Once complete, JSZip bundles the blobs into a folder and triggers a programmatic click event on a temporary anchor <a> tag to download the final .zip file.

Conclusion

You've successfully built a fast, client-side bulk image compressor.

By leveraging the Canvas API alongside JSZip, you created a highly practical utility that saves users bandwidth and storage without compromising their privacy. Because the entire codebase is self-contained without HTML boilerplate, it's ready to be deployed instantly on almost any modern web platform.

If you want to see this codebase deployed in a live, production environment, you can test out the functionality over at this Live Image Compressor Tool. Happy coding!

Those fifteen lines will also quietly report that a broken controller is fine.

I found this out the slow way, building a browser-based controller tester. A user emailed to say the site told him his gamepad was healthy when the stick was visibly drifting in every game he owned. He was right. The browser had handed us zeros.

This article covers the parts of the Gamepad API that aren't in the spec docs and that cost me real debugging time: why you have to poll, why the values you get on page load aren't the values the hardware sent, why you can't tell what controller is plugged in, and how to tell a drifting analog stick apart from a person holding one.

All the code here runs in a browser console with a controller connected. Press a button first, or the API will pretend nothing is plugged in.

Table of Contents

• Prerequisites

• The Tester That Doesn't Work

• Why You Have to Poll

• The Sanitization Rule

• You Can't Identify the Hardware, Either

• Telling Drift from a Human Hand

• Known Limits

• Wrapping Up

Prerequisites

This is a hands-on guide. There's nothing to install and no build step, but a few things need to be true before the code below will do anything.

What you should already know:

• JavaScript at a working level: functions, arrays and array methods like reduce and filter, arrow functions, and destructuring.

• What an animation frame loop is. Several of the examples run inside requestAnimationFrame.

• How to open your browser's developer tools and paste code into the console.

One section does a little vector arithmetic: the mean of a set of x and y samples and the length of that mean vector. If Math.hypot(x, y) makes sense to you, that section will too.

What you need to have:

• A desktop browser that supports the Gamepad API. Chrome, Edge, Firefox and Safari have all supported it since 2017, so whatever you have open is almost certainly fine.

• A physical game controller, connected by USB or Bluetooth. There's no way to fake one in software, and none of the code below does anything useful without hardware attached.

• Ideally, a controller you know is faulty, like one with stick drift if you have it. Several of the behaviours in this article only show up on broken hardware. A healthy controller will hide them from you.

You don't need any frameworks or libraries, or npm install. Every block below is plain JavaScript that runs as written.

The Tester That Doesn't Work

Here's the version almost everyone writes first. It's the version in most tutorials.

window.addEventListener("gamepadconnected", (e) => {
  const pad = navigator.getGamepads()[e.gamepad.index];
  console.log(pad.axes);    // [0, 0, 0, 0]
  console.log(pad.buttons.filter(b => b.pressed).length);   // 0
});

Plug in a controller with severe stick drift, one that pulls a character across the screen on its own in every game, and this prints [0, 0, 0, 0].

There are two separate bugs in those five lines, and the second one is the interesting one.

Why You Have to Poll

The first bug is that there are no input events. gamepadconnected and gamepaddisconnected fire, and that's the entire event surface. There's no gamepadaxischange and no gamepadbuttondown. If you want to know what the sticks are doing, you have to ask, over and over, usually in requestAnimationFrame.

The second part of the same bug: you have to call navigator.getGamepads() again every single frame. It returns snapshots. Holding on to a Gamepad object and reading it later gets you the values from the moment you grabbed it, frozen, forever.

function loop() {
  const pads = navigator.getGamepads();     // re-read every frame, do not cache
  for (const pad of pads) {
    if (!pad) continue;                     // the array has empty slots, always guard
    render(pad.index, pad.axes, pad.buttons);
  }
  requestAnimationFrame(loop);
}
requestAnimationFrame(loop);

Two practical notes on that loop.

First, the array is sparse. navigator.getGamepads() returns a fixed length array with null in the slots that have nothing connected, so a plain for...of without the guard will throw on the first null.

Second, polling isn't free. A requestAnimationFrame loop that starts at page load and runs forever is real main thread work on a page that may have no controller connected at all and never will.

Here's a pattern that works well: idle at a low rate, something like 8 times a second with setTimeout, purely to notice a controller appearing, then switch to full requestAnimationFrame once one is actually connected, and drop back down when it disconnects. The API is cheap to sample, but sampling it 60 times a second on every page view for nothing is a waste you'll see in a performance profile.

The Sanitization Rule

Now the part that is genuinely under-documented, and the reason the drifting controller reported zeros.

Chromium won't report an axis's real value until it has seen that axis at rest at least once.

Not until the user moves it. Until the browser observes it near zero.

The mechanism is in one file, device/gamepad/gamepad_pad_state_provider.cc. The browser keeps two bitfields per connected controller, an axis_mask and a button_mask. While an axis's bit is unset, its reported value is forced to 0.0. The bit gets set the first time that axis reports a magnitude below a constant called kMinAxisResetValue, which is 0.1f. From then on, real values flow through.

Buttons work the same way through button_mask, with a stricter test: the bit is set the first time the button reports as not pressed. A button that's held down as the page loads, or a trigger that a broken spring is holding halfway, reports pressed: false and value: 0 until the browser sees it released once.

This isn't a bug, and it's worth understanding why it's there. The comment in the source explains it: a controller can report input when nobody is touching it, because of a hardware fault or because something heavy is leaning on a stick. Without this rule, that stray input would be treated as a user gesture, and the page would learn about a device the user never chose to reveal. So each axis and each button has to prove it can sit at rest before the browser will tell you anything about it.

Read the consequence carefully, because it's the opposite of what you would guess:

The worse the drift, the longer the browser insists the controller is fine.

A stick with a small offset will pass under 0.1 on some frame soon enough and unmask itself. A badly worn stick that never settles back inside that window stays masked indefinitely. The controller that most needs reporting is the one that reports nothing.

This also explains something that looks like magic in controller testers. Instructions like "move both sticks in a full circle" don't work because movement unlocks the axis. They work because a full circle passes through the centre on the way back.

Here is a demo you can paste into a console. Connect a controller, load the page, and don't touch the sticks. Then push the left stick to the edge and let it spring back.

const start = performance.now();
let woke = false;

requestAnimationFrame(function loop() {
  const pad = navigator.getGamepads()[0];
  if (pad && !woke) {
    const [x, y] = pad.axes;
    if (x !== 0 || y !== 0) {
      woke = true;
      console.log(
        "left stick started reporting after",
        Math.round(performance.now() - start), "ms,",
        "first values:", x.toFixed(3), y.toFixed(3)
      );
    }
  }
  requestAnimationFrame(loop);
});

On a healthy controller sitting still, the axes unmask almost immediately, because a healthy stick rests at roughly zero. On a drifting one, nothing is logged until you send the stick through the centre yourself.

The practical rule that falls out of this: never draw a conclusion about hardware from the first frame after connection. Wait until you've seen each axis report a non zero value at least once, or ask the user to move the sticks, and only then trust what you're reading.

You Can't Identify the Hardware, Either

The second surprise is smaller but it will bite you in the UI layer.

The spec gives you pad.id, a string the browser makes up. On Linux and often on macOS it contains a USB vendor and product ID in hex, and you can look the device up. On Windows, XInput devices (which is to say most Xbox style controllers) expose no vendor or product ID at all. The string looks like "Xbox 360 Controller (XInput STANDARD GAMEPAD)", and a third party clone reports exactly the same thing as first party hardware.

macOS has its own version of this. A DualShock 4 connected to Chrome on macOS arrives as "Wireless Controller (STANDARD GAMEPAD)". No vendor ID, no product ID, and a name generic enough that half a dozen unrelated controllers share it.

That last one cost me a real bug. Our glyph rendering keyed off a parsed id string, so every DualShock 4 on a Mac fell through to the generic fallback and drew Xbox-style button labels on a PlayStation controller. Every Mac user of that feature saw the wrong thing for months, and no Windows or Linux test would ever have caught it.

Branch on capability instead:

function describe(pad) {
  return {
    standard: pad.mapping === "standard",   // trust axes/buttons ordering only if true
    axes: pad.axes.length,                  // 4 on a normal twin stick pad
    buttons: pad.buttons.length,            // 17 on standard mapping with a guide button
    analogTriggers: pad.buttons.slice(6, 8).every(b => typeof b.value === "number"),
    rumble: Boolean(pad.vibrationActuator)
  };
}

Use pad.id for display, and to let the user confirm what they have plugged in. Don't use it to decide what your code does.

Telling Drift from a Human Hand

Once you can actually read the sticks, you hit the real problem: an off centre reading doesn't mean the hardware is broken. It usually means a person is holding the stick.

The obvious detector is a threshold and a timer. If an axis stays past some value for N milliseconds, call it drift. I shipped that. It was wrong, and it was wrong in the worst direction, because our own on screen instruction told users to rotate both sticks in full circles, and a slow circle holds an axis past a threshold for a long time. The tester told people their working controllers were broken.

What separates the two cases isn't how far the stick is from centre. It's two things together.

Gate one, is it near rest. Real drift is a small persistent offset, typically well under half deflection. A hand on a stick is usually much further out. Require the mean magnitude over the sample window to be below about 0.6.

Gate two, is it directionally coherent. This is the one that does the work. Drift comes from a worn or miscalibrated sensor, so it holds one direction with very little variation. A human hand wanders, even when trying to hold still. Compare the length of the mean vector to the mean of the individual magnitudes. If every sample points the same way, those two numbers are nearly equal and the ratio approaches 1. If the samples fan out, the mean vector is shorter than the mean magnitude and the ratio drops. Require above about 0.9.

// samples: array of { x, y } collected over a rolling window, one per frame
function looksLikeDrift(samples) {
  if (samples.length < 30) return false;                 // not enough evidence yet

  const magnitude = s => Math.hypot(s.x, s.y);
  const meanMagnitude =
    samples.reduce((sum, s) => sum + magnitude(s), 0) / samples.length;

  if (meanMagnitude < 0.02) return false;                // resting at centre, nothing wrong
  if (meanMagnitude > 0.6) return false;                 // gate 1: too far out to be drift

  const meanX = samples.reduce((sum, s) => sum + s.x, 0) / samples.length;
  const meanY = samples.reduce((sum, s) => sum + s.y, 0) / samples.length;
  const coherence = Math.hypot(meanX, meanY) / meanMagnitude;

  return coherence > 0.9;                                // gate 2: holds one direction
}

And the collector that feeds it:

const window_ = [];
const WINDOW = 120;   // about two seconds at 60fps

requestAnimationFrame(function loop() {
  const pad = navigator.getGamepads()[0];
  if (pad) {
    window_.push({ x: pad.axes[0], y: pad.axes[1] });
    if (window_.length > WINDOW) window_.shift();
    if (looksLikeDrift(window_)) console.log("left stick looks like drift");
  }
  requestAnimationFrame(loop);
});

The measurement, because a claim like this is worth a number: replaying the same 64 seconds of recorded live controller input, the threshold and timer version raised a drift condition on 2,144 frames. The two gate version raised it on zero. That recording contained no drifting hardware. Every one of those 2,144 frames was a person moving a stick, mostly following our own instructions.

Neither gate is magic, and it's worth saying where this one still fails. A hand held deliberately still and off centre in a single direction passes both gates, because that's genuinely hard to tell apart from a worn sensor by looking at the numbers alone. The fix isn't a third gate, it's context: run the check during a moment when you've asked the user to let go of the sticks, rather than against arbitrary input. Detection logic gets much easier when you control the conditions it runs in.

One design rule came out of that, and it generalises well beyond controllers: a gate may only suppress a report, never create one. Both gates can veto. Neither can raise the alarm on its own. If you find yourself adding a rule that turns a quiet signal into a loud one, you're building a false positive generator.

Known Limits

There are four things worth knowing before you ship.

Vibration isn't portable. Feature detect pad.vibrationActuator and treat rumble as a bonus, never as a requirement.

Non-standard mappings are real. When pad.mapping isn't "standard", the axis and button ordering is whatever the browser and driver agreed on, and index 0 isn't guaranteed to be anything in particular. Handle that case or refuse it explicitly, but don't assume it away.

Bluetooth polling is less consistent than USB. Sample intervals wobble, so anything you compute from timing should be tolerant of jitter rather than assuming a steady 60Hz.

And browser support for newer controllers lags the hardware. A controller released last year may be recognised by one browser and not another on the same machine, which makes "my controller doesn't work" a browser question at least as often as a hardware one.

Wrapping Up

The Gamepad API is small and mostly pleasant to work with. The thing to carry away is that it's not a direct line to the hardware. The browser sits in between, protecting the user from the page, and the value it hands you isn't always the value the controller sent.

So poll instead of listening, re-read getGamepads() every frame, wait for each axis to prove itself before you trust it, branch on capability rather than on the id string, and require more than a threshold before you tell someone their hardware is broken.

And test with a controller you know is broken. A tester that has only ever been run against working hardware hasn't been tested at all.

I build JoyCheck, a browser-based gamepad tester, which is where these measurements come from.

An AI assistant can help with that workflow, but only if it can query a real catalog and persist what you already decided. Chat history isn't a database. A hallucinated deadline is worse than no deadline at all.

The Model Context Protocol (MCP) is the standard way to give AI apps that kind of access. In this tutorial, you'll build a scholarship research MCP server with Node.js, Express, and MongoDB.

When you finish, Cursor, Claude Desktop, or any other MCP host will be able to search awards, match them to a student profile, bookmark a shortlist, and store research notes. The model stays the reasoning layer while your server owns the data.

You'll build:

• A MongoDB catalog of scholarships, plus saved-list and notes collections

• Nine MCP tools for search, matching, deadlines, comparison, and research tracking

• Resources so a client can read the catalog without calling a tool

• Prompts that turn a student profile into a research plan

• An Express app that serves MCP over Streamable HTTP

The sample catalog in this project is a teaching dataset. Amounts, dates, and eligibility rules are simplified. Always confirm details on the official application page before applying.

What You Need

You should be comfortable with JavaScript and basic Express routing. You don't need prior MCP experience.

Install:

• Node.js 20 or later

• MongoDB running locally, or a free MongoDB Atlas cluster

• An MCP client if you want to try the last section. Cursor and Claude Desktop both work.

Docker is enough for MongoDB:

docker run -d --name mongo -p 27017:27017 mongo:7

Table of Contents

• What You Need

• What Is the Model Context Protocol?

• Why a Scholarship Research Server?

• How the Architecture Fits Together

• How to Set Up the Project

• How to Connect MongoDB

• How to Model Scholarship Data

• How to Write the Scholarship Service

• How to Register MCP Tools

• How to Expose Resources and Prompts

• How to Serve MCP Over Express

• How to Seed the Catalog

• How to Test the Server

• How to Connect Cursor and Claude Desktop

• How a Research Session Runs

• How the Matching Logic Works

• What You Can Build Next

• Conclusion

What Is the Model Context Protocol?

MCP is an open protocol that lets an AI application talk to external tools and data sources through a shared contract. Anthropic introduced it in 2024, and it's now maintained as an open standard.

Anthropic described MCP as "a USB-C port for AI applications": one connector, many hosts. (Source: Introducing the Model Context Protocol) Instead of writing one integration for Cursor, another for Claude Desktop, and a third for a custom agent, you implement the protocol once.

The official architecture overview splits MCP into two layers:

• The data layer is JSON-RPC 2.0. Clients and servers exchange requests such as tools/list and tools/call.

• The transport layer moves those messages. Local servers usually use stdio. Remote or long-running servers use Streamable HTTP.

This tutorial uses Streamable HTTP, because Express is already an HTTP server and you want the catalog available to any client on your machine.

Hosts, Clients, and Servers

Three roles show up in every MCP setup:

• The host is the AI app. Cursor and Claude Desktop are hosts.

• The client lives inside the host. The host creates one client per connected server.

• The server is your program. It advertises tools, resources, and prompts, then handles calls.

Your scholarship app is the server. You never talk to the model SDK directly. The host does that.

Tools, Resources, and Prompts

MCP servers expose three primitives. You'll use all three.

Tools are actions. The model decides to call them, the way it might call a function in a regular tool-calling API. Search, save, and compare belong here.

Resources are data the host can read and attach as context. A catalog URI and a per-scholarship URI belong here. The model doesn't have to "take an action" to see them.

Prompts are named templates. People usually invoke them from a slash command or a menu. A "research plan" prompt belongs here, because it's a workflow you want to start on purpose.

That split matters. If you put everything in tools, the model has to guess when to look things up. Resources and prompts give the host better knobs.

Why a Scholarship Research Server?

A weather MCP demo is a single API call. Scholarship research is closer to a real product:

• The catalog must be queryable. Keyword search, GPA filters, and deadline windows all live in the database.

• The workflow must persist. A saved shortlist and research notes should survive a new chat.

• Eligibility is logic, not prose. A GPA minimum is a number. First-generation-only is a boolean. Put those checks in code so the model can't invent a match.

• The output must be inspectable. Students should be able to open the official URL and verify every claim.

MongoDB fits this well. Each scholarship is a document with nested arrays for fields of study, citizenship, and requirements. Saved items and notes are separate collections with references back to the catalog.

You could wrap a public API instead of storing documents. That's a good follow-up. Starting with your own catalog keeps the tutorial self-contained and makes the MCP contract obvious.

How the Architecture Fits Together

The finished project looks like this:

MCP host (Cursor or Claude Desktop)
        |
        |  Streamable HTTP  POST /mcp
        v
Express app  (createMcpExpressApp)
        |
        |  createMcpHandler factory
        v
McpServer  tools / resources / prompts
        |
        v
Scholarship service
        |
        v
MongoDB  scholarships, savedScholarships, researchnotes

A few design choices are worth calling out before you write code.

The MCP handler is stateless. The SDK runs your server factory once per HTTP request. That's the recommended v2 pattern for Streamable HTTP. Don't keep tool state on the McpServer instance. Keep it in MongoDB.

The database connection is process-wide. Connecting on every request would be slow and pointless. You connect once at startup and close over that pool from the tools.

The HTTP surface is small on purpose. /health is for you. /mcp is for the protocol. You don't need a REST API in front of the same data unless you want one later.

How to Set Up the Project

Create a folder and initialize a Node.js project. ESM is required because the MCP SDK is ESM-first.

mkdir scholarship-research-mcp-server
cd scholarship-research-mcp-server
npm init -y

Open package.json and set "type": "module". Then install the SDK, Express, Mongoose, Zod, and dotenv:

npm install @modelcontextprotocol/server @modelcontextprotocol/express @modelcontextprotocol/node express mongoose dotenv zod

The SDK split into packages in v2:

• @modelcontextprotocol/server is the McpServer class and createMcpHandler

• @modelcontextprotocol/express gives you createMcpExpressApp, including DNS rebinding protection

• @modelcontextprotocol/node adapts the web-standard handler to Node's req/res

Create a .env file:

MONGODB_URI=mongodb://127.0.0.1:27017/scholarship_research
PORT=3000
HOST=127.0.0.1

Add a .gitignore that excludes node_modules and .env.

Your source layout can stay small:

src/
  index.js
  config.js
  db.js
  models/
  services/
  mcp/
  utils/
data/
  scholarships.json
scripts/
  seed.js
  smoke-test.js

src/config.js reads environment variables with defaults:

export const config = {
  mongodbUri: process.env.MONGODB_URI ?? "mongodb://127.0.0.1:27017/scholarship_research",
  port: Number(process.env.PORT ?? 3000),
  host: process.env.HOST ?? "127.0.0.1",
};

Keep configuration in one file. Tools shouldn't read process.env directly.

How to Connect MongoDB

Mongoose 9 works cleanly with ESM. A short src/db.js is enough:

import mongoose from "mongoose";
import { config } from "./config.js";

export async function connectDatabase() {
  mongoose.set("strictQuery", true);
  await mongoose.connect(config.mongodbUri);
  return mongoose.connection;
}

Call this once in src/index.js before app.listen. If the connection fails, the process should exit. A running Express server with a dead database is harder to debug than a failed startup.

How to Model Scholarship Data

You need three collections.

Scholarship

This is the catalog. Store the fields a matching engine actually uses, not a blob of markdown.

import mongoose from "mongoose";

const scholarshipSchema = new mongoose.Schema(
  {
    title: { type: String, required: true, trim: true },
    provider: { type: String, required: true, trim: true },
    description: { type: String, required: true },
    amountMin: { type: Number, default: 0 },
    amountMax: { type: Number, default: 0 },
    currency: { type: String, default: "USD" },
    deadline: { type: Date, default: null },
    rolling: { type: Boolean, default: false },
    educationLevels: { type: [String], default: ["undergraduate"] },
    fieldsOfStudy: { type: [String], default: ["any"] },
    gpaMinimum: { type: Number, default: null },
    citizenship: { type: [String], default: ["any"] },
    countries: { type: [String], default: ["any"] },
    firstGenerationOnly: { type: Boolean, default: false },
    womenOnly: { type: Boolean, default: false },
    numberOfAwards: { type: Number, default: 1 },
    renewable: { type: Boolean, default: false },
    applicationUrl: { type: String, required: true },
    applicationRequirements: { type: [String], default: [] },
  },
  { timestamps: true },
);

scholarshipSchema.index({
  title: "text",
  provider: "text",
  description: "text",
  fieldsOfStudy: "text",
});
scholarshipSchema.index({ deadline: 1 });
scholarshipSchema.index({ amountMax: -1 });

export const Scholarship = mongoose.model("Scholarship", scholarshipSchema);

A few field choices are doing real work:

• fieldsOfStudy: ["any"] means the award is field-open. The matcher treats any as a wildcard.

• deadline: null plus rolling: true covers programs that accept applications year-round.

• applicationUrl is mandatory. Every tool result should point back to a human-verifiable source.

• gpaMinimum: null means the provider didn't publish a cutoff. That's different from 0.

Saved Scholarship

A research list is a join between a person and a catalog row.

const savedScholarshipSchema = new mongoose.Schema(
  {
    researcherId: { type: String, required: true, default: "default" },
    scholarship: {
      type: mongoose.Schema.Types.ObjectId,
      ref: "Scholarship",
      required: true,
    },
    status: {
      type: String,
      enum: ["saved", "applying", "submitted", "won", "rejected"],
      default: "saved",
    },
  },
  { timestamps: true },
);

savedScholarshipSchema.index({ researcherId: 1, scholarship: 1 }, { unique: true });

The unique index makes save_scholarship idempotent. Saving the same award twice updates the status instead of creating duplicates.

researcherId is a plain string. For a tutorial, that's enough. In production you would take it from an auth token.

Research Note

Notes are a separate collection so one scholarship can have many of them.

const researchNoteSchema = new mongoose.Schema(
  {
    researcherId: { type: String, required: true, default: "default" },
    scholarship: {
      type: mongoose.Schema.Types.ObjectId,
      ref: "Scholarship",
      required: true,
    },
    body: { type: String, required: true, trim: true },
  },
  { timestamps: true },
);

You now have a catalog, a shortlist, and a notebook. That's the whole product surface the MCP tools will wrap.

How to Write the Scholarship Service

Keep MongoDB queries out of the MCP layer. Tools should call a service, get plain objects back, and format text. That makes the same functions reusable from a seed script, a smoke test, or a future REST route.

Search

Search is a filter builder. Each optional argument adds a clause. Open or rolling awards stay in the result set. Closed deadlines drop out.

const OPEN_DEADLINE_FILTER = {
  $or: [{ rolling: true }, { deadline: null }, { deadline: { $gte: new Date() } }],
};

export async function searchScholarships(filters) {
  const query = { ...OPEN_DEADLINE_FILTER };
  const and = [query];

  if (filters.keyword) {
    and.push({
      $or: [
        { title: { $regex: escapeRegex(filters.keyword), $options: "i" } },
        { provider: { $regex: escapeRegex(filters.keyword), $options: "i" } },
        { description: { $regex: escapeRegex(filters.keyword), $options: "i" } },
        { fieldsOfStudy: { $regex: escapeRegex(filters.keyword), $options: "i" } },
      ],
    });
  }

  if (filters.fieldOfStudy) {
    and.push({
      $or: [
        { fieldsOfStudy: { $regex: `^any$`, $options: "i" } },
        { fieldsOfStudy: { $regex: escapeRegex(filters.fieldOfStudy), $options: "i" } },
      ],
    });
  }

  // educationLevel, citizenship, country, minAmount, gpa, flags...

  const results = await Scholarship.find({ $and: and })
    .sort({ deadline: 1, amountMax: -1 })
    .limit(100)
    .lean();

  return rankByFieldMatch(results, filters.fieldOfStudy).slice(0, filters.limit ?? 10);
}

Two details are easy to skip and worth keeping.

Escape user input before you drop it into $regex. A keyword of ( shouldn't become a broken regular expression.

When a student searches for computer science, field-open awards (any) are eligible, but the specific CS scholarships should appear first. Rank in memory after the query. MongoDB already did the eligibility filter. You're only adjusting display order.

Match

Matching is not the same as search. Search is "find documents that look like this." Matching is "here is a student, score every open award."

The service loads open scholarships, then applies hard filters and a score:

Hard filters prevent false hope. Soft scores rank the rest. Each result also returns a reasons array, so the model can explain the match instead of inventing one.

That last point is the whole reason to put matching on the server. If you only return raw documents, the model will sometimes "helpfully" include an award the student can't apply for. Returning score and reasons keeps the explanation grounded in code.

Save, Notes, Deadlines, Compare

The remaining functions are thin:

• saveScholarship upserts by (researcherId, scholarshipId)

• addResearchNote inserts a note after confirming the scholarship exists

• getUpcomingDeadlines queries deadline between now and now + N days

• compareScholarships loads two or three documents and returns the same fields for each

Validate MongoDB ids with mongoose.Types.ObjectId.isValid before you query. An LLM will occasionally pass a title where you asked for an id. Fail clearly. Don't throw a CastError into the MCP transport.

How to Register MCP Tools

Create src/mcp/server.js as a factory. The HTTP handler will call it on every request.

import { McpServer } from "@modelcontextprotocol/server";
import { registerPrompts } from "./prompts.js";
import { registerResources } from "./resources.js";
import { registerTools } from "./tools.js";

export function createScholarshipServer() {
  const server = new McpServer({
    name: "scholarship-research",
    version: "1.0.0",
  });

  registerTools(server);
  registerResources(server);
  registerPrompts(server);

  return server;
}

Keep this factory cheap. No database connections, no file reads, and no caches that belong at module scope. The HTTP serving guide is explicit about this: create connection pools once at startup, and close over them.

One Tool, Fully

registerTool takes a name, a config object, and a handler. The inputSchema is a Zod object. The SDK turns that schema into JSON Schema for tools/list, validates arguments before your handler runs, and infers types if you're on TypeScript.

import * as z from "zod/v4";

server.registerTool(
  "search_scholarships",
  {
    title: "Search scholarships",
    description:
      "Search the scholarship catalog by keyword, field of study, education level, citizenship, country, GPA, and award amount.",
    inputSchema: z.object({
      keyword: z.string().min(1).optional().describe("Free-text search across title, provider, description, and fields"),
      fieldOfStudy: z.string().optional().describe("For example computer science, public health, or engineering"),
      educationLevel: z.enum(["undergraduate", "graduate", "doctoral"]).optional(),
      citizenship: z.string().optional(),
      country: z.string().optional(),
      minAmount: z.number().nonnegative().optional(),
      gpa: z.number().min(0).max(4).optional(),
      firstGeneration: z.boolean().optional(),
      womenOnly: z.boolean().optional(),
      limit: z.number().int().min(1).max(25).optional(),
    }),
    annotations: { readOnlyHint: true, openWorldHint: false },
  },
  async (args) => {
    const results = await searchScholarships(args);
    return toolText(formatScholarshipList(results));
  },
);

Write descriptions as if the model is the only docs the tool will ever get. .describe() on a Zod field survives conversion to JSON Schema. That's how the host tells the model what fieldOfStudy means.

title is the human label. description is the model-facing contract. They're not the same string.

Annotations

Annotations don't change how the SDK runs the tool. Hosts use them to decide how cautious to be.

• readOnlyHint: true for search, get, match, list, compare, and deadlines

• readOnlyHint: false for save and add-note

• idempotentHint: true on save, because of the unique index

• openWorldHint: false because this server talks to your database, not the open web

A host can auto-approve a read-only search and ask the user before a write. That's worth five extra keys in the config.

Return Shape

Every tool returns MCP content blocks:

export function toolText(text, isError = false) {
  return {
    content: [{ type: "text", text }],
    isError,
  };
}

Return isError: true for domain failures such as "scholarship not found." Throw only for unexpected failures. The spec treats those differently. A validation error from Zod never reaches your handler. The SDK already converts it into an isError result.

Format lists for a person who is skimming a chat transcript. Include the MongoDB id on every row. Later tools need that id, and the model can't invent a valid ObjectId.

The Full Tool Set

The server registers nine tools:

That's enough for a research loop: find, inspect, match, save, annotate, and compare.

Resist the urge to add a delete_everything tool. Destructive tools need extra confirmation and aren't part of this workflow.

How to Expose Resources and Prompts

Tools aren't the only way a host gets context.

Resources

A static resource is a fixed URI. The catalog fits that:

server.registerResource(
  "scholarship-catalog",
  "scholarship://catalog",
  {
    title: "Scholarship catalog",
    description: "Open scholarships currently stored in MongoDB",
    mimeType: "application/json",
  },
  async (uri) => {
    const catalog = await listCatalog();
    return {
      contents: [
        {
          uri: uri.href,
          mimeType: "application/json",
          text: JSON.stringify(catalog, null, 2),
        },
      ],
    };
  },
);

A resource template covers a family of URIs. Use scholarship://item/{id} rather than scholarship://{id}. If the pattern is scholarship://{id}, the URI scholarship://catalog becomes ambiguous.

import { ResourceTemplate } from "@modelcontextprotocol/server";

server.registerResource(
  "scholarship-record",
  new ResourceTemplate("scholarship://item/{id}", {
    list: async () => {
      const catalog = await listCatalog(20);
      return {
        resources: catalog.map((scholarship) => ({
          uri: `scholarship://item/${scholarship._id}`,
          name: scholarship.title,
          mimeType: "text/plain",
        })),
      };
    },
  }),
  {
    title: "Scholarship record",
    description: "Full details for one scholarship",
    mimeType: "text/plain",
  },
  async (uri, { id }) => {
    const scholarship = await getScholarshipById(id);
    return {
      contents: [
        {
          uri: uri.href,
          mimeType: "text/plain",
          text: scholarship
            ? formatScholarship(scholarship)
            : `No scholarship found with id ${id}.`,
        },
      ],
    };
  },
);

list is required on a template. Pass undefined if you can't enumerate instances. Here you can, so the host can show a picker.

Prompts

Prompts are workflows you want a person to start. The research plan prompt doesn't query MongoDB itself. It tells the model to use the tools, then structure the answer.

server.registerPrompt(
  "research-plan",
  {
    title: "Scholarship research plan",
    description: "Build a week-by-week research and application plan from a student profile.",
    argsSchema: z.object({
      fieldOfStudy: z.string(),
      educationLevel: z.string(),
      citizenship: z.string(),
      gpa: z.string(),
      weeks: z.string().optional(),
    }),
  },
  ({ fieldOfStudy, educationLevel, citizenship, gpa, weeks }) => ({
    messages: [
      {
        role: "user",
        content: {
          type: "text",
          text: `Create a ${weeks || "6"}-week scholarship research plan for this student.

Field of study: ${fieldOfStudy}
Education level: ${educationLevel}
Citizenship: ${citizenship}
GPA: ${gpa}

Use the scholarship research tools to find real awards first. Then produce a shortlist, a week-by-week plan, and risks. Name actual scholarships and dates from the tool results.`,
        },
      },
    ],
  }),
);

Notice the instruction "use the scholarship research tools." A prompt isn't a substitute for tools. It's a script that makes tool use more likely and the output shape more consistent.

A second prompt, application-checklist, takes a scholarship id and asks for a document list and a backward calendar. That's the kind of repetitive work MCP prompts are good at.

Prompt arguments are strings in many hosts, even when the value is a number. Typing gpa as a string avoids a frustrating expected number, received string error from a slash-command form.

How to Serve MCP Over Express

This is the part that used to be a page of session-handling code. In SDK v2 it's a factory plus one route.

import "dotenv/config";
import { createMcpExpressApp } from "@modelcontextprotocol/express";
import { toNodeHandler } from "@modelcontextprotocol/node";
import { createMcpHandler } from "@modelcontextprotocol/server";
import { config } from "./config.js";
import { connectDatabase } from "./db.js";
import { createScholarshipServer } from "./mcp/server.js";

const mcpHandler = createMcpHandler(() => createScholarshipServer());
const nodeHandler = toNodeHandler(mcpHandler);

const app = createMcpExpressApp({
  host: config.host,
  allowedHosts: ["127.0.0.1", "localhost"],
});

app.get("/health", (_req, res) => {
  res.json({
    status: "ok",
    service: "scholarship-research-mcp",
    transport: "streamable-http",
  });
});

app.all("/mcp", (req, res) => {
  void nodeHandler(req, res, req.body);
});

async function start() {
  await connectDatabase();
  app.listen(config.port, config.host, () => {
    console.log(`Scholarship research MCP server listening on http://${config.host}:${config.port}/mcp`);
  });
}

start();

Walk through what each helper is doing.

createMcpHandler takes a function that returns a fresh McpServer. It exposes a web-standard fetch. That's the same handler you would export from a Cloudflare Worker.

toNodeHandler adapts that fetch to Express (req, res). You pass req.body as the third argument because createMcpExpressApp already ran express.json(). If you omit the body, the adapter tries to read a stream Express already consumed.

createMcpExpressApp is express() with two extras: JSON parsing, and Host/Origin checks. Those checks exist because of DNS rebinding. A malicious page can point its own domain at 127.0.0.1 and, without a Host check, your browser would treat the local MCP server as same-origin. The default bind is 127.0.0.1 for that reason. The Express serving guide covers this in more detail.

app.all("/mcp", ...) is intentional. Streamable HTTP uses POST for JSON-RPC, and GET for SSE streams. Registering only POST will break some clients.

Shut the handler down on SIGINT:

process.on("SIGINT", async () => {
  await mcpHandler.close();
  process.exit(0);
});

close() waits for in-flight requests. Then you can exit.

Add npm scripts:

{
  "scripts": {
    "start": "node src/index.js",
    "dev": "node --watch src/index.js",
    "seed": "node scripts/seed.js",
    "smoke": "node scripts/smoke-test.js"
  }
}

node --watch is enough for local development. You don't need nodemon for this project.

How to Seed the Catalog

MCP tools against an empty database will work and return "no scholarships matched." That's correct, and also a bad first impression.

Put 20 to 30 realistic records in data/scholarships.json. Mix:

• Undergraduate and graduate awards

• STEM and field-open awards

• Country-specific programs and global ones

• Rolling deadlines and hard dates

• First-generation and women-only flags

A seed script should replace the catalog, not append:

import "dotenv/config";
import { readFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { connectDatabase } from "../src/db.js";
import { Scholarship } from "../src/models/index.js";

const __dirname = path.dirname(fileURLToPath(import.meta.url));

async function seed() {
  await connectDatabase();
  const raw = await readFile(path.join(__dirname, "..", "data", "scholarships.json"), "utf8");
  await Scholarship.deleteMany({});
  const inserted = await Scholarship.insertMany(JSON.parse(raw));
  console.log(`Seeded ${inserted.length} scholarships.`);
  process.exit(0);
}

seed();

Run it:

npm run seed

Treat the JSON as sample data. Names of well-known programs help the tutorial feel real. They also create a duty to say, clearly, that students must verify every number and date on the official site. The applicationUrl field exists so that reminder has somewhere to point.

If you later replace the JSON with a live source, keep the same schema. The MCP tools shouldn't care where the documents came from.

How to Test the Server

Start MongoDB, seed, then start the process:

npm run seed
npm start

You should see:

Scholarship research MCP server listening on http://127.0.0.1:3000/mcp

Health Check

curl -s http://127.0.0.1:3000/health

A JSON status: ok means Express is up. It doesn't mean MCP is wired correctly. For that, send a JSON-RPC request.

List Tools

curl -s -X POST http://127.0.0.1:3000/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

The response is an SSE event whose data: line contains the JSON-RPC result. You should see all nine tools, each with a JSON Schema derived from Zod.

The Accept header matters. MCP Streamable HTTP can return JSON or an event stream. Asking for both is the compatible choice.

Call a Tool

curl -s -X POST http://127.0.0.1:3000/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{
    "jsonrpc":"2.0",
    "id":2,
    "method":"tools/call",
    "params": {
      "name": "search_scholarships",
      "arguments": {
        "fieldOfStudy": "computer science",
        "limit": 3
      }
    }
  }'

You should get a numbered list with ids, deadlines, and GPA minimums. Copy one id and pass it to get_scholarship.

A small Node smoke test is nicer than raw curl once you're calling several methods. Parse the data: line, then print result.content[0].text. The repo includes scripts/smoke-test.js for that.

Inspector

The MCP Inspector is the official GUI for servers. Point it at http://127.0.0.1:3000/mcp and you can list tools, fill in arguments, and read resources without a host app in the way.

Use Inspector when a host "can't see" your server. If Inspector works and the host doesn't, the bug is in the host config. If Inspector fails, the bug is in your process.

How to Connect Cursor and Claude Desktop

Keep npm start running. MCP over HTTP is a live server, not a one-shot CLI.

Cursor

Add a server entry in Cursor's MCP settings. A Streamable HTTP server looks like this:

{
  "mcpServers": {
    "scholarship-research": {
      "url": "http://127.0.0.1:3000/mcp"
    }
  }
}

Restart the MCP session if Cursor had a previous failed connection cached. Then ask:

I am a first-generation undergraduate studying computer science in the United States, GPA 3.6. Use the scholarship research tools to build a shortlist and a six-week plan.

You should see the host call match_scholarships or search_scholarships, then get_scholarship for the interesting rows, then maybe save_scholarship. If it never calls a tool, the server isn't actually connected. Check the MCP logs in Cursor before you change code.

You can also invoke the research-plan prompt from the host's prompt menu if it surfaces prompts.

Claude Desktop

Claude Desktop's config file lives at:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

• Windows: %APPDATA%\Claude\claude_desktop_config.json

Claude Desktop prefers stdio. Bridge your HTTP server with mcp-remote:

{
  "mcpServers": {
    "scholarship-research": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "http://127.0.0.1:3000/mcp"]
    }
  }
}

Restart Claude Desktop after you save the file. The scholarship tools should appear in the tool list.

Don't put MongoDB credentials in the Claude config. The Node process already loaded .env. The host only needs the URL.

How a Research Session Runs

Here's a realistic session against the seed catalog. The student is a first-generation undergraduate in computer science, a US citizen, GPA 3.6.

The host calls match_scholarships with that profile. The server returns ranked rows. A women-in-technology award and a first-generation program both score well, for different reasons. A graduate-only award never appears.

The host then calls get_scholarship on the top two ids. Each result includes the official URL, the requirement list, and the deadline. That's the moment to tell the student to open the URL. The model shouldn't be the last word on eligibility.

If an award is worth pursuing, the host calls save_scholarship with a researcherId such as ada and status saved. Later it can set applying. list_saved_scholarships is how a new chat picks up the shortlist. Persistence is the whole point of MongoDB here. Without it, every conversation starts from zero.

add_research_note is for the messy human details: "Ask Dr. Chen for a recommendation by October 1." get_upcoming_deadlines is the weekly sweep. compare_scholarships is for the moment the student has two finalists and needs amount, GPA, and requirements in one view.

The research-plan prompt packages that loop. It injects the profile into a user message that tells the model to call tools first and then produce a week-by-week plan. If you invoke the prompt without a connected server, you get a generic essay. If you invoke it with this server running, you get named awards and real dates.

That's the product: a catalog the model can query, a shortlist it can't forget, and prompts that make the workflow repeatable.

How the Matching Logic Works

It's worth slowing down on matching, because this is the part people are tempted to hand to the model.

Suppose the student is:

• GPA 3.6

• Computer science

• Undergraduate

• United States citizen

• First-generation

• A woman

The matcher walks every open award.

A women-in-technology scholarship with a 3.3 GPA minimum, CS as a listed field, and US eligibility scores high: education, GPA, citizenship, women-only flag, and field all hit. A first-generation program that is field-open also scores high, because any counts as a field match. A graduate-only award is skipped, even if the title looks relevant. A 3.8 GPA cutoff is skipped, even if everything else fits.

The tool then returns ranked rows with reasons:

1. Palantir Women in Technology Scholarship — score 90
   Why: education level matches; GPA 3.6 meets the 3.3 minimum; citizenship is eligible; women-only award matches; field of study matches

The model can still write a warm paragraph around that. It shouldn't be the component that decided eligibility.

If you extend this later, keep the same split. New eligibility rules belong in the service. New prose belongs in the prompt.

What You Can Build Next

The server you have is complete enough to use. It's also a base for a more serious research tool.

First, you could replace the seed file with a live source. Official feeds such as Grants.gov and college-maintained lists are safer than scraping commercial aggregators. Keep your schema. Write an importer that upserts by a stable external id.

You could also add authentication. createMcpExpressApp works with requireBearerAuth. Map researcherId from the verified token instead of a tool argument. The Express adapter documents that middleware.

Try adding full-text search. The schema already has a text index. For a large catalog, Atlas Search or a dedicated search engine will beat a pile of regex filters.

You can track documents, not just notes. A documents collection for transcripts, recommendation status, and essay drafts turns the shortlist into an application tracker.

You could also write tests around the matcher. Eligibility code is where silent bugs hurt people. A table of profiles and expected include/exclude lists will pay for itself.

And you could deploy it. Bind to 127.0.0.1 on your laptop. If you put this on a network, set allowedHosts, terminate TLS, and require a bearer token. An open MCP server is an open database with a helpful English interface.

Conclusion

In this guide, you built a scholarship research MCP server that's more than a toy tool list.

You stored awards, shortlists, and notes in MongoDB. You exposed search, matching, and research-tracking as MCP tools, with Zod schemas the host can advertise to a model. You added resources for the catalog and prompts for repeatable workflows. You served the whole thing over Streamable HTTP with Express, including the Host header checks the SDK enables for localhost.

The pattern transfers. Any research workflow with a catalog and a personal working set can use the same three layers: a service that owns the rules, an McpServer factory that registers primitives, and a small Express app that speaks the protocol.

If you take one idea from this tutorial, take this one: let the model write the plan, and let your server decide what's true.

The sample catalog is for learning. Confirm every scholarship on its official application page before you apply, and before you tell someone else to apply.

Then production traffic arrives.

The email API takes 2 seconds instead of 200 milliseconds. A few requests time out. During a provider outage, every signup starts returning 500. Support asks why users can create accounts but never get the confirmation link.

The bug isn't your email template. The bug is where you decided to send the email.

In this article, you'll learn why sending email inside the HTTP request path is a production trap, what goes wrong as you scale, and how to move email (and other side effects) into a background job queue so your API stays fast and reliable.

Prerequisites

You'll get more out of this article if you know:

• How REST APIs work (request, response, status codes)

• Basic Node.js or any backend language that can call an email API

• What a background job or queue is at a high level

You don't need prior experience with BullMQ, Redis, or a specific email provider.

Table of Contents

• Prerequisites

• 1. What "Inside the Request" Really Means

• 2. The Tempting Code That Causes the Problem

• 3. Why This Breaks in Production

• 4. The Rule: Your API Should Enqueue Work, Not Do It

• 5. How to Move Email to a Background Queue

• 6. How to Handle Retries, Failures, and Idempotency

• 7. What Else Should Leave the Request Path

• 8. A Practical Checklist

• Conclusion

1. What "Inside the Request" Really Means

When a client hits your API, the request path is everything that must finish before you send the HTTP response.

That usually includes:

• validating input

• checking auth

• writing to your database

• returning JSON

It should almost never include:

• calling an SMTP server

• waiting on Resend, SendGrid, SES, Mailgun, or Postmark

• rendering a big HTML template and uploading attachments

• retrying a flaky third-party network call

Email providers are external systems. They have their own latency, rate limits, and outages. If your API waits on them, their problems become your users' problems.

Bad path (synchronous):

Client ──▶ API ──▶ Database ──▶ Email provider ──▶ Client
                         ▲
                         └── user waits for all of this

2. The Tempting Code That Causes the Problem

Here's the pattern most teams start with. It's clear, short, and works on your laptop.

app.post('/signup', async (c) => {
  const body = await c.req.json();

  const user = await db.users.create({
    email: body.email,
    passwordHash: await hash(body.password),
  });

  // This is the trap.
  await emailClient.send({
    to: user.email,
    subject: 'Confirm your account',
    html: renderConfirmEmail(user),
  });

  return c.json({ id: user.id }, 201);
});

The request can't finish until the email provider answers. If that call is slow, your response is slow. If that call throws, your signup either fails or you catch the error and silently skip the email.

Neither option is good.

3. Why This Breaks in Production

Your Response Time Becomes Someone Else's SLA

A healthy signup endpoint should often respond in tens or hundreds of milliseconds. Email APIs regularly take longer than that. Under load, they take much longer.

Your p95 latency is no longer owned by your code. It's owned by the slowest third-party call on the path.

Failures Create Ugly Tradeoffs

If send() throws, you have two bad choices:

1. Fail the whole request: the user sees an error even though the account may already exist

2. Swallow the error: the user is created, but never gets the email

That second case is especially painful with confirmation or password-reset flows. The database says success but the inbox says nothing.

Timeouts Cascade

API gateways, load balancers, and browsers all have timeouts. A slow email call can turn a successful business action into a client-side failure. The user retries. Now you may create duplicate users or send duplicate emails.

Rate Limits Punish Traffic Spikes

Email providers rate-limit you. Product launches, digests, and password-reset spikes can all hit those limits. If sending happens inside the request, a rate limit error becomes a user-facing API error.

You Can't Retry Cleanly

Background jobs can retry with backoff. HTTP requests can't. Once you've returned a response, the request is over. If the email failed and you already told the client "success," you need another system to repair that. That system is a queue.

What the user experiences during a provider slowdown:

Signup click
   │
   ▼
API waits on email provider .............. 2s ... 5s ... timeout
   │
   ▼
"Something went wrong"
   │
   ▼
User clicks again → duplicate attempts, confused support tickets

4. The Rule: Your API Should Enqueue Work, Not Do It

Treat email as asynchronous work that must happen because of the request, not inside the request.

The request path should:

1. validate the input

2. commit the business change

3. enqueue a job that describes the email

4. return quickly

A worker somewhere else should:

1. pick up the job

2. render the template

3. call the email provider

4. retry on failure

5. record the result

Good path (asynchronous):

Client ──▶ API ──▶ Database ──▶ Queue ──▶ Client (fast response)
                                   │
                                   ▼
                                Worker ──▶ Email provider

This is the same idea behind processing uploads, generating PDFs, calling AI APIs, or syncing data to a CRM. Email is just the most common example.

5. How to Move Email to a Background Queue

Here's a minimal Node.js shape using BullMQ and Redis. The exact library matters less than the boundary.

Define the Job

// jobs/email.ts
export type SendEmailJob = {
  to: string;
  template: 'confirm-account' | 'password-reset' | 'welcome';
  variables: Record<string, string>;
  idempotencyKey: string;
};

Enqueue from the API

import { emailQueue } from '../queues/email-queue';

app.post('/signup', async (c) => {
  const body = await c.req.json();

  const user = await db.users.create({
    email: body.email,
    passwordHash: await hash(body.password),
  });

  await emailQueue.add(
    'send-email',
    {
      to: user.email,
      template: 'confirm-account',
      variables: {
        name: user.name,
        confirmUrl: `https://example.com/confirm/${user.confirmToken}`,
      },
      idempotencyKey: `confirm-account:${user.id}`,
    },
    {
      jobId: `confirm-account:${user.id}`,
      attempts: 5,
      backoff: { type: 'exponential', delay: 2000 },
      removeOnComplete: 1000,
      removeOnFail: 5000,
    },
  );

  return c.json({ id: user.id }, 201);
});

Notice what changed. The API still creates the user. It no longer waits for SMTP. It waits for Redis to accept a job, which is usually far cheaper and more predictable than waiting for an email provider.

Process the Job in a Worker

// worker/email-worker.ts
import { Worker } from 'bullmq';
import { emailClient } from '../lib/email-client';
import { renderTemplate } from '../lib/templates';

new Worker(
  'email',
  async (job) => {
    const { to, template, variables } = job.data;

    await emailClient.send({
      to,
      subject: subjectFor(template),
      html: renderTemplate(template, variables),
    });
  },
  {
    connection: redisConnection,
    concurrency: 5,
  },
);

Run the worker as a separate process from the API. That way an email outage slows the worker, not every HTTP response.

Return the Right Status

For signup, 201 Created is still correct if the account exists and the confirmation email is queued.

For endpoints that only trigger work, prefer 202 Accepted and return a job or delivery id when the client needs to track progress.

6. How to Handle Retries, Failures, and Idempotency

Moving email to a queue isn't enough. You also need failure rules.

Retry Transient Errors

Network blips, 429 Too Many Requests, and 503 Service Unavailable should retry with backoff. Permanent errors like an invalid recipient should fail fast and land in a dead-letter queue or failed-jobs table.

Make Jobs Idempotent

Workers can run more than once. Use a stable jobId or idempotencyKey so a retry doesn't send five welcome emails for one signup.

A practical approach:

• derive the key from the business event: confirm-account:user_123

• store provider message ids when a send succeeds

• skip work if that key was already delivered

Don't Lose the Business Event

If the process crashes between db.users.create() and emailQueue.add(), you can lose the email.

There are two common ways to reduce that risk:

1. Outbox pattern: write the email intent into an outbox table in the same database transaction as the user row. A separate publisher reads the outbox and enqueues jobs.

2. Transactional enqueue where supported: keep the write and the enqueue as close together as possible, and run reconciliation jobs for missed sends.

For many apps, a simple queue plus monitoring is enough at first. Add an outbox when missed emails become a real reliability requirement.

Separate Urgent Mail from Bulk Mail

Password reset and OTP emails shouldn't sit behind a newsletter blast.

Use priorities or separate queues:

• high priority: OTP, password reset, confirmation

• normal priority: receipts, onboarding

• low priority: digests, marketing

That keeps user-blocking email fast even when bulk traffic spikes.

7. What Else Should Leave the Request Path

If a task calls a third-party network API, can take more than a few hundred milliseconds, or it needs retries, keep it out of the request path. Same if it can fail without undoing the main business action or it does CPU-heavy work like image or PDF generation.

Here are some common examples of these types of tasks:

• emails and SMS

• webhook fan-out

• search indexing

• thumbnail generation

• invoice PDF creation

• CRM sync

• AI transcription or summarization

• slow report generation

Your API should confirm the intent and persist the state. Workers should do the slow parts.

8. A Practical Checklist

Before you ship a feature that sends email, ask:

1. Does the user need the email to finish before seeing success?

2. Can the email provider being down break this endpoint?

3. If sending fails after the database write, how do we recover?

4. If the job runs twice, will the user get duplicate messages?

5. Do OTP and password-reset emails have a faster lane than bulk email?

6. Can we see failed jobs in logs, metrics, or a dashboard?

7. Do we alert when the queue depth or failure rate spikes?

If you can't answer those questions, the design isn't ready for production.

Conclusion

Sending email inside your API request couples your response time and your error rate to a third-party system you don't control. It feels simple in a demo and becomes expensive in production.

The better default is simple:

• commit the business change

• enqueue the email

• return quickly

• let a worker send, retry, and report failures

That pattern makes your API faster, your email delivery more reliable, and your failure modes much easier to reason about.

Once you make that split for email, you'll start seeing the same boundary everywhere: if the user shouldn't wait for it, it doesn't belong in the request.

I ran into this while building Temploracraft, a résumé tool that takes an uploaded document and turns it into structured data the application can edit.

The input is genuinely unpredictable: two-column PDFs, tables that aren't really tables, and dates written in about fourteen different formats. The output has to be strict, because every extracted field lands in a form that a person is going to look at. When the model gets a date wrong, the user notices in about two seconds.

This article is about the layer that sits between "the model returned some text" and "my application has data it can trust." It covers the three mechanisms for constraining output, why designing the schema first beats writing a longer prompt, how to build a retry loop that doesn't set money on fire, and what to do about the failures that no retry will ever fix.

Table of Contents

• Prerequisites

• Why Prompting for JSON Isn't Enough

• The Three Ways to Constrain Output

• Start with the Schema, Not the Prompt

• Validation Is Two Jobs, Not One

• Building a Retry Loop That Doesn't Burn Tokens

• Streaming Structured Output

• The Failures You Can't Retry Away

• What This Actually Costs

• When You Don't Need Any of This

• Wrapping Up

• References

Prerequisites

To follow along comfortably, you'll want a few things in place:

• Working knowledge of TypeScript: The examples use type inference and generics lightly, and you should be able to read a type annotation without pausing.

• You have called a language model API at least once: You don't need to be an expert, but you should know what a system prompt is and roughly what a token is.

• Familiarity with JSON Schema is helpful but not required: I explain the parts that matter as they come up.

• Node.js 18 or later if you want to run the examples, since they use the native fetch and async iterators.

The code samples use Zod for schema definition and the Anthropic SDK for model calls, but every technique here translates directly to other validation libraries and other providers. The ideas matter more than the specific packages.

Why Prompting for JSON Isn't Enough

The first instinct when you need structured data is to ask for it politely. You write something like "Respond with valid JSON matching this shape, and do not include any other text," you paste an example, and it works. It keeps working through development. It works in your demo.

The problem is that a language model generates one token at a time based on probability, and your instruction is only one influence among many. It competes with the model's training, the shape of the input document, and whatever the model produced in the preceding few hundred tokens. Most of the time your instruction wins. Occasionally it does not.

Here are failures I've actually collected from a production extraction pipeline, all from a model that was explicitly told to return strict JSON:

• The model wrapped the response in a Markdown code fence, despite being told twice not to.

• It returned "2019 - Present" as a single string where the schema defined separate startDate and endDate fields.

• It invented an endDate of "2024-12-31" for a role the document clearly marked as current.

• It returned the string "null" instead of an actual null.

• It emitted seven bullet points for a role where the schema set a maximum of five.

• It truncated mid-object because the response hit the output token limit.

Notice that these aren't all the same kind of failure. The code fence and the truncation are syntax problems, and you can often fix them locally without calling the model again. The merged date string and the extra bullets are schema problems, where the JSON parses fine but doesn't match the shape you need. The invented end date is a semantic problem, where the output is both valid JSON and schema-conformant but factually wrong about the source document.

Those three categories need three different responses, and conflating them is the most common architectural mistake I see in extraction code. The rest of this article is largely about separating them.

The Three Ways to Constrain Output

Before writing any validation code, it's worth understanding what the API itself can enforce for you. There are three mechanisms, and they give you meaningfully different guarantees.

JSON mode is the weakest of the three. You set a flag such as response_format: { type: "json_object" }, and the provider guarantees the response will be syntactically valid JSON. That's genuinely useful, because it eliminates the code fence and truncation problems in one move. What it doesn't do is guarantee anything about the shape. You can get valid JSON with the wrong keys, the wrong types, or a completely different structure than you asked for.

Tool calling is the mechanism I reach for most often. You describe a function with a JSON Schema for its parameters, then force the model to call it. The model returns arguments matching that schema rather than free text. Support is broad, the schema travels with the request so you're not duplicating it in the prompt, and providers tend to constrain the output more aggressively than they do in plain JSON mode.

Here's what that looks like with the Anthropic SDK:

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  tools: [
    {
      name: "emit_resume",
      description: "Return the parsed resume as structured data.",
      input_schema: jsonSchema,
    },
  ],
  tool_choice: { type: "tool", name: "emit_resume" },
  messages: [{ role: "user", content: resumeText }],
});

const block = response.content.find((b) => b.type === "tool_use");
const candidate = block?.input;

The tool_choice field is the important part. Without it the model decides whether to call the tool, and sometimes it will just answer in prose instead. Forcing the specific tool removes that branch entirely.

Constrained decoding is the strongest option and the least widely available. Instead of asking the model to follow a schema, the runtime masks the token sampler at every step so that tokens which would produce invalid output can't be selected at all. Invalid output becomes impossible rather than unlikely.

OpenAI exposes a version of this through strict structured outputs, and if you run models locally you can use GBNF grammars in llama.cpp or a library such as Outlines.

The tradeoff is that constrained decoding can push the model into awkward corners. If the schema demands a field the document genuinely doesn't contain, the model can't decline, so it fills the slot with something. You've traded a parsing failure for a hallucination, which is harder to detect. I make required fields nullable for exactly this reason, which I'll come back to shortly.

My default is tool calling with a nullable-heavy schema, plus validation on top. That combination gives most of the benefit without the issues.

Start with the Schema, Not the Prompt

The instinct when output quality is poor is to write a longer prompt. More examples, more emphasis, and more capital letters. This helps a little and scales badly, because the prompt is prose and prose isn't enforceable.

A better approach is to treat the schema as the primary artifact and let everything else derive from it. Define it once, and use that single definition to generate the TypeScript type, the JSON Schema sent to the API, and the runtime validator. When the shape changes, all three change together, and there's no way for them to drift apart.

With Zod that looks like this:

import { z } from "zod";

const YearMonth = z
  .string()
  .regex(/^\d{4}-\d{2}$/, "Expected a YYYY-MM date");

const Role = z.object({
  company: z.string().min(1),
  title: z.string().min(1),
  startDate: YearMonth,
  endDate: YearMonth.nullable(),
  bullets: z.array(z.string().min(1)).min(1).max(5),
});

const Resume = z.object({
  name: z.string().min(1),
  email: z.string().email().nullable(),
  roles: z.array(Role),
});

export type Resume = z.infer<typeof Resume>;

Three details in there are doing real work.

The YearMonth regex is narrower than z.string(), and that narrowness is the point. A bare string field invites the model to return "January 2019" or "2019 - Present" or "01/2019", and all of those pass validation. Constraining the format at the schema level means the mismatch surfaces immediately instead of three layers deeper in your date-handling code.

The endDate field is nullable rather than optional. This is the single change that most improved extraction quality for me. An optional field lets the model quietly omit it, and you can't distinguish "the document did not say" from "the model forgot." A nullable field forces an explicit decision, and null is a meaningful answer that means the role is current.

The max(5) on bullets encodes a product constraint directly into the contract rather than trimming the array afterward. If the model exceeds it, you want to know, because it usually means the model is padding rather than extracting.

To send the schema to the API, derive the JSON Schema from the same definition:

import { zodToJsonSchema } from "zod-to-json-schema";

const jsonSchema = zodToJsonSchema(Resume, { target: "openApi3" });

One habit worth adopting: write a .describe() on any field where the name alone is ambiguous. Those descriptions end up in the JSON Schema, which means they reach the model as part of the tool definition, which means they function as targeted, structured prompt instructions attached to exactly the field they concern.

const Role = z.object({
  company: z.string().min(1),
  title: z.string().min(1).describe("The person's job title, not the team name"),
  startDate: YearMonth,
  endDate: YearMonth.nullable().describe("null if this role is current"),
  bullets: z
    .array(z.string().min(1))
    .min(1)
    .max(5)
    .describe("Verbatim from the document. Do not rewrite or summarise."),
});

That last description eliminated an entire class of failure for me, where the model would helpfully improve the wording of a person's own bullet points.

Validation Is Two Jobs, Not One

Once the response comes back, it's tempting to run schema.parse() and consider the job done. Zod will tell you whether the shape is right, and if it is, you move on.

But shape validation and semantic validation are different jobs, and only the first one is free. Zod can tell you that endDate is a string matching YYYY-MM. It can't tell you that the end date falls before the start date, or that the date is in the future, or that a role listed as current also has an end date. Those are all schema-valid and all wrong.

So run two passes. The first is structural and comes from the schema. The second is a plain function that encodes what you know about the domain:

function findSemanticProblems(resume: Resume): string[] {
  const problems: string[] = [];
  const currentMonth = new Date().toISOString().slice(0, 7);

  for (const role of resume.roles) {
    if (role.startDate > currentMonth) {
      problems.push(`${role.company}: start date is in the future`);
    }
    if (role.endDate && role.endDate < role.startDate) {
      problems.push(`${role.company}: end date precedes start date`);
    }
  }

  const currentRoles = resume.roles.filter((r) => r.endDate === null);
  if (currentRoles.length > 1) {
    problems.push("More than one role is marked as current");
  }

  return problems;
}

None of this is clever code, and that's rather the point. It's ordinary business logic that happens to be checking a model's work rather than a user's. Write it as you discover failures, and treat each new check as a permanent regression test for a mistake the model made once.

There's a useful asymmetry here. Semantic problems are often better surfaced to the user than retried, because the model frequently can't do better with the information available. If a document genuinely lists two current roles, that's what the document says, and asking the model again won't change it.

Building a Retry Loop That Doesn't Burn Tokens

When validation fails, the naïve fix is to call the model again with the same prompt and hope for a different sample. That works often enough to feel reasonable and is wasteful enough to notice on the bill, because you're paying full input cost for a request that has already mostly succeeded.

Two changes make a large difference.

The first is to attempt a local repair before spending anything. A meaningful share of failures are cosmetic, and you can fix them with string handling:

function salvage(raw: string): string {
  let text = raw.trim();

  // Strip a Markdown fence the model added despite instructions.
  const fenced = text.match(/^```(?:json)?\s*([\s\S]*?)\s*```$/);
  if (fenced) text = fenced[1];

  // Drop any prose before the first brace or after the last one.
  const start = text.indexOf("{");
  const end = text.lastIndexOf("}");
  if (start !== -1 && end > start) text = text.slice(start, end + 1);

  return text;
}

The second is that when you do call the model again, you should send a repair request rather than a fresh attempt. Include the original prompt, the output that failed, and the specific validation errors. The model has already done the hard extraction work, and you're asking it to fix a small number of named problems rather than start over. Repairs converge faster and produce shorter outputs, which means they cost less.

async function requestRepair(
  originalPrompt: string,
  badOutput: string,
  error: z.ZodError,
): Promise<string> {
  const issues = error.issues
    .map((issue) => `${issue.path.join(".") || "root"}: ${issue.message}`)
    .join("\n");

  const response = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 4096,
    messages: [
      { role: "user", content: originalPrompt },
      { role: "assistant", content: badOutput },
      {
        role: "user",
        content:
          `That output failed validation with these problems:\n${issues}\n\n` +
          `Return the corrected JSON only. Keep everything that was already correct.`,
      },
    ],
  });

  return response.content[0].type === "text" ? response.content[0].text : "";
}

Putting it together, the full loop separates the failure classes and caps the spend:

type Outcome<T> =
  | { ok: true; value: T; attempts: number }
  | { ok: false; error: string; attempts: number };

async function extract<T>(
  schema: z.ZodType<T>,
  prompt: string,
  maxAttempts = 3,
): Promise<Outcome<T>> {
  let lastRaw = "";
  let lastError: z.ZodError | null = null;

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    lastRaw =
      attempt === 1
        ? await callModel(prompt)
        : await requestRepair(prompt, lastRaw, lastError!);

    let candidate: unknown;
    try {
      candidate = JSON.parse(salvage(lastRaw));
    } catch {
      continue; // Syntax failure. Try again without a schema error to report.
    }

    const result = schema.safeParse(candidate);
    if (result.success) {
      return { ok: true, value: result.data, attempts: attempt };
    }
    lastError = result.error;
  }

  return {
    ok: false,
    error: lastError?.message ?? "Output was never parseable",
    attempts: maxAttempts,
  };
}

Two things about this loop are deliberate. It returns the attempt count, which you should log, because attempts per success is the single most useful health metric for an extraction pipeline. And it caps attempts at three. If three tries haven't produced valid output, a fourth rarely helps, and by then you're better off degrading gracefully than continuing to pay.

One more distinction worth respecting: a rate limit error and a schema validation error aren't the same failure and shouldn't share a retry policy. Rate limits want exponential backoff, because the problem is timing. Schema failures want an immediate repair request, because the problem is content, and waiting changes nothing.

Streaming Structured Output

Streaming and structured output pull against each other. Streaming exists so the user sees progress before the response finishes, but you can't parse a JSON object until the closing brace arrives. If your extraction takes twelve seconds, the best options are to show a spinner for twelve seconds or find a way to stream something meaningful.

There are two reasonable approaches.

The first is a partial JSON parser, which takes an incomplete string and returns the largest valid structure it can infer, closing open braces and dropping the trailing incomplete value. Libraries such as best-effort-json-parser do this. It works, and it's the right choice when the output is genuinely one large object. The cost is that intermediate states can be misleading, because a field can appear with a truncated value that looks complete.

The second approach, which I prefer when the output is a list, is to change the output format so that streaming is natural. Instead of asking for one array of objects, ask for one object per line. Each line is independently parseable, so you can validate and emit items the moment they complete:

const stream = client.messages.stream({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: prompt }],
});

let buffer = "";

for await (const event of stream) {
  if (event.type !== "content_block_delta") continue;
  buffer += event.delta.text ?? "";

  const lines = buffer.split("\n");
  buffer = lines.pop() ?? ""; // Keep the incomplete tail for the next chunk.

  for (const line of lines) {
    if (!line.trim()) continue;

    try {
      const parsed = Role.safeParse(JSON.parse(line));
      if (parsed.success) onRole(parsed.data);
    } catch {
      // A malformed line is dropped rather than failing the whole stream.
    }
  }
}

The buffer handling is the part that people get wrong. Network chunks don't align with line boundaries, so a chunk will frequently end mid-object. Popping the final element back into the buffer and carrying it forward is what makes the loop correct.

This pattern gives you per-item validation for free, which is a real advantage over parsing one large object. One bad item doesn't invalidate the rest of the extraction.

The Failures You Can't Retry Away

Everything so far assumes the model can produce the right answer if you ask correctly. Some failures don't work that way, and treating them as retry candidates wastes money while producing confidently wrong data.

The most important of these is extraction from something the source doesn't contain. If a résumé genuinely has no email address and your schema requires one, the model will produce something plausible.

Retrying produces a different plausible thing. This is the failure mode that constrained decoding makes worse rather than better, because the grammar guarantees a well-formed value in a slot that should've been empty. The fix is at the schema level, which is why nearly every field in my extraction schemas is nullable.

Ambiguity in the source is a similar case. When a document lists a date range next to two different job titles, there's no correct extraction, only a guess. Retrying gives you a different guess with the same confidence. These situations want a confidence signal in the schema and a review step in the interface, not another API call.

Then there's silent truncation. If the response hits the output token limit mid-object, you get syntactically broken JSON, and a naïve retry loop treats it as a transient parse failure and tries again with the same limit, failing identically each time. Check the stop reason on the response. If the model stopped because it ran out of tokens, retrying without raising the limit or splitting the input is guaranteed to fail again.

The general principle is that your retry policy should ask what kind of failure this is before deciding whether repeating the call could possibly help.

What This Actually Costs

Every layer above has a price, and it's worth measuring rather than assuming.

The main figure to track is attempts per successful extraction. Log it on every request and watch the p95 rather than the mean, because the mean hides the tail where the money goes. When a schema change causes a quality regression, this number moves before anything else does.

Prompt caching matters more here than in most workloads. Extraction prompts are unusually cache-friendly because the system prompt, the schema, and any few-shot examples are byte-identical on every request, and only the document changes.

On a pipeline where the schema and instructions ran to a couple of thousand tokens, moving that block into a cached prefix cut input cost substantially, and the saving compounds with every retry because repairs re-send the same prefix.

Repair requests are cheaper than fresh attempts for a reason worth understanding. The input grows, since you're now sending the original prompt plus the failed output plus the error list. But the output shrinks considerably, because the model is correcting a handful of fields rather than generating the full document again. Output tokens are the expensive direction on most providers, so the trade is usually favourable.

Two other things are worth instrumenting from the start: the distribution of validation error paths, which tells you exactly which schema fields are causing trouble and is far more actionable than an overall failure rate, and your local-repair hit rate, since if salvage() is fixing a large fraction of responses you have a prompt problem you can solve once rather than paying for repeatedly.

When You Don't Need Any of This

This machinery earns its place in a specific set of circumstances, and it's genuinely overkill outside them.

If the model's output goes straight to a human who will read it as prose, you don't need structured output at all. A chat interface, a summary, or a draft email all have a person as the validator, and adding a schema in front of that just constrains the model for no benefit.

If you're extracting one or two fields rather than a document's worth, a well-targeted prompt with a light safeParse and a single retry will serve you fine. The retry loop, semantic validation layer, and streaming parser are answers to problems that appear at scale and with schema complexity.

If your volume is low and a human reviews every result anyway, the validation layer is duplicating work someone is already doing. Surface the raw output, let the reviewer correct it, and spend the engineering time elsewhere.

And if you're still exploring what the feature should be, resist building this early. The schema is the most expensive thing to change once extraction code, validation rules, and stored data all depend on it. Prototype loosely, learn what fields you actually need, and tighten afterwards.

The safest rule is to start with a schema and safeParse, and add each subsequent layer only when a real failure justifies it. Every technique in this article came from a specific production incident rather than from a design document.

Wrapping Up

The gap between a working demo and a reliable feature is almost entirely in this layer. The model isn't the hard part any more, and the prompt isn't usually the hard part either. The hard part is deciding what you'll accept, detecting when you didn't get it, and responding sensibly when that happens.

Three ideas carry most of the weight:

1. The schema is the contract, and everything derives from it. One definition producing your type, your API schema, and your validator means those three can never drift apart.

2. Separate syntax failures, schema failures, and semantic failures. They have different causes and different fixes, and a retry loop that treats them identically will waste money on problems that repeating the call can't solve.

3. Make fields nullable rather than optional. Forcing the model to say "this was not present" rather than allowing it to quietly omit the field converts a whole class of silent hallucination into an explicit, checkable value.

Get those right and the rest is ordinary engineering. You're writing validation code and retry logic, which developers have been doing against unreliable inputs for decades. A language model is just a new kind of unreliable input, and it responds well to the same discipline.

References

• Zod: the schema library used throughout these examples.

• zod-to-json-schema: converts a Zod schema into the JSON Schema an API expects.

• Anthropic tool use documentation: forcing structured arguments through a tool definition.

• OpenAI structured outputs: strict schema conformance through constrained decoding.

• Outlines: grammar-constrained generation for locally hosted models.

• best-effort-json-parser: parsing incomplete JSON while a response is still streaming.

Try forcing too much through it, and either the lawn floods (your DOM) or the pipe bursts (your app).

There's a second observation that pairs with the first. Your laptop has 8 to 16 CPU cores. Your React app uses 1 of them, almost always. The main thread handles JavaScript, the DOM, layout, and paint setup. The other cores sit idle while the main thread struggles to keep a 60fps frame budget.

Both problems have the same shape: you need to keep React out of the hot path, and you need to use more than one thread. The patterns that get you there also happen to be the patterns behind Figma's canvas engine, Bloomberg's trading dashboards, and every biosignal viewer you've seen.

In one project, I had to visualise 19 EEG (brainwave) channels, each sending about 1,000 data points per second. That's almost 19,000 updates per second. If you feed all of that directly into React, the UI doesn't just slow down. It faints dramatically.

This article is the end-to-end architecture I landed on: the ring buffers, workers, shared memory, off-main rendering, and specific patterns that hold up under sustained multi-hour load.

Table of Contents

• Prerequisites

• Who's Already Doing This?

• The 1kHz Math

• Where it Usually Goes Wrong

• The Mental Model: Air Traffic Control Plus a Kitchen Brigade

• Step 1: Stop Putting Samples in React State

• Step 2: Separate Shape from Values

• Step 3: Move Heavy Work Off the Main Thread

• Step 4: Render Off Main with OffscreenCanvas

• Step 5: Decimate Before You Draw

• Step 6: Wrap an External Renderer

• Step 7: When Canvas isn't Enough, Reach for WebGL

• Step 8: Keep Memory Flat

• Step 9: Scheduling Strategies

• Step 10: Measure Sustained Performance

• Case Study: 19 EEG Channels

• Benchmarks: Single Thread vs Multi-Thread

• The COOP/COEP Catch

• Production Tradeoffs

• Should You Build Like This?

• Wrapping Up

• References

Prerequisites

To get the most out of this article, you'll want:

• Working knowledge of React 18 or 19. You should be comfortable with useState, useEffect, useRef, and the difference between mounting and re-rendering.

• TypeScript basics. Most examples are in TypeScript. You should be able to read type annotations without stopping.

• A rough sense of the browser main thread and event loop. You don't need to have written a Web Worker, but knowing what "blocking the main thread" means will make Step 3 easier.

• Familiarity with Canvas 2D or a chart library is a plus, not a requirement. If you've drawn anything on a canvas, you're ready.

• A laptop that can run modern Chrome or Edge. The examples rely on SharedArrayBuffer, OffscreenCanvas, and Atomics, which need a Chromium-based browser and cross-origin isolation (covered later in the article).

You don't need prior experience with Web Workers, ring buffers, or WebGL. This article introduces each in the context of a real problem.

Who's Already Doing This?

The patterns in this article aren't experimental. They're the architecture behind production apps that ingest and render high-frequency data:

• Trading and finance dashboards (Bloomberg, Hyperliquid, dYdX, every serious market viewer) push thousands of price ticks per second through canvas-rendered grids.

• Figma runs its entire canvas engine in WebAssembly inside a worker. The main thread renders React for the chrome only.

• Google Docs and Microsoft Loop run their document models in workers, with the DOM as the projection.

• Charting libraries like LightningChart, uPlot, Plotly, and ECharts draw on Canvas or WebGL and treat React as a wrapper.

• Biosignal, ECG, EEG, and motion-capture apps routinely process samples at 1kHz or higher and stream them to live plots.

• Observability and APM tools (Datadog live tail, Grafana real-time panels) decouple ingestion from render to keep tabs responsive.

• Audio editors and visualisers, plus anything using the Web Audio API with a waveform display.

• transformers.js and ONNX Runtime Web place ML inference in workers by default.

Different domains, same trick: React owns what changes rarely, something else owns what changes at refresh rate, and heavy work happens on threads that aren't the main one.

The 1kHz Math

Some numbers to make the problem concrete.

• A sample arrives every 1ms.

• A 60Hz display refreshes every 16.67ms.

• So in one frame, you'll receive roughly 16 to 17 samples per stream.

• With 19 active streams (the EEG case), that's 300 to 320 samples per frame.

If you setState on each sample, React tries to do around 19,000 renders per second. It can't, so it skips frames. The UI stutters and your laptop fans take off.

If you setState once per frame with the batch of ~320 samples, React does 60 renders per second, which is easy.

That single reframe is the entire trick, and it will echo through every step below.

Where it Usually Goes Wrong

Here's the version of the code I see in most real-time React apps the first time they try this. It looks reasonable. It's also the source of every bit of jank the team will spend the next two weeks tracking down.

import { useEffect, useState } from "react";

export default function NaiveChart({ socket }) {
  const [data, setData] = useState<number[]>([]);

  useEffect(() => {
    socket.on("newPoint", (point: number) => {
      setData((prev) => [...prev, point]); // re-renders every time
    });
  }, [socket]);

  return <div>{data.length} points</div>;
}

Three problems baked into seven lines:

1. Every incoming sample triggers a re-render: At 1kHz that's 1,000 renders per second. React was never going to be happy about that.

2. [...prev, point] allocates a new array on every push: At 1kHz that's a new array per millisecond, all of which the garbage collector has to clean up. The heap climbs, GC pauses lengthen, the fan kicks in.

3. There's no upper bound on the array: Run this for an hour and you have 3.6 million numbers in memory, all of which React has to consider on every render.

The fix isn't one trick. It's a stack of small ones, each addressing one of those failure modes and, eventually, the deeper problem of the main thread being the only thread.

The Mental Model: Air Traffic Control Plus a Kitchen Brigade

Before the code, two metaphors that will keep the pieces straight.

First, air traffic control: Three roles, one airport.

• The control tower (your store) sees every plane (every sample) and tracks where it is.

• The ground crew (React) sets up the runways and gates: the layout that planes use.

• The pilots (the draw loop) actually fly. They look at the tower for clearance and act every few seconds.

The control tower doesn't pull a gate from the ground crew every time a plane moves. It just holds the data. The ground crew rearranges gates when the schedule changes, which is rare. The pilots act constantly, at their own rate, off the tower's data.

Second, a kitchen brigade: A restaurant kitchen at peak hour. One head chef can't make every dish. The classical brigade has stations: sauce, fish, pastry, garde manger, plating, and service. Each station owns a slice of the meal. The expediter coordinates timing.

• The expediter is your main thread.

• The stations are your workers.

• The plating window is your shared memory.

• The dishes going to tables are your rendered frames.

A single chef trying to do everything serially is your main-thread-only frontend. A brigade is your worker-based one. The brigade is faster because cuts, sauces, and sears happen in parallel, not because any one cook is faster than the soloist. The expediter doesn't cook, they orchestrate.

Everything below is a specific application of these two ideas.

Step 1: Stop Putting Samples in React State

The single biggest mistake in real-time React apps is treating every sample as state, which it isn't. State is what determines which components exist and how they're arranged. A live plot is one component. The 60,000 samples scrolling across it aren't 60,000 pieces of state. They're one buffer.

You want a store that lives outside React. A ring buffer over a typed array gives you constant-time inserts and a bounded heap:

type Listener = () => void;

export function createSampleStore(capacity: number) {
  const buf = new Float32Array(capacity);
  let head = 0;
  let size = 0;
  const listeners = new Set<Listener>();

  return {
    push(sample: number) {
      buf[head] = sample;
      head = (head + 1) % capacity;
      if (size < capacity) size++;
    },
    pushBatch(samples: Float32Array) {
      for (let i = 0; i < samples.length; i++) {
        buf[head] = samples[i];
        head = (head + 1) % capacity;
        if (size < capacity) size++;
      }
    },
    read(): Float32Array {
      if (size < capacity) return buf.subarray(0, size);
      const out = new Float32Array(capacity);
      out.set(buf.subarray(head));
      out.set(buf.subarray(0, head), capacity - head);
      return out;
    },
    subscribe(l: Listener) {
      listeners.add(l);
      return () => listeners.delete(l);
    },
  };
}

Notice what's not here: no useState, no setter, no React anything. It's just plain JavaScript. The store can accept a million pushes per second and React won't care, because React isn't subscribed.

Here are the ring buffer mechanics, visualised:

We have head advances on every push and wraps at capacity. Reads pull a window of the last N samples in chronological order. The result is constant time and a bounded heap.

A simpler stop-gap, when you don't want a typed buffer yet, is to put the rolling window in a ref instead of state:

import { useEffect, useRef } from "react";

export default function RefChart({ socket }) {
  const bufferRef = useRef<number[]>([]);

  useEffect(() => {
    socket.on("newPoint", (point: number) => {
      bufferRef.current.push(point);
      if (bufferRef.current.length > 1000) {
        bufferRef.current.shift();
      }
    });
  }, [socket]);

  return <div>Streaming {bufferRef.current.length} points</div>;
}

This is the "10x faster than NaiveChart, still not great" version. The render doesn't trigger on every push, but you also won't see updates unless something else re-renders. For real plotting, pair the ref with a requestAnimationFrame draw loop (see Step 2).

Step 2: Separate Shape from Values

React renders when the shape of the UI changes. New plot? Re-render. Removed plot? Re-render. Switched from line to bar? Re-render. None of those happen at 1000Hz. They happen a few times a minute, when the user clicks something.

The values inside each plot change at the data rate. Those should never touch React.

Concretely:

function LivePlot({ store }: { store: SampleStore }) {
  const canvasRef = useRef<HTMLCanvasElement>(null);

  useEffect(() => {
    const canvas = canvasRef.current;
    if (!canvas) return;
    const ctx = canvas.getContext("2d")!;

    let raf = 0;
    const draw = () => {
      const samples = store.read();
      drawSeries(ctx, samples);
      raf = requestAnimationFrame(draw);
    };
    raf = requestAnimationFrame(draw);
    return () => cancelAnimationFrame(raf);
  }, [store]);

  return <canvas ref={canvasRef} width={1200} height={300} />;
}

React renders this component once. The useEffect runs once. The requestAnimationFrame loop reads from the store on every frame.

The store can be pushed to at any rate the source can manage. The user sees a smooth 60fps line whether the source is sending 100 samples per second or 100,000. The instinct to wire samples into useState is wrong, so resist it.

Selective Subscriptions When You Do Need React in the Loop

Sometimes a component genuinely depends on the data (summary stats, axis labels, or a live value badge). For that, use a store with selective subscriptions so only the components that care re-render. Zustand makes this trivial:

import { create } from "zustand";

const useDataStore = create<{ latest: number | null; setLatest: (v: number) => void }>((set) => ({
  latest: null,
  setLatest: (v) => set({ latest: v }),
}));

function LatestBadge() {
  // Only re-renders when latest changes, not when other store fields do.
  const latest = useDataStore((state) => state.latest);
  return <div>Latest: {latest?.toFixed(2)}</div>;
}

Pair this with RAF coalescing on the writes (call setLatest once per frame, not once per sample) and you get a React component that updates smoothly at 60fps no matter what the data rate is.

useSyncExternalStore is the native equivalent and works against any pub-sub store, including the ring buffer above. Use whichever feels lighter for your team.

Step 3: Move Heavy Work Off the Main Thread

The store and imperative draw loop handle React's contribution to the bottleneck. The main thread itself is still doing all the ingest, parsing, and math. Everything past ~50,000 samples per second per stream needs more.

The browser gives you four escape hatches: Web Workers, transferable objects, SharedArrayBuffer + Atomics, and OffscreenCanvas. Each solves a specific problem.

The main thread runs JavaScript, the DOM, layout, and paint setup. Workers are isolated JavaScript contexts with their own event loops. They can't touch the DOM, they can't share memory by default, and every cross-thread message is async and copied unless you transfer it.

Three properties matter as you design against this model:

• Workers can't touch the DOM: That's the point. They run pure JavaScript. Perfect for data work, network parsing, math, codecs, ML inference.

• Communication is async: No shared variable to read in the middle of a function. Plan APIs around requests and events.

• Data copies, unless you transfer or share it: Structured clone is the default, transferable objects skip the copy, and SharedArrayBuffer skips it permanently.

Dedicated Workers in Practice

Here's a minimal worker for ingest:

// worker.ts
self.onmessage = (event) => {
  const { samples } = event.data;
  // Decode, filter, decimate. The main thread doesn't see any of this.
  const summary = computeStats(samples);
  postMessage(summary);
};

// main thread
import { useEffect, useRef } from "react";

export default function WorkerChart({ socket, store }) {
  const workerRef = useRef<Worker>();

  useEffect(() => {
    workerRef.current = new Worker(new URL("./worker.ts", import.meta.url), {
      type: "module",
    });

    workerRef.current.onmessage = (event) => {
      store.pushBatch(event.data); // pre-processed, cheap to ingest
    };

    socket.on("newPoint", (point: number) => {
      workerRef.current?.postMessage({ point });
    });

    return () => workerRef.current?.terminate();
  }, [socket, store]);

  return <LivePlot store={store} />;
}

Three things to internalise about workers.

First, workers are real processes from the runtime's perspective. Each has its own heap, event loop, and globalThis. Starting one costs about 1 to 5ms. Don't spin them up inside hot paths.

Second, module workers are the modern default. type: "module" enables ES modules inside the worker, including import. The legacy importScripts is for classic workers, so avoid it for new code.

Finally, bundlers know about workers. Vite, Webpack, esbuild, and Rspack all detect the new Worker(new URL("./x.ts", import.meta.url)) pattern and produce a separate chunk for the worker.

A Worker Pool That Scales with Cores

For CPU-bound work (parsing binary frames, decoding audio, computing FFTs), a pool spreads jobs across all available cores:

class WorkerPool {
  private workers: Worker[];
  private next = 0;
  private pending = new Map<string, (result: unknown) => void>();

  constructor(scriptUrl: URL, size = Math.max(1, navigator.hardwareConcurrency - 1)) {
    this.workers = Array.from({ length: size }, () => {
      const w = new Worker(scriptUrl, { type: "module" });
      w.onmessage = (event) => {
        const cb = this.pending.get(event.data.id);
        if (!cb) return;
        this.pending.delete(event.data.id);
        cb(event.data.result);
      };
      return w;
    });
  }

  async run<T>(kind: string, payload: unknown, transferables: Transferable[] = []): Promise<T> {
    const id = crypto.randomUUID();
    const result = new Promise<T>((resolve) => this.pending.set(id, resolve as (r: unknown) => void));
    const worker = this.workers[this.next];
    this.next = (this.next + 1) % this.workers.length;
    worker.postMessage({ id, kind, payload }, transferables);
    return result;
  }
}

export const pool = new WorkerPool(new URL("./decoder.worker.ts", import.meta.url));

Round-robin distribution with one worker per core minus one. Pending requests resolve when the matching response arrives. It's cheap, predictable, and scales linearly until you hit memory bandwidth limits.

Transferable Objects, the No-copy Path

postMessage clones the payload by default. Cloning a 10MB buffer takes milliseconds and doubles your memory use. The fix is to transfer the buffer instead.

The semantics: when you transfer a buffer, the sender loses access. The receiver gains it, with no copy and an O(1) handoff.

const buf = new ArrayBuffer(10 * 1024 * 1024); // 10 MB
new Uint8Array(buf).set(somePayload);

worker.postMessage({ buf }, [buf]); // second arg = list of transferables
// `buf` is now detached on this side. Accessing it throws.

The list of transferable types in 2026 includes ArrayBuffer (and any typed-array view backed by one), MessagePort, ImageBitmap, OffscreenCanvas, the stream types (ReadableStream, WritableStream, TransformStream), RTCDataChannel, VideoFrame, AudioData, and the WebTransport streams. The most useful for React plus real-time work are ArrayBuffer, OffscreenCanvas, and MessagePort.

A common pitfall is that forgetting to transfer creates silent slowness. The app works, but it copies every message. Profile worker postMessage calls. If they're showing milliseconds for "small" payloads, you're cloning when you should be transferring.

// Bad: copies every frame.
worker.postMessage({ samples: float32Array });

// Good: transfers the underlying buffer.
worker.postMessage({ samples: float32Array }, [float32Array.buffer]);

The buffer is detached after transfer, so the sender needs to re-allocate (or pull from a pool of pre-allocated buffers) if it wants to keep producing.

SharedArrayBuffer and Atomics

Transfer hands a buffer off. Sharing lets both threads see the same memory simultaneously.

const sab = new SharedArrayBuffer(1024 * 1024); // 1 MB shared
worker.postMessage({ sab });

// Both main and worker now hold references to the same memory.
const viewMain = new Int32Array(sab);
// Inside worker:
// const viewWorker = new Int32Array(event.data.sab);

Three properties matter.

SharedArrayBuffer needs cross-origin isolation. Your page must be served with Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp. Without these headers, SharedArrayBuffer is undefined in the browser. Covered in more detail later in the article.

You also need Atomics for synchronisation. Multiple threads writing to the same memory without coordination produces undefined results. Atomics gives you compare-and-swap, load, store, add, sub, wait, and notify operations on typed array views.

const sab = new SharedArrayBuffer(8);
const view = new Int32Array(sab);

// Main thread: wake any worker that's waiting on slot 0.
Atomics.store(view, 0, 1);
Atomics.notify(view, 0, 1);

// Worker: block until slot 0 changes from 0.
const result = Atomics.wait(view, 0, 0); // "ok", "not-equal", or "timed-out"

And lock-free ring buffers are the killer app. A producer-consumer queue between two threads with no locking, no postMessage round-trip, and no GC pressure. The data sits in the shared buffer. Atomics coordinate read/write positions.

Here's a minimal SPSC (single-producer, single-consumer) ring buffer:

type SharedRing = {
  data: Float32Array;          // payload
  control: Int32Array;         // [head, tail]
};

function createSharedRing(capacity: number): SharedRing {
  const sab = new SharedArrayBuffer(capacity * 4 + 16);
  const control = new Int32Array(sab, 0, 4);   // [head, tail, ...]
  const data = new Float32Array(sab, 16, capacity);
  return { data, control };
}

function push(ring: SharedRing, value: number): boolean {
  const head = Atomics.load(ring.control, 0);
  const tail = Atomics.load(ring.control, 1);
  const next = (head + 1) % ring.data.length;
  if (next === tail) return false; // full
  ring.data[head] = value;
  Atomics.store(ring.control, 0, next);
  return true;
}

function pop(ring: SharedRing): number | null {
  const tail = Atomics.load(ring.control, 1);
  const head = Atomics.load(ring.control, 0);
  if (tail === head) return null; // empty
  const value = ring.data[tail];
  Atomics.store(ring.control, 1, (tail + 1) % ring.data.length);
  return value;
}

The producer writes from one thread. The consumer reads from another, neither blocks, and there's no postMessage between them. At signal rates, this is the only architecture that scales.

For multi-producer or multi-consumer queues, you need compare-and-swap loops (Atomics.compareExchange). They get fiddly fast. Most production setups use SPSC where they can and fall back to message-passing where they can't.

In Electron, Ingest in the Main Process

The same idea applies one level up. With Electron and a native SDK, you can ingest samples in the main process, buffer there, and forward batches across IPC at the renderer's refresh cadence. One IPC message per sample at 1kHz will saturate IPC. One IPC message per frame with a batch of 16 samples is trivial.

// electron main: buffer in Node, flush at 60Hz
let pending: number[] = [];

device.on("sample", (value) => pending.push(value));

setInterval(() => {
  if (pending.length === 0) return;
  const batch = new Float32Array(pending);
  pending = [];
  // Transferable to avoid the structured-clone copy.
  mainWindow.webContents.send("device:samples", batch.buffer, [batch.buffer]);
}, 1000 / 60);

// preload: expose a thin subscription API to the renderer
contextBridge.exposeInMainWorld("device", {
  onSamples: (cb: (samples: Float32Array) => void) => {
    const handler = (_: unknown, buf: ArrayBuffer) => cb(new Float32Array(buf));
    ipcRenderer.on("device:samples", handler);
    return () => ipcRenderer.removeListener("device:samples", handler);
  },
});

// renderer: feed the store from the bridge
useEffect(() => {
  return window.device.onSamples((batch) => store.pushBatch(batch));
}, []);

The renderer thread of the renderer touches only one batch per frame, no matter how fast the device is. The data work happens upstream.

Step 4: Render Off Main with OffscreenCanvas

The DOM is single-threaded. The Canvas API used to be too. OffscreenCanvas breaks that: a canvas you can transfer to a worker, where it draws independently of main.

// Main thread
const canvas = canvasRef.current!;
const offscreen = canvas.transferControlToOffscreen();
worker.postMessage({ canvas: offscreen }, [offscreen]);

// Worker
let ctx: OffscreenCanvasRenderingContext2D | null = null;

self.onmessage = (event) => {
  if (event.data.canvas) {
    ctx = event.data.canvas.getContext("2d");
    return;
  }
  if (event.data.samples && ctx) {
    drawSeries(ctx, event.data.samples);
  }
};

The main thread is now free to handle clicks, hover, and other interaction without competing with the draw loop. The worker draws at its own rate, against whatever data it has.

Combined with SharedArrayBuffer, you get the cleanest real-time rendering pipeline available in the browser:

The ingest worker writes samples into the shared buffer. The render worker reads them and draws. Main only reads summaries (FPS, latest value, channel labels) and renders chrome. No data ever crosses through main's event loop.

For 19-channel signal at 1kHz, this is the only architecture that keeps a guaranteed 60fps on a laptop, with headroom.

Step 5: Decimate Before You Draw

A 1200-pixel-wide canvas can show, at best, 1200 distinct X positions. If you have 60,000 samples in your window and you draw all of them, you do 50 times more work than the user can see.

Pick the right point per pixel column. The "min-max" pattern works well for signals: for each pixel column, find the minimum and maximum value in that range and draw a vertical line between them. This is visually identical to drawing every point, but it's much cheaper.

function drawDecimated(
  ctx: CanvasRenderingContext2D,
  samples: Float32Array,
  width: number,
) {
  const samplesPerPixel = samples.length / width;
  ctx.beginPath();
  for (let x = 0; x < width; x++) {
    const start = Math.floor(x * samplesPerPixel);
    const end = Math.floor((x + 1) * samplesPerPixel);
    let min = Infinity;
    let max = -Infinity;
    for (let i = start; i < end; i++) {
      const v = samples[i];
      if (v < min) min = v;
      if (v > max) max = v;
    }
    ctx.moveTo(x, scale(min));
    ctx.lineTo(x, scale(max));
  }
  ctx.stroke();
}

Decimation is the single biggest CPU win in real-time visualisation, and almost nobody does it.

A more visually faithful variant is LTTB (Largest Triangle Three Buckets), an algorithm that picks one representative sample per bucket while preserving the visual shape better than min-max. It's worth the read if you're plotting non-signal data like stock charts where peaks and dips matter individually. Most good chart libraries (uPlot, Plotly, ECharts) include decimation out of the box.

Step 6: Wrap an External Renderer

You don't always need to write the draw loop yourself. React is great at lifecycle management. Imperative chart libraries are great at raw performance. The right move is often to let React mount and unmount the chart while the library handles the fast inner loop.

import Uplot from "uplot";
import "uplot/dist/uPlot.min.css";
import { useEffect, useRef } from "react";

export default function UPlotChart({ data }: { data: AlignedData }) {
  const ref = useRef<HTMLDivElement>(null);
  const plotRef = useRef<Uplot | null>(null);

  useEffect(() => {
    const opts = {
      title: "Realtime Chart",
      width: 600,
      height: 300,
      series: [{}, { label: "Signal" }],
    };
    plotRef.current = new Uplot(opts, data, ref.current!);
    return () => plotRef.current?.destroy();
  }, []);

  useEffect(() => {
    plotRef.current?.setData(data); // imperative update, no React render
  }, [data]);

  return <div ref={ref} />;
}

uPlot, TimeChart, ECharts, Plotly, LightningChart, and the others all follow this shape: instantiate inside useEffect, call setData imperatively, and destroy on unmount. React orchestrates and the library renders.

The general pattern, for any imperative renderer, looks like this:

function ChartWrapper({ config, data }) {
  const ref = useRef<HTMLDivElement>(null);

  useEffect(() => {
    const chart = new SomeFastChartLib(ref.current, config);
    chart.setData(data);
    return () => chart.destroy();
  }, [config, data]);

  return <div ref={ref} />;
}

This is the highest-ROI move if you're not in a domain where you need pixel-level control over the drawing. Reach for a library first, and only write your own draw loop when no library fits.

Step 7: When Canvas isn't Enough, Reach for WebGL

Canvas 2D handles a few thousand line segments per frame comfortably. Past that, you start spending milliseconds in stroke() itself.

WebGL (or its higher-level wrappers like regl, twgl, pixi.js, deck.gl) moves the rendering to the GPU. You pay an upfront cost (writing shaders and managing buffers) for the ability to draw millions of points without breaking a sweat.

Here's a minimal WebGL "line strip with a single vertex shader" sketch:

const gl = canvas.getContext("webgl2")!;

const program = gl.createProgram()!;
// ... compile vertex + fragment shaders, link, get attribute location ...

const samplesBuffer = gl.createBuffer();
function drawWebGL(samples: Float32Array) {
  gl.bindBuffer(gl.ARRAY_BUFFER, samplesBuffer);
  gl.bufferData(gl.ARRAY_BUFFER, samples, gl.STREAM_DRAW);
  gl.useProgram(program);
  gl.drawArrays(gl.LINE_STRIP, 0, samples.length);
}

The pattern is the same as Canvas 2D: an imperative draw call inside the RAF loop. The difference is the GPU does the actual rasterising. Most real-time charting libraries that claim "millions of points" are doing exactly this under the hood.

Default to Canvas 2D, and escalate to WebGL only when you can prove Canvas is the bottleneck on the Profiler.

Step 8: Keep Memory Flat

Real-time apps die slowly. They run smoothly for an hour and then the laptop fan kicks in. The cause is almost always memory.

There are three rules that keep the heap calm.

First, use typed arrays for numeric data. A Float32Array of 60,000 floats is 240KB. A plain JavaScript array of the same size is around 1.4MB and creates GC pressure on every push.

Second, bound everything. Use ring buffers, not growing arrays. Put caps on history, queues, and pending work. Anything that can grow unbounded eventually will.

Third, reuse buffers. Allocate once in setup, and reuse across frames. Allocating a fresh array per frame at 60fps is 60 allocations per second per plot. Multiply by plot count.

// Bad: fresh array every frame.
function drawFrame() {
  const snapshot = store.read(); // returns a new Float32Array
  drawDecimated(ctx, snapshot, width);
}

// Better: reuse a draw buffer.
const drawBuf = new Float32Array(WINDOW_SIZE);

function drawFrame() {
  store.readInto(drawBuf); // writes into the existing buffer
  drawDecimated(ctx, drawBuf, width);
}

These rules aren't React-specific. They're the rules any real-time system follows. The reason they need stating is that React-shaped thinking ("derive a new array each render") is the opposite of what real-time wants.

Step 9: Scheduling Strategies

A worker pool with no scheduling is just a queue. Sometimes you want priority: a user-driven action should jump ahead of a background sweep.

Priority Queues

A small priority scheduler:

type Priority = "high" | "normal" | "low";

class PriorityPool {
  private queues: Record<Priority, Job[]> = { high: [], normal: [], low: [] };
  private idle: Worker[];

  enqueue(job: Job, priority: Priority = "normal") {
    if (this.idle.length > 0) {
      const worker = this.idle.pop()!;
      this.dispatch(worker, job);
    } else {
      this.queues[priority].push(job);
    }
  }

  private next(): Job | null {
    for (const p of ["high", "normal", "low"] as const) {
      const j = this.queues[p].shift();
      if (j) return j;
    }
    return null;
  }

  private dispatch(worker: Worker, job: Job) {
    worker.postMessage(job.payload, job.transferables);
    worker.onmessage = (event) => {
      job.resolve(event.data);
      const next = this.next();
      if (next) this.dispatch(worker, next);
      else this.idle.push(worker);
    };
  }
}

Three buckets are usually enough: high (user-initiated), normal (steady-state work), low (background sweeps, prefetch, telemetry flushing).

Chunkable Work

A job that takes 2 seconds blocks a worker for 2 seconds. If you want to keep workers responsive to higher-priority jobs, the job has to be chunkable.

// In the worker:
self.onmessage = async (event) => {
  const { id, kind, payload } = event.data;
  if (kind === "decode_large") {
    const total = payload.byteLength;
    for (let i = 0; i < total; i += CHUNK) {
      const chunk = decodeChunk(payload, i, Math.min(i + CHUNK, total));
      self.postMessage({ id, kind: "progress", chunk, offset: i });
      // Yield so the worker can check its message queue.
      await new Promise((r) => setTimeout(r, 0));
    }
    self.postMessage({ id, kind: "done" });
  }
  if (kind === "cancel") {
    // ... abort the current job
  }
};

Yielding inside a worker isn't free, but it lets the worker process cancellation messages or higher-priority jobs interleaved with the big task. For long jobs (like a firmware flash, large file decode, or big render), this is essential.

Fan-out and Fan-in

Split a big job into N pieces, dispatch each to a different worker, and gather the results.

async function decodeFile(buffer: ArrayBuffer): Promise<DecodedFrame[]> {
  const chunks = splitBuffer(buffer, 8);
  const results = await Promise.all(
    chunks.map((chunk) => pool.run<DecodedFrame[]>("decode", chunk, [chunk])),
  );
  return results.flat();
}

This gives you linear speedup on parallelisable workloads, up to the worker count. It's critical for batch operations like decoding a recording, summarising a long document, or computing embeddings for a folder.

Step 10: Measure Sustained Performance

A spike on the Profiler is one thing. A slow heap creep over an hour is another. For real-time apps, you measure two things.

First, frame consistency. Are you holding 60fps consistently, or dropping occasional frames? A simple FPS meter:

let lastTime = performance.now();
let frames = 0;

function rafLoop(now: number) {
  frames++;
  if (now - lastTime >= 1000) {
    const fps = (frames * 1000) / (now - lastTime);
    frames = 0;
    lastTime = now;
    console.log(`fps: ${fps.toFixed(1)}`);
  }
  requestAnimationFrame(rafLoop);
}
requestAnimationFrame(rafLoop);

A more honest measure is the worst frame in a window. Average frame time hides the jank.

Second, heap stability. Open DevTools Memory tab, take a heap snapshot, run the app for ten minutes, take another snapshot, and then compare. The diff should be flat. If it's growing, you have a leak: retained listeners, growing arrays, or closures holding onto large objects.

For real apps, wire heap usage and FPS into your analytics so you spot regressions before users complain.

Case Study: 19 EEG Channels

Back to the project that started this article: we have nineteen channels, each at 1kHz, all rendered simultaneously, and all needing to stay smooth across multi-hour recording sessions.

The first build used LightningChart. It's powerful, capable, and beautiful. But it's also heavy. Memory usage climbed noticeably, the chart's own internals were doing a lot of work for our use case (which was 19 simple line plots, not full multi-axis financial charts), and the licensing was a friction point.

We switched to uPlot. It's tiny, fast, and written specifically for time-series. Memory usage dropped, render time per frame went from "occasionally over budget" to "always under," and my machine stopped sounding like it was about to take off. The chart library change alone bought us most of the headroom we needed.

The architecture around the chart did the rest. The pipeline runs on four threads:

• One ingest worker reads from the device SDK over IPC (Electron main process to renderer).

• One SharedArrayBuffer holds the rolling window for all 19 channels.

• One render worker reads the SAB and draws to an OffscreenCanvas.

• Main thread renders chrome: channel labels, controls, and the FPS meter.

The shared buffer layout is one big Float32Array indexed as [channel * samplesPerChannel + sampleIndex]. The ingest worker writes new samples and advances per-channel head pointers (stored in an Int32Array slice of the SAB). The render worker reads the latest window each frame.

const CHANNELS = 19;
const WINDOW_SAMPLES = 60_000;

const sab = new SharedArrayBuffer(CHANNELS * WINDOW_SAMPLES * 4 + CHANNELS * 4);
const heads = new Int32Array(sab, 0, CHANNELS);
const samples = new Float32Array(sab, CHANNELS * 4, CHANNELS * WINDOW_SAMPLES);

function writeSample(channel: number, value: number) {
  const head = Atomics.load(heads, channel);
  samples[channel * WINDOW_SAMPLES + head] = value;
  Atomics.store(heads, channel, (head + 1) % WINDOW_SAMPLES);
}

The renderer reads the channel buffers, decimates per pixel column, and draws:

// render.worker.ts
function drawFrame() {
  const ctx = offscreenCtx;
  ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height);
  for (let ch = 0; ch < CHANNELS; ch++) {
    const start = ch * WINDOW_SAMPLES;
    drawDecimatedRow(ctx, samples.subarray(start, start + WINDOW_SAMPLES), ch);
  }
  requestAnimationFrame(drawFrame);
}

On a 2024 M3 MacBook Pro, this holds 60fps with 19 channels at 1kHz, and 144fps if the display supports it. The main thread stays under 1% utilisation. Workers consume the spare cores. The user feels something that used to require a native app.

The lesson, in short: most of the work is choosing the right tool for the inner loop, and getting out of its way.

Benchmarks: Single Thread vs Multi-Thread

Here are numbers from running comparable workloads on a 2024 M3 MacBook Pro. They're indicative, not promissory.

The pattern: workers alone help, and workers plus shared memory help more. Workers plus shared memory plus OffscreenCanvas is what gets you to "the main thread is doing nothing and the chart is still smooth."

For chart-heavy apps, the leap from "workers" to "workers plus OffscreenCanvas" is the biggest single architectural improvement available without leaving the browser.

The COOP/COEP Catch

SharedArrayBuffer and high-resolution timers were tightened in 2020 after Spectre/Meltdown. To use them, your page must be served with two HTTP headers:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

This opts your page into "cross-origin isolation." Within an isolated context, SharedArrayBuffer exists, performance.now() is high-resolution, and various other restricted APIs work.

Outside an isolated context, SharedArrayBuffer is undefined, performance.now() is throttled to about 1ms precision, and Atomics throw.

The cost is significant. require-corp means every cross-origin resource (images from a CDN, embedded YouTube videos, third-party fonts, analytics scripts) must explicitly opt in by setting Cross-Origin-Resource-Policy: cross-origin or Cross-Origin-Embedder-Policy: credentialless. Many third-party services don't, which breaks their embedding.

There are two practical options:

• For an Electron app: the renderer process can be configured to use these headers easily. Most production Electron apps that want real-time visualisation enable them by default.

• For a browser app: weigh the embeds you'd lose against the performance you'd gain. If your app is the main attraction (Figma, Google Docs), opt in. If you depend on third-party widgets, the cost is real.

For chart-heavy or signal-heavy apps that need SAB, the Electron path is usually cleaner.

Production Tradeoffs

Here are the five real costs of everything above.

• Code complexity: A worker-driven app has 2 to 4 times the source files of a single-threaded one (main + workers + shared types). It's worth it for the right scale, but painful for a trivial app.

• Debugging: Stack traces split across threads. Chrome DevTools handles this well in 2026 (each worker has its own debugger panel), but it's still more work than a single-thread bug.

• Bundle size: Each worker is a separate chunk. Tree-shaking inside workers is sometimes worse than in main (less mature). Audit worker bundles separately.

• Startup latency: Spinning up workers at app start adds 50 to 200ms. Pre-warm them during the splash screen, or accept the first-frame delay.

• Browser API gaps: localStorage, document, and most DOM APIs aren't available in workers. Some libraries silently rely on them and break. Test in a worker context before bundling a library you haven't tried there.

There are three trade-offs specific to the imperative rendering pattern:

• Declarative animation of the data: The chart frame, labels, and controls all stay declarative. The data inside the chart becomes imperative.

• Easy snapshot testing of the rendered output: A canvas has no DOM you can query. Test the data path separately from the draw path. Snapshot the store output, not the pixels.

• React's component story for the inner loop: The draw loop is a closure. Composing draw loops is harder than composing components. Pick your component boundary carefully so each canvas does one thing.

For most apps these costs aren't worth paying. For an app that has to render 1kHz data smoothly, they're the price of admission.

Should You Build Like This?

If your data rate is below 30 updates per second per stream, none of this is needed. Naïve setState per batch will work. Profile first, optimise second.

This architecture earns its keep when:

• You have many streams or sensors

• Ingestion is sustained, not bursty

• The UX promise is smooth motion for hours, not seconds

• You'd rather not rewrite the UI in a native language to get there

The boring rule: start with requestAnimationFrame coalescing and external stores. Promote to workers when those aren't enough. Promote to shared memory when worker postMessage is the bottleneck. Promote to OffscreenCanvas when the render loop itself becomes the bottleneck. Each step is a real architectural investment. Take them in order.

Wrapping Up

React can handle real-time visualisation if you use it the right way. Instead of pushing React to do everything, use it as the conductor. Let specialised libraries and workers handle the heavy lifting.

These are the three rules that hold up across every high-frequency React app I've built:

1. Workers do the work, while main does the UI. If main is doing math, you've put the math in the wrong place.

2. Transfer if you can, share if you must. Both beat cloning. Sharing is more complex than transferring.

3. Let React orchestrate. Let specialised tools render. The store owns the data, the draw loop owns the values, and React owns the shape.

Get those right and your React app stops being a one-core system that flinches at high-frequency data. It becomes a real multi-core system that scales with the hardware, ingests without dropping, and renders without stuttering.

The cores are right there. Use them.

References

• uPlot: tiny, fast, focused on time-series.

• TimeChart: high-performance real-time chart on WebGL.

• Using Web Workers in React: MDN reference.

• SharedArrayBuffer on MDN: the shared-memory primitive.

• OffscreenCanvas on MDN: rendering off the main thread.

• COOP and COEP explainer: what cross-origin isolation buys you.

Then you read the diff, and you find that it pulled in a validation library that's not in your package.json, it wrote the test in Jest even though your team moved to the Node test runner last spring, and it reached into the database from inside the route handler because it had no way of knowing that every other handler in the codebase delegates to a service.

The code runs, the tests it wrote pass, but you still have to rewrite most of it.

None of that is a reasoning failure on the model's part. It produced a sensible solution to the problem as it understood it, but it understood the problem badly because nobody told it how this particular codebase works.

Your conventions live in your team's heads, in code review comments, and in decisions made eighteen months ago that nobody wrote down. The agent can't see any of that, so it falls back on the average of every repository it has ever been trained on, which is exactly what you got.

The fix isn't a longer prompt, since you would have to retype it every session and your teammates would each write a different version of it. The fix is a set of files that live in the repository, load automatically, and are maintained the same way you maintain code.

This tutorial shows you how to structure those files, how to keep one source of truth across the four or five formats the different tools expect, and, most importantly, how to stop them from quietly going out of date. After all, a context file that describes a codebase you deleted six months ago is worse than no context file at all.

Everything here is built on a companion repository you can clone and run: github.com/Adeniyikayodee/MCF. It has no dependencies, so Node 20 or newer is all you need.

Table of Contents

• What You Need Before You Start

• Why the Context Window is the Real Constraint

• The Three Layers

• Picking a Format Without Maintaining Four Copies

• Writing the Root File

• Scoping Rules to a Directory

• Pointing Instead of Inlining

• Making Context Files Verifiable

• Give the Agent Something to Verify Against

• Checking Whether it Actually Worked

• Keeping the Files Healthy

• Mistakes Worth Avoiding

• Where to Start

What You Need Before You Start

You should be comfortable with Git and a terminal, you should have Node 20 or newer installed, and you should have used at least one coding agent such as Claude Code, Cursor, GitHub Copilot, or Codex on a real project.

You don't need to know anything about how models work internally, since everything in this tutorial is about files on disk.

Why the Context Window is the Real Constraint

Everything an agent knows while it works on your task lives in one buffer called the context window. That buffer holds the system prompt, your conversation, every file the agent opened, every command it ran, and every stack trace those commands printed.

But it's important to know that it's finite, and it fills up faster than most people expect. A single debugging session, for example, can burn tens of thousands of tokens before the agent has written a line of code.

The part that matters for this tutorial is what happens as that buffer fills. Anthropic's engineering team describes an effect they call context rot, where a model's ability to retrieve a specific instruction degrades as the token count climbs. The model isn't ignoring you out of stubbornness, it's working with an attention budget that gets thinner as more material competes for it.

That single fact overturns the intuition most people bring to context files. Writing more feels safer, because you've covered more cases and left less to chance. But every line you add competes with every other line for a finite amount of attention.

The Claude Code documentation puts the consequence plainly, noting that a bloated instructions file causes the agent to ignore the rules inside it. Also, it notes that the symptom of an over-long file is the agent repeatedly breaking a rule you've clearly written down.

Here's roughly how a session budget gets spent on a real task:

system prompt and tool definitions        ~12,000 tokens
context files loaded at startup            ~4,800 tokens
three source files the agent opened        ~9,000 tokens
one test run with a stack trace            ~3,500 tokens

The 4,800 token context file in that list is competing with the stack trace the agent needs to read in order to fix the bug. A 600 token file that names the right paths would leave room for the agent to go and read the code itself, which it's very good at.

Context files are a budget allocation problem before they're a documentation problem, and almost every improvement in this tutorial comes from taking that seriously.

The Three Layers

The structure that works treats context as three distinct layers with different costs.

The always loaded layer is a single file at the root of your repository that the agent reads at the start of every session, whether the task is a typo fix or a migration. You pay for this file on every single request, so it holds only what applies to every task in the repository. It also stays small enough that you could read it aloud in under a minute.

The scoped layer is made up of nested files that load only when the agent works inside a particular directory. Rules about your API layer sit in src/AGENTS.md, so a task that only touches the frontend never pays for them.

The on demand layer is ordinary documentation that the root file points at by path rather than inlining. A path costs a handful of tokens while the document behind it might cost two thousand, so the agent spends that budget only when the task actually calls for it.

This mirrors how a new engineer works, since they don't memorise your architecture document on day one. They remember that it exists and go and read it when they need it.

The finished layout in the companion repository looks like this:

MCF/
├── AGENTS.md                          always loaded, budgeted
├── CLAUDE.md                          generated from AGENTS.md
├── .github/copilot-instructions.md    generated from AGENTS.md
├── .cursor/rules/testing.mdc          glob scoped, hand written
├── .claude/
│   ├── settings.json                  hook that runs the context linter
│   └── skills/add-endpoint/SKILL.md   workflow, loaded on demand
├── docs/
│   ├── architecture.md
│   ├── testing.md
│   └── decisions/0001-in-memory-store.md
├── scripts/
│   ├── context-lint.mjs
│   └── sync-context.mjs
├── src/
│   ├── AGENTS.md                      scoped to the source tree
│   ├── api/tasks.js
│   ├── services/tasks.js
│   ├── lib/validate.js
│   ├── router.js
│   └── server.js
└── tests/

Picking a Format Without Maintaining Four Copies

Every vendor picked a different filename for the same idea, which is annoying but manageable once you decide which one is the source of truth.

AGENTS.md is the closest thing to a shared convention. It's plain Markdown with no required schema, its governance sits with the Agentic AI Foundation under the Linux Foundation, and it's read natively by Claude Code, Codex, Cursor, Copilot, Gemini CLI, Aider, Windsurf, Zed, and a long list of others.

Nested files are part of the spec, the file closest to the code being edited takes precedence, and anything you type directly into the chat overrides all of it.

The tool-specific formats still exist alongside it. Claude Code reads CLAUDE.md, walks up the directory tree concatenating every one it finds, and resolves @path/to/file imports. Cursor uses .mdc files inside .cursor/rules/ with YAML frontmatter that can scope a rule to a glob such as tests/**/*.js, which makes it the most expressive of the formats and also the least portable, since nothing outside Cursor reads it. GitHub Copilot, for its part, reads a single .github/copilot-instructions.md at the repository root.

The practical answer is to write AGENTS.md once, generate the rest from it, and hand write a separate file only when a tool offers something the shared format can't express. In practice, this means Cursor's glob scoping. You can do the generating with symlinks:

ln -s AGENTS.md CLAUDE.md

Symlinks are the shortest path, though they cause trouble for contributors on Windows and for some CI checkout configurations, so the companion repository uses a small script instead. The script writes a banner into every file it generates, which stops a well-meaning teammate from editing the copy and losing their work on the next sync:

// scripts/sync-context.mjs
const banner = `<!-- Generated from ${SOURCE} by \`npm run sync:context\`. Edit ${SOURCE} instead. -->`;

export const targets = [
  // Claude Code resolves @path imports, so its file stays a pointer plus what is specific to it.
  { path: 'CLAUDE.md', render: () => `${banner}\n\n@${SOURCE}\n\n${CLAUDE_EXTRAS}` },
  // Copilot has no import syntax, so the source is inlined.
  { path: '.github/copilot-instructions.md', render: (source) => `${banner}\n\n${source}` },
];

Because Claude Code resolves imports, its generated file stays a pointer plus the handful of instructions that only make sense for that tool, which keeps it at around 130 tokens rather than duplicating the whole thing:

<!-- Generated from AGENTS.md by `npm run sync:context`. Edit AGENTS.md instead. -->

@AGENTS.md

## Claude Code specific

- Use plan mode for any change that touches more than three files, and skip it for a one line fix.
- Delegate codebase exploration to a subagent so the findings come back summarised rather than as
  a hundred file reads in the main context.

Running the script regenerates both files, and running it again does nothing. This is what you want from something a hook or a CI job will call repeatedly:

Writing the Root File

This is where most of the value is, and it's also where most people go wrong, because the instinct is to write everything down.

Use one editing test on every line you're tempted to add: would removing this line cause the agent to make a mistake? If the answer is no, the line is costing you attention budget and buying you nothing, so cut it. Applied honestly, that test removes most of what people put in these files.

Here's the kind of file the test is designed to catch:

# AGENTS.md

## About this project
This project is a REST API for managing tasks. It was originally built in 2023 by the platform
team and has since been maintained by the core services group. The codebase is written in modern
JavaScript using ES modules.

## Code style
- Use meaningful variable names
- Write clean, maintainable code
- Follow the DRY principle
- Use const instead of var
- Add comments where the code is complex

## Structure
- `src/server.js` contains the server
- `src/router.js` contains the router
- `src/api/tasks.js` contains the task handlers
- `src/services/tasks.js` contains the task service

Every line there fails the test. The model already knows what const is for, it can see that the file called router.js contains the router, and knowing which team owned the code in 2023 won't change a single decision it makes.

Meanwhile the one thing an agent genuinely can't work out on its own, which is that this project deliberately has no dependencies, is nowhere in the file.

This is the version that ships in the companion repository:

# AGENTS.md

Task API used as the worked example for a tutorial on managing context files. This file is the
single source of truth for agent instructions, and `CLAUDE.md` plus
`.github/copilot-instructions.md` are generated from it by `npm run sync:context`, so edit this
file and never the generated ones.

## Commands

- Install: nothing to install, the project has zero dependencies
- Run the tests: `npm test`
- Start the server on port 3000: `npm start`
- Check the context files: `npm run lint:context`
- Regenerate the tool specific context files: `npm run sync:context`

## Conventions that are not obvious from the code

- The test runner is the Node built in runner invoked through `node --test`, so do not add Jest,
  Vitest, or any other test dependency to this repository.
- This project stays dependency-free on purpose, so solve problems with the Node standard library
  rather than by adding a package.
- Handlers in `src/api/` return `{ data }` or `{ error: { code, message } }` and never choose an
  HTTP status, because `src/router.js` owns the mapping from error code to status.
- Handlers never touch the store directly, so any logic that reads or writes tasks belongs in
  `src/services/tasks.js`.
- The store is module level state that survives between test cases, so any test file that creates
  a task has to call `resetTasks()` in a `beforeEach` hook.

## Definition of done

Run `npm test` and `npm run lint:context` before you report a task as finished, and paste the
output rather than asserting that it passed.

## Where to look

- Architecture and request flow: `docs/architecture.md`
- Testing conventions and how to add a case: `docs/testing.md`
- Why the store is in memory: `docs/decisions/0001-in-memory-store.md`
- Rules that apply only to the API layer: `src/AGENTS.md`

Notice what each section is doing. The commands are there because an agent can't reliably guess your script names, and guessing wrong costs a failed run. The conventions are all things that are either invisible from reading the code or actively contradicted by what the model would otherwise assume, and each one states the reason, since a rule with a reason attached survives situations the rule author didn't anticipate. The last section is nothing but paths, which is the on demand layer doing its job.

Rough guidance on what earns its place:

Getting the Altitude Right

There's a second way to write a bad rule, which is to pitch it at the wrong level of specificity. Anthropic's guidance frames this as finding the right altitude, sitting between hardcoded logic that shatters on the first case it didn't anticipate, and vague encouragement that gives the model nothing to act on.

Too rigid, and it breaks on the first handler that does not fit:
- Every route handler must be exactly 40 lines and call validate() on line 3.

Too vague, and it changes nothing about what the agent does:
- Write clean, maintainable code.

Right altitude:
- Route handlers parse and validate input, then delegate to a function in `src/services/`.
  Handlers do not touch the store directly. See `src/api/tasks.js` for the pattern to copy.

The third version tells the agent the shape of the rule, the boundary it must not cross, and where to find a worked example, which is roughly what you would tell a competent new hire on their first day.

Scoping Rules to a Directory

Anything that only matters inside one part of the tree belongs in a nested file, and the test for whether a rule qualifies is simple: would a developer working in a different directory ever need to know this? If not, move it down.

<!-- src/AGENTS.md -->
# Source layer

Rules below apply to everything under `src/`, and they sit on top of the root `AGENTS.md` rather
than replacing it.

## Adding an endpoint

1. Add the handler to `src/api/tasks.js` following the shape the neighbouring handlers use.
2. Add one entry to the `routes` array in `src/router.js` with its success status.
3. Add a case to `tests/api.test.js` that covers the success path and the failure path.

## Validation

Validators live in `src/lib/validate.js`, they return an array of problem strings rather than
throwing, and they report every failing field instead of stopping at the first one, so a caller can
show the user all of their mistakes at once.

That validation rule is a good example of something worth writing down, because the code alone doesn't explain itself. An agent reading src/lib/validate.js sees a function returning an array and has no way to know whether that's a deliberate convention or an accident of one implementation, so it might reasonably throw an exception in the next validator it writes:

// src/lib/validate.js
export function validateTaskInput(input) {
  if (typeof input !== 'object' || input === null || Array.isArray(input)) {
    return ['body must be a JSON object'];
  }

  const problems = [];

  if (typeof input.title !== 'string' || input.title.trim() === '') {
    problems.push('title is required and must be a non-empty string');
  } else if (input.title.length > TITLE_MAX) {
    problems.push(`title must be ${TITLE_MAX} characters or fewer`);
  }

  if (input.done !== undefined && typeof input.done !== 'boolean') {
    problems.push('done must be a boolean when present');
  }

  return problems;
}

Pointing Instead of Inlining

The Where to look section of the root file is the cheapest thing in this whole setup. Four lines of paths cost almost nothing to load, and behind them sit several thousand tokens of architecture notes, testing conventions, and decision records that the agent pulls in only when a task needs them.

Architecture decision records are the natural home for the reasoning that would otherwise bloat your root file. The companion repository has one explaining why the task store is a plain Map rather than a database, and its most useful paragraph is the last one:

An agent working here should not add a database, an ORM, or a persistence layer unless the task
explicitly asks for one, and should treat the missing persistence as a deliberate choice rather than
a gap to fill.

Without that, an agent asked to "make the API production ready" will helpfully add Postgres. With it, the agent knows the absence is intentional and asks before changing it. That sentence costs you nothing until the day it saves you an afternoon.

The same logic applies to workflows that only come up occasionally. A step by step procedure for adding an endpoint is genuinely useful, and it would be dead weight in a file loaded on every task, so it lives in a skill file that loads when someone actually asks for an endpoint:

---
name: add-endpoint
description: Add a new endpoint to the task API following the layering this repository uses
---

# Add an endpoint

This workflow loads only when someone asks for a new endpoint, which is why it lives here instead
of in `AGENTS.md` where every session would pay for it.

Read `docs/architecture.md` first if you have not already, then work through these steps in order.

1. Decide which layer owns the new behaviour. Anything that reads or writes tasks belongs in
   `src/services/tasks.js`, and anything about request shape belongs in `src/api/tasks.js`.
2. Add or extend a validator in `src/lib/validate.js` if the endpoint accepts input, returning an
   array of problem strings so the handler can report every failure at once.
3. Add the handler to `src/api/tasks.js`, returning `{ data }` on success and
   `{ error: { code, message } }` on failure, and using an existing error code where one fits.
4. Register the route in the `routes` array in `src/router.js` with the success status it should
   return, and add the error code to `STATUS_BY_ERROR_CODE` if you introduced a new one.
5. Add at least one success case and one failure case to `tests/api.test.js`.
6. Run `npm test` and `npm run lint:context`, then paste both outputs into your summary.

Do not add a dependency, do not introduce a persistence layer, and do not set a status code inside
a handler.

Making Context Files Verifiable

Everything so far is fairly standard advice, and on its own it has a short shelf life. Context files rot for exactly the same reason documentation rots, which is that nothing breaks when they're wrong. You rename src/services/task.js to src/services/tasks.js, and your context file keeps confidently pointing at a path that no longer exists. You delete the typecheck script, and six months later an agent burns two turns trying to run it. Nobody notices either of those, because nothing in your pipeline is checking.

So put a check in the pipeline and let it fail. The companion repository has a linter in scripts/context-lint.mjs that runs four checks, and it's about 150 lines of dependency-free JavaScript that you can adapt to your own repository in an afternoon.

The first check is a token budget on every file that loads at startup:

// Loaded at the start of every session whether the task needs them or not. When one of these keeps
// pushing against its ceiling, move the detail into docs/ and leave a path behind.
const ALWAYS_LOADED = [
  { path: 'AGENTS.md', budget: 800 },
  { path: 'CLAUDE.md', budget: 300 },
  { path: '.github/copilot-instructions.md', budget: 900 },
  { path: 'src/AGENTS.md', budget: 400 },
];

// Rough average for English prose. Precision is not the point, catching a file that doubled is.
const CHARS_PER_TOKEN = 4;

const estimateTokens = (text) => Math.ceil(text.length / CHARS_PER_TOKEN);

Four characters per token is an approximation rather than a real tokenizer count, and it runs a little optimistic on code heavy files. This is fine because the number you care about is the ceiling. A file creeping from 400 tokens to 800 is the signal, and being off by 8% on the absolute figure changes nothing about how you respond to it.

The second and third checks read your context files as prose and verify that the things they mention are real. Anything in single backticks that looks like a path has to exist on disk, and any npm script has to exist in package.json:

// Fenced blocks are stripped first so an example inside a snippet is never read as a real reference.
function inlineCodeSpans(text) {
  const prose = text.replace(/```[\s\S]*?```/g, '');
  return [...prose.matchAll(/`([^`\n]+)`/g)].map((match) => match[1].trim());
}

for (const span of spans) {
  if (looksLikePath(span)) {
    if (!existsSync(join(ROOT, span))) {
      problems.push(`${file} points at a path that does not exist: ${span}`);
    }
    continue;
  }

  const script = span.match(/^npm run ([\w:-]+)$/) ?? span.match(/^npm (test|start)$/);

  if (script && !scripts.includes(script[1])) {
    problems.push(`${file} mentions an npm script that is not in package.json: ${span}`);
  }
}

Stripping fenced code blocks before scanning matters more than it looks, since your documentation is full of illustrative examples that were never meant to be real references, and a linter that fails on those gets switched off within a week.

The fourth check reruns the sync script in a dry run mode and fails if any generated file no longer matches AGENTS.md, which catches the teammate who edited CLAUDE.md directly despite the banner.

On a healthy repository the whole thing takes well under a second:

The interesting output is what happens when something rots. Adding one plausible looking line to AGENTS.md that mentions a script that was deleted and a file that was renamed produces this:

The script exits with a non-zero status, so wiring it into CI takes four lines and means the files can't drift quietly:

# .github/workflows/ci.yml
      - name: Run the test suite
        run: npm test

      # The context files are checked on every pull request, which is what stops them from
      # drifting away from the code they describe.
      - name: Check the context files
        run: npm run lint:context

This is the part I would keep if I had to throw away everything else in this tutorial. A mediocre context file that's verifiably true beats a beautifully written one that describes last year's architecture, because the agent has no way to tell the difference and will act on both with equal confidence.

Give the Agent Something to Verify Against

There's one more line in that root file worth dwelling on, and it's the definition of done.

An agent stops when the work looks finished, and without a check it can run for itself, "looks finished" is the only signal available to it, which quietly makes you the verification loop. Every mistake then waits for you to notice it.

Naming a command that returns a pass or a fail converts that into something the agent can act on by itself, so it writes the code, runs the check, reads the result, and keeps going until the check passes.

That's why Run npm test and npm run lint:context before you report a task as finished does more for output quality than any amount of style guidance you could write. Asking the agent to paste the output rather than assert success matters too, since reviewing evidence takes you a few seconds and re-running the verification yourself takes minutes.

Instructions in a context file are advice, though, and advice gets lost as the context fills. When something must happen every single time without exception, use a hook, which runs a script at a fixed point in the agent's loop and can't be talked out of it:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "npm run lint:context --silent"
          }
        ]
      }
    ]
  }
}

The rule of thumb is that anything advisory belongs in prose, and anything mandatory belongs in a hook or in CI.

Checking Whether it Actually Worked

You shouldn't take any of this on faith, and there's a cheap way to test it on your own repository.

Pick a task with an obviously correct shape, write the prompt down so it stays identical across runs, and run it twice: once on your current branch, and once on a branch where you have deleted the context files. In the companion repository a good candidate is "add a GET /tasks/count endpoint that returns the number of open tasks, with tests".

Then compare the two runs on four points. Did the tests pass without you intervening? How many corrections did you have to make? Did the code follow the existing layering, or did it reach into the store from the handler? Did any new dependency appear?

This is a sample of one rather than a benchmark, and you should treat it as such. But it's enough to tell you whether your files are pulling their weight, and it makes it very obvious which specific rule was missing when something goes wrong.

Keeping the Files Healthy

Treat these files the way you treat code, which means reviewing them when something breaks rather than on a schedule.

Two diagnostics will cover most of the situations you run into. If the agent keeps violating a rule that's written down, the file is almost certainly too long and the rule is getting lost in the noise. Prune aggressively rather than adding emphasis.

If the agent asks you a question that the file already answers, the wording is ambiguous, so rewrite that line rather than adding a second one next to it.

Beyond that, delete any rule the agent already follows without being told, since the model's defaults improve with every release and a rule that was necessary last year may be dead weight now.

Watch the token budget in the linter output as a rough health metric, because a file that keeps creeping toward its ceiling is telling you that detail needs to move into docs/.

Mistakes Worth Avoiding

The most common failure is the kitchen sink file, where every convention anyone ever mentioned gets appended until the file is three thousand tokens and the agent follows roughly half of it. The fix is the removal test applied without sentiment.

The second is duplicating your README into your context file, which doubles the cost of every session while adding nothing, since the two documents have different audiences and the agent can read the README when it needs to.

The third is documenting things the model can see for itself. The giveaway is any line that describes what a file contains rather than what you expect an agent to do about it.

The fourth is writing rules that can't be verified, such as asking for readable code or good performance, which sound reasonable and give the agent no way to tell whether it has complied.

The fifth, and the one that gets teams eventually, is letting each tool keep its own hand-maintained copy. They start out identical, they diverge within a month, and then Cursor and Claude Code are working from contradictory instructions in the same repository. Generate the copies, and check the generation in CI.

Where to Start

If you only do one thing after reading this, run a token estimate on the context file you already have, and then read it line by line asking whether removing each line would cause a mistake. Most people cut somewhere between a third and a half of the file on the first pass, and notice the agent following the remainder more reliably.

After that, add the pointers so your documentation becomes reachable without being expensive, and put the linter in CI so the whole thing stays honest as the codebase moves underneath it.

The full setup, including the linter, the sync script, the hook, and the CI workflow, is at github.com/Adeniyikayodee/MCF. Clone it, run npm run lint:context to watch it pass, then break something in AGENTS.md and watch it fail.

You can adapt the linter to your own conventions rather than copying it verbatim, since the checks worth running are the ones that match the ways your particular repository tends to drift.

Fork the repository if you want your own copy to experiment in, since a fork gives you a branch point you can modify freely without losing the ability to pull later changes back in. If you would rather be told when those changes land, use the Watch button next to Fork and choose releases or all activity, because that's the control that actually sends you notifications while a fork only captures the code as it stands on the day you take it.

Further Reading

• Effective context engineering for AI agents, Anthropic

• Best practices for Claude Code, Anthropic

• The AGENTS.md convention

In some cases, converting a document to grayscale reduces distractions, creates printer-friendly versions, lowers printing costs, or prepares files for black-and-white publishing.

A PDF to Grayscale Converter automates this process. Instead of editing every page manually, users can upload a PDF, choose how the grayscale conversion should be applied, preview the results, and download a newly generated document, all from within the browser.

In this tutorial, you'll build a browser-based PDF to Grayscale Converter using JavaScript. Users will be able to upload a PDF, preview every page, adjust the grayscale intensity, choose between multiple conversion modes, select which pages to process, generate a grayscale PDF, preview the final result, rename the output file, and download it without uploading their document to an external server.

We'll use PDF.js to render PDF pages, the HTML Canvas API to manipulate image pixels, and PDF-lib to generate the final downloadable PDF.

By the end of this tutorial, you'll have a complete client-side PDF processing application similar to the one available on All In One Tools.

Table of Contents

• What This PDF to Grayscale Converter Does and How It Works

• Project Setup

• Libraries Used

• Creating the HTML Layout

• Uploading and Previewing PDFs

• Building the Conversion Settings

• Converting PDF Pages to Grayscale

• Generating the Final PDF

• Previewing the Result

• Renaming and Downloading

• Demo: How the PDF to Grayscale Converter Works

• Performance Tips

• Common Mistakes

• Conclusion

What This PDF to Grayscale Converter Does and How It Works

A PDF to Grayscale Converter transforms colorful PDF pages into shades of gray while preserving the document's layout, page dimensions, text placement, and images. Instead of removing content, it recalculates the color of every pixel so the entire page appears in grayscale.

This is useful for creating printer-friendly documents, reducing color distractions, preparing files for monochrome printing, improving consistency across scanned documents, or producing black-and-white versions for review and archival purposes.

In this project, users can upload a PDF, browse through every page, adjust the grayscale intensity, choose between different conversion modes, decide whether all pages or only selected pages should be converted, generate a new grayscale PDF, preview the completed document, rename the output file, and download it directly from the browser.

Behind the scenes, PDF.js renders each PDF page onto an HTML canvas. Once the page has been rendered, JavaScript reads the RGB values for every pixel and calculates a grayscale value using a luminance formula. The updated pixels are written back to the canvas before PDF-lib assembles all processed pages into a brand-new PDF.

A typical pixel contains four values:

const pixel = {
    red: 180,
    green: 95,
    blue: 40,
    alpha: 255
};

To convert that pixel into grayscale, JavaScript calculates a single luminance value and applies it equally to the red, green, and blue channels.

const gray = 0.299 * red + 0.587 * green + 0.114 * blue;

The resulting pixel becomes:

pixel.red = gray;
pixel.green = gray;
pixel.blue = gray;

Repeating this process for every pixel on every selected page creates a new grayscale version of the original PDF while preserving the overall structure of the document.

Project Setup

Before writing the conversion logic, let's create a simple project structure.

We'll build everything using HTML, CSS, and JavaScript, together with PDF.js, the Canvas API, and PDF-lib.

Our project structure looks like this:

pdf-to-grayscale/
│── index.html
│── style.css
│── script.js
│── pdf.worker.min.js
│── assets/

Keeping the HTML, styling, and JavaScript separate makes the project easier to maintain as more PDF features are added.

Libraries Used

The PDF to Grayscale Converter relies on three browser technologies that work together to render PDF pages, process image pixels, and generate a new downloadable document.

PDF.js renders PDF pages directly inside the browser.

The HTML Canvas API provides access to every pixel so JavaScript can convert colors into grayscale.

PDF-lib creates the final PDF after all selected pages have been processed.

Include the required libraries before loading your application.

<script src="https://cdnjs.cloudflare.com/ajax/libs/pdf.js/4.4.168/pdf.min.js"></script>
<script src="https://unpkg.com/pdf-lib/dist/pdf-lib.min.js"></script>
<script src="script.js"></script>

Configure the PDF.js worker.

pdfjsLib.GlobalWorkerOptions.workerSrc = "pdf.worker.min.js";

Using a worker allows PDF rendering to happen in the background without freezing the browser interface.

Creating the HTML Layout

The application is divided into four main sections:

• Upload area

• PDF preview

• Conversion settings

• Download section

Create the basic page structure.

<section id="uploadSection"></section>
<section id="previewSection" hidden></section>
<section id="settingsSection" hidden></section>
<section id="downloadSection" hidden></section>

Only the upload area is visible when the page first loads. The remaining sections appear after a PDF has been successfully opened.

Selecting the Main Elements

Store references to the elements that will be used throughout the application.

const uploadSection = document.getElementById("uploadSection");
const previewSection = document.getElementById("previewSection");
const settingsSection = document.getElementById("settingsSection");
const pdfCanvas = document.getElementById("pdfCanvas");

Using these references makes it easier to update the interface as users move through the conversion process.

Uploading and Previewing PDFs

The upload area accepts both drag-and-drop and manual file selection.

When a file is selected, first verify that it's a PDF.

async function uploadPdf(file) {
    if (!file || file.type !== "application/pdf") {
        alert("Please select a PDF file.");
        return;
    }

    await loadPdf(file);
}

Once validation succeeds, the document is loaded into memory for processing.

Loading the PDF

Convert the uploaded file into an ArrayBuffer before opening it with PDF.js.

async function loadPdf(file) {
    const bytes = await file.arrayBuffer();

    pdfDocument = await pdfjsLib.getDocument({
        data: bytes
    }).promise;

    currentPage = 1;
    renderPage(currentPage);
}

The loaded document is stored so every page can later be converted to grayscale.

Rendering PDF Pages

PDF.js renders one page at a time onto an HTML canvas.

Retrieve the page.

const page = await pdfDocument.getPage(currentPage);

Create the viewport.

const viewport = page.getViewport({
    scale: 1.5
});

Resize the canvas.

pdfCanvas.width = viewport.width;
pdfCanvas.height = viewport.height;

Render the page.

await page.render({
    canvasContext: pdfCanvas.getContext("2d"),
    viewport
}).promise;

Once rendering finishes, the selected page appears inside the preview area.

Navigating Between Pages

Most PDF documents contain multiple pages, so users need simple navigation controls.

Track the current page.

let currentPage = 1;
let pdfDocument = null;

Move to the previous page.

previousButton.addEventListener("click", async () => {
    if (currentPage > 1) {
        currentPage--;
        await renderPage(currentPage);
    }
});

Move to the next page.

nextButton.addEventListener("click", async () => {
    if (currentPage < pdfDocument.numPages) {
        currentPage++;
        await renderPage(currentPage);
    }
});

Update the page indicator.

pageCounter.textContent = `Page ${currentPage} of ${pdfDocument.numPages}`;

Users can now browse through the document before deciding how the grayscale conversion should be applied.

Building the Conversion Settings

After the PDF has been loaded and previewed, users can configure how the document should be converted to grayscale. The settings panel allows users to adjust the grayscale intensity, choose a conversion mode, decide which pages should be processed, and start the conversion.

Adjusting Grayscale Intensity

The intensity slider controls how strongly the grayscale effect is applied.

Lower values retain more of the original color, while higher values produce a true grayscale appearance.

Create the slider.

<input type="range" id="grayIntensity" min="0" max="100" value="100">

Read the selected value.

const intensity = Number(document.getElementById("grayIntensity").value);

The selected intensity will later be used when calculating the final grayscale color.

Choosing the Conversion Mode

The tool provides multiple grayscale modes for different use cases.

Create the radio buttons.

<input type="radio" name="mode" value="standard" checked>
Standard Grayscale
<input type="radio" name="mode" value="threshold">
Black & White
<input type="radio" name="mode" value="soft">
Soft Gray

Retrieve the selected mode.

const conversionMode = document.querySelector(
    'input[name="mode"]:checked'
).value;

Each mode uses a different algorithm when processing the canvas pixels.

Selecting the Pages

Users can convert either the entire document or only selected pages.

Create the page selection controls.

<input type="radio" name="pages" value="all" checked>
All Pages
<input type="radio" name="pages" value="custom">
Specific Pages
<input type="text" id="pageRange" placeholder="e.g., 1, 3-5, 10">

Read the selected option.

const applyMode = document.querySelector(
    'input[name="pages"]:checked'
).value;

Retrieve the custom page range.

const pageRange = document.getElementById("pageRange").value.trim();

When All Pages is selected, every page in the PDF is processed. Otherwise, only the pages specified by the user are converted.

Converting PDF Pages to Grayscale

Once the settings have been configured, users can begin the conversion.

Create the action button.

<button id="convertPdf">Convert to Grayscale</button>

Start the conversion.

convertButton.addEventListener("click", async () => {
    await convertPdf();
});

Starting Over

Users can clear the current document and return the application to its initial state.

Create the reset button.

<button id="resetTool">Start Over</button>

Reset the application.

resetTool.addEventListener("click", () => {
    location.reload();
});

This removes the current PDF and restores the default settings so another document can be processed.

Reading Canvas Pixels

After a page has been rendered, retrieve its pixel data.

const imageData = context.getImageData(
    0,
    0,
    canvas.width,
    canvas.height
);

The pixel information is stored in an array.

const pixels = imageData.data;

Each pixel contains four values:

• Red

• Green

• Blue

• Alpha

We'll update the RGB values while leaving the alpha channel unchanged.

Converting Colors to Grayscale

The standard grayscale algorithm calculates a luminance value using the red, green, and blue channels.

Loop through every pixel.

for (let i = 0; i < pixels.length; i += 4) {
    const gray =
        0.299 * pixels[i] +
        0.587 * pixels[i + 1] +
        0.114 * pixels[i + 2];

    pixels[i] = gray;
    pixels[i + 1] = gray;
    pixels[i + 2] = gray;
}

This formula produces a natural-looking grayscale image because it reflects how the human eye perceives brightness.

Applying the Selected Conversion Mode

Different conversion modes use different pixel calculations.

For example, the Black & White (Threshold) mode converts each pixel into either pure black or pure white.

const threshold = 128;
const color = gray >= threshold ? 255 : 0;
pixels[i] = color;
pixels[i + 1] = color;
pixels[i + 2] = color;

The Soft Gray mode blends the original color with the grayscale value to create a less aggressive effect.

const softGray =
    (gray * 0.6) +
    (pixels[i] * 0.4);

pixels[i] = softGray;
pixels[i + 1] = softGray;
pixels[i + 2] = softGray;

Once the selected mode has been applied, write the updated pixels back to the canvas.

context.putImageData(
    imageData,
    0,
    0
);

Processing the Selected Pages

Instead of processing the entire document every time, convert only the pages selected by the user.

Loop through the page range.

for (let page = startPage; page <= endPage; page++) {
    await processPage(page);
}

Each processed page is temporarily stored before the final PDF is created.

Generating the Final PDF

Create a new PDF document.

const outputPdf = await PDFLib.PDFDocument.create();

Convert the processed canvas into an image.

const imageBytes = await canvasToBytes(pdfCanvas);

Embed the image into the PDF.

const image = await outputPdf.embedPng(imageBytes);

Create a new page.

const page = outputPdf.addPage([
    image.width,
    image.height
]);

Draw the processed image.

page.drawImage(image, {
    x: 0,
    y: 0,
    width: image.width,
    height: image.height
});

Repeat this process for every selected page until the new grayscale document is complete.

Saving the Generated PDF

After all pages have been processed, save the completed document.

const pdfBytes = await outputPdf.save();

Create a downloadable PDF file.

generatedPdfBlob = new Blob([pdfBytes], {
    type: "application/pdf"
});

The grayscale PDF is now ready for preview, renaming, and downloading.

Previewing the Result

Before downloading the converted document, it's useful to let users review the final output. This allows them to verify that the selected pages have been converted correctly and that the grayscale appearance meets their expectations.

Load the generated PDF using PDF.js.

let finalPdf = null;

async function showResult() {
    const bytes = await generatedPdfBlob.arrayBuffer();

    finalPdf = await pdfjsLib.getDocument({
        data: bytes
    }).promise;

    renderResultPage(1);
}

Render the selected page.

async function renderResultPage(pageNumber) {
    const page = await finalPdf.getPage(pageNumber);

    const viewport = page.getViewport({
        scale: 1.5
    });

    resultCanvas.width = viewport.width;
    resultCanvas.height = viewport.height;

    await page.render({
        canvasContext: resultCanvas.getContext("2d"),
        viewport
    }).promise;
}

Users can browse through every converted page before downloading the PDF.

Renaming and Downloading

Before saving the converted PDF, users can provide a custom filename.

Create the filename input.

<input
    type="text"
    id="outputFilename"
    value="grayscale-document.pdf"
>

Retrieve the filename.

function getFilename() {
    let filename = outputFilename.value.trim();

    if (!filename) {
        filename = "grayscale-document.pdf";
    }

    if (!filename.toLowerCase().endsWith(".pdf")) {
        filename += ".pdf";
    }

    return filename;
}

Display information about the generated PDF.

pageCount.textContent = `${finalPdf.numPages} Pages`;
fileSize.textContent = formatFileSize(generatedPdfBlob.size);

Download the completed PDF.

downloadButton.addEventListener("click", () => {
    const url = URL.createObjectURL(generatedPdfBlob);

    const link = document.createElement("a");
    link.href = url;
    link.download = getFilename();
    link.click();

    URL.revokeObjectURL(url);
});

Everything happens locally inside the browser, allowing users to keep their documents private throughout the conversion process.

Demo: How the PDF to Grayscale Converter Works

Let's walk through the complete workflow.

Step 1: Upload the PDF

Users begin by dragging a PDF into the upload area or clicking Select PDF.

Step 2: Preview the Document

PDF.js renders the uploaded document page by page, allowing users to review the file before conversion.

Step 3: Configure the Conversion

Users adjust the grayscale intensity, choose a conversion mode, and decide whether to process all pages or only selected pages.

Step 4: Convert the PDF

Click Convert to Grayscale to begin processing the selected pages.

Step 5: Review the Converted Document

After processing is complete, the application displays a preview of the generated grayscale PDF.

Step 6: Rename and Download

Finally, users can rename the output file, review the page count and file size, and download the converted PDF.

Performance Tips

Large PDF files require more processing time because every page must be rendered and converted. Processing only the selected pages helps improve performance.

for (let page = startPage; page <= endPage; page++) {
    await processPage(page);
}

After downloading the file, release temporary resources to reduce memory usage.

URL.revokeObjectURL(downloadUrl);

These simple optimizations help keep the converter responsive when working with large multi-page PDF documents.

Common Mistakes

One common mistake is converting the same page multiple times without first rendering the original page again. Always start with the original PDF page before applying another grayscale conversion.

await renderPage(currentPage);

Another issue is allowing users to specify invalid page numbers.

if (pageNumber < 1 || pageNumber > pdfDocument.numPages) {
    return;
}

Finally, remember that higher output quality usually produces larger PDF files. Choosing the appropriate quality setting helps balance image clarity and file size.

Conclusion

In this tutorial, you built a browser-based PDF to Grayscale Converter using JavaScript.

You learned how to upload and preview PDF documents, render pages with PDF.js, convert colorful pages into grayscale using the Canvas API, process selected pages, generate a new PDF with PDF-lib, preview the completed document, rename the output file, and download it directly from the browser.

Because all processing happens locally, users can convert PDF documents to grayscale without uploading sensitive files to an external server.

You can explore the complete workflow using the PDF to Grayscale Converter.

From here, you can extend the project with additional features such as sepia conversion, brightness and contrast controls, custom grayscale presets, batch PDF processing, or support for additional image filters.

That means collecting a business type, pulling in registration documents, and showing the user where their verification stands, all without making onboarding feel like a customs form.

This article breaks down a working three step KYC (Know Your Customer) flow built with Shadcn UI: a stepper for progress, a radio group for account type, a file upload zone for documents, and an alert for verification status. You'll see the actual component code, not a simplified stand-in, along with the reasoning behind each decision.

You can try the finished flow at onboarding-kyc-flow.vercel.app. Click through it once before reading on, as it makes the code below easier to follow. And it also comes in dark and light mode.

Table of Contents

• Prerequisites

• What You're Building

• Project Structure

• Radix UI vs Base UI: Which Primitives this Flow Uses

• Scaffolding the Flow with v0 and an MCP Server

• Step 1: Account Type with a Radio Group

• Step 2: Document Upload with Drag and Drop

• Step 3: Verification Status with an Alert

• Adding a Stepper to the Flow

• Small Details that Make it Feel Finished

• Accessibility notes

• Key Concepts Recap

• Conclusion

• Resources

Prerequisites

Before working through this flow, you should be comfortable with React function components and hooks, specifically useState, useRef, and useEffect.

You should have:

• A Next.js project with the App Router and shadcn/ui already initialized, since this article doesn't cover that initial setup.

• A v0 account is optional. You can also use Bolt or Lovable, which support the same shadcn MCP prompt feature.

What You're Building

The flow has three steps:

1. Account type: The user picks Startup, Enterprise, or Government. This decision drives the rest of the experience. It's shown back to the user as a confirmation line, and would typically decide which workspace defaults get applied.

2. Document upload: The user drags in a business registration document, a tax return, or a company registry export, in PDF or CSV format.

3. Verification status: The user sees a live status: checking in progress, then either verified or an issue that needs attention.

Project Structure

The project is a standard Next.js app with shadcn/ui already initialized. Here's the top-level layout:

onboarding-kyc-flow/
├── .vercel/
├── app/
├── components/
├── lib/
├── public/
├── .env.development.local
├── .gitignore
├── components.json
├── next-env.d.ts
├── next.config.mjs
├── package.json
├── pnpm-lock.yaml
├── postcss.config.mjs
├── tsconfig.json
└── tsconfig.tsbuildinfo

components.json is the file the shadcn CLI reads to know where your components live and which style and primitives you're using. components/ holds the shared UI pieces (Alert, Badge, Button, Card, Progress, RadioGroup, Separator) that the flow is built from. lib/utils.ts provides the cn helper used throughout the flow to combine conditional class names. app/ holds the page itself, shown in full below.

Radix UI vs Base UI: Which Primitives this Flow Uses

Shadcn components aren't tied to one underlying primitive library. Most of the ecosystem defaults to Radix UI, but Base UI has become a solid alternative, and it's what this flow is built on.

The underlying primitive library can affect how a component behaves and how you work with it in your project. If you're pulling components from a set like Shadcn UI, check which primitive library it targets before mixing components from different sources.

Mixing Radix-based and Base UI-based components generally works, but it means using two different unstyled primitive libraries in the same project. You can compare Radix UI and Base UI here.

Scaffolding the Flow with v0 and an MCP Server

An MCP (Model Context Protocol) server exposes a component library to an AI coding assistant as a set of callable tools. Instead of the assistant guessing at component names and props from training data, it queries the server for the real, current API.

This matters here specifically, since there are now several shadcn-style component sets with similar names and different props.

The Shadcn Components library publishes an MCP server for its free set, connected to v0 by following its getting started guide. The video below covers the connection step by step. The same generated output can also be copied into Lovable or Bolt through their copy prompt feature, so the workflow isn't locked to one AI builder.

The prompt used to scaffold this flow looked like this:

Create an Enterprise SaaS Onboarding & KYC Flow. Use free components of the shadcn space MCP server: shadcn alert, shadcn radio group, shadcn stepper, shadcn file upload. Only use free components, not pro ones, and list which free component was used for each part.

Step 1: Account Type (stepper) - radio group for Startup, Enterprise, or Government

Step 2: Upload Documents (stepper) - file upload for a business registration document

Step 3: Verification (stepper) - alert showing verification status

This produces a working first draft fast. What follows is the result after cleaning that draft up: real state management, real validation, and states that a generated draft tends to skip.

Step 1: Account Type with a Radio Group

Account type is the first decision in the flow because it's the one most likely to affect what comes after it. Asking it early keeps the rest of the flow feeling relevant to the choice the user just made.

const tiers: { id: Tier; name: string; description: string; tag: string }[] = [
  { id: 'startup', name: 'Startup', description: 'For teams building and scaling fast', tag: 'Up to 25 seats' },
  { id: 'enterprise', name: 'Enterprise', description: 'For established teams with advanced needs', tag: 'Unlimited seats' },
  { id: 'government', name: 'Government', description: 'For public sector and regulated teams', tag: 'FedRAMP-ready' },
]

<RadioGroup value={tier} onValueChange={(value) => setTier(value as Tier)} className="grid gap-3">
  <fieldset className="contents">
    <legend className="sr-only">Account type</legend>
    {tiers.map((item) => (
      <label
        key={item.id}
        htmlFor={item.id}
        className={cn(
          'flex cursor-pointer items-start gap-4 rounded-xl border p-4 transition-colors hover:border-primary/50',
          tier === item.id && 'border-primary bg-primary/5'
        )}
      >
        <RadioGroupItem value={item.id} id={item.id} className="mt-0.5" />
        <span className="flex flex-1 flex-col gap-1">
          <span className="flex flex-wrap items-center gap-2 text-sm font-semibold">
            {item.name}
            {item.id === 'enterprise' && <Badge variant="secondary">Recommended</Badge>}
          </span>
          <span className="text-sm text-muted-foreground">{item.description}</span>
          <span className="mt-1 font-mono text-[11px] uppercase tracking-wide text-muted-foreground">{item.tag}</span>
        </span>
      </label>
    ))}
  </fieldset>
</RadioGroup>

Two things worth noticing here. The tier data lives in a plain array outside the component, so adding a fourth tier later is a one-line change, not a markup change. And the fieldset with a visually hidden (sr-only) legend groups the three options as one related choice for screen readers. Sighted users never see it, since the card title above already states "Choose your account type" visually.

This step uses a shadcn radio group rather than a select or checkboxes, since account type is a single, mutually exclusive choice, and a radio group is the only one of the three that makes both the options and the current selection visible at a glance.

Live Preview:

Step 2: Document Upload with Drag and Drop

The upload zone needs to handle three states cleanly: nothing selected yet, a file selected and ready, and a rejected file with a specific reason why.

function FileUpload({ file, onFile, onRemove, error }: {
  file: File | null
  onFile: (file: File) => void
  onRemove: () => void
  error: string
}) {
  const inputRef = useRef<HTMLInputElement>(null)
  const [dragging, setDragging] = useState(false)

  const accept = (candidate: File) => {
    if (candidate.type !== 'application/pdf' && candidate.type !== 'text/csv' && !candidate.name.toLowerCase().endsWith('.csv')) {
      return 'Upload a PDF or CSV file only.'
    }
    if (candidate.size > 10 * 1024 * 1024) {
      return 'Files must be smaller than 10 MB.'
    }
    onFile(candidate)
    return ''
  }

  return (
    <div className="flex flex-col gap-3">
      {!file ? (
        <button
          type="button"
          className={cn(
            'group flex min-h-44 flex-col items-center justify-center rounded-xl border border-dashed bg-muted/30 px-6 text-center transition-colors hover:border-primary hover:bg-primary/5',
            dragging && 'border-primary bg-primary/10'
          )}
          onClick={() => inputRef.current?.click()}
          onDragOver={(event) => { event.preventDefault(); setDragging(true) }}
          onDragLeave={() => setDragging(false)}
          onDrop={(event) => {
            event.preventDefault()
            setDragging(false)
            const dropped = event.dataTransfer.files[0]
            if (dropped) accept(dropped)
          }}
        >
          <input
            ref={inputRef}
            className="sr-only"
            type="file"
            accept=".pdf,.csv,application/pdf,text/csv"
            onChange={(event) => {
              const selected = event.target.files?.[0]
              if (selected) accept(selected)
            }}
          />
          <span className="mb-3 flex size-11 items-center justify-center rounded-lg border bg-background text-primary shadow-sm">
            <UploadCloud className="size-5" aria-hidden="true" />
          </span>
          <span className="text-sm font-semibold">Drop your business document here</span>
          <span className="mt-1 text-xs text-muted-foreground">or click to browse · PDF or CSV · max 10 MB</span>
        </button>
      ) : (
        <div className="flex items-center gap-3 rounded-xl border bg-muted/30 p-4">
          <span className="flex size-10 items-center justify-center rounded-lg bg-primary/10 text-primary">
            <FileText className="size-5" aria-hidden="true" />
          </span>
          <div className="min-w-0 flex-1">
            <p className="truncate text-sm font-semibold">{file.name}</p>
            <p className="text-xs text-muted-foreground">{(file.size / 1024 / 1024).toFixed(2)} MB · Ready to verify</p>
          </div>
          <Badge variant="secondary" className="hidden sm:inline-flex">Uploaded</Badge>
          <Button type="button" variant="ghost" size="icon-sm" aria-label="Remove file" onClick={onRemove}>
            <X className="size-4" aria-hidden="true" />
          </Button>
        </div>
      )}
      {error && (
        <Alert variant="destructive">
          <AlertCircle className="size-4" aria-hidden="true" />
          <AlertTitle>Unsupported document</AlertTitle>
          <AlertDescription>{error}</AlertDescription>
        </Alert>
      )}
    </div>
  )
}

The accept function is the whole validation layer, and it runs from two different places: the change handler on the hidden file input, and the drop handler on the drag zone.

Both paths call the same function, so a file dragged in gets the same validation checks as a file selected by clicking browse. It ensures that only PDF or CSV files are allowed, regardless of how the file is added.

This is where shadcn file upload earns its place over a plain <input type="file">: the drag zone, the selected state, and the rejected state are all handled as one component instead of three separate pieces wired together by hand.

Live Preview:

Step 3: Verification Status with an Alert

Verification isn't instant, so the interface needs to say clearly what's happening and what happens next, rather than showing a spinner with no explanation.

{verified ? (
  <Alert className="border-primary/30 bg-primary/5">
    <CheckCircle2 className="size-4 text-primary" aria-hidden="true" />
    <AlertTitle>Verification complete</AlertTitle>
    <AlertDescription>
      Your {selectedTier.name.toLowerCase()} workspace is ready to configure.
    </AlertDescription>
  </Alert>
) : (
  <>
    <Alert>
      <AlertCircle className="size-4" aria-hidden="true" />
      <AlertTitle>Verification in progress</AlertTitle>
      <AlertDescription>
        This usually takes a few moments. You can keep this tab open while we finish.
      </AlertDescription>
    </Alert>
    <div className="flex flex-col gap-3">
      <div className="flex items-center justify-between text-sm">
        <span className="font-medium">Checking business registry</span>
        <span className="font-mono text-xs text-muted-foreground">{checking ? '68%' : '100%'}</span>
      </div>
      <Progress value={checking ? 68 : 100} />
      <div className="flex items-center gap-2 text-xs text-muted-foreground">
        <Building2 className="size-3.5" aria-hidden="true" /> Matching company details and tax identifiers
      </div>
    </div>
  </>
)}

Pairing the shadcn alert with a progress bar does two jobs at once: the alert states the current status in words, while the progress bar gives a rough sense of how much is left, without promising a specific time. Neither one alone tells the full story, the alert alone feels static, and a progress bar alone doesn't explain what's actually being checked.

Worth adding here, and easy to skip when a demo only shows the success path: a mismatch state, where the tax ID on the document doesn't match the company registry, deserves its own alert with a clear next step: contact support or re-upload a corrected document. It's not shown above, since the flow currently resolves to either checking or verified, but it's the state a production version of this flow would hit the most.

Live Preview:

Adding a Stepper to the Flow

The stepper is the visual anchor of the whole flow. It's the piece that tells the user how much is left before the checking and account-type-selecting are done.

function Stepper({ current }: { current: Step }) {
  return (
    <nav
      aria-label="Onboarding progress"
      className="grid grid-cols-[minmax(0,1fr)_minmax(2rem,5rem)_minmax(0,1fr)_minmax(2rem,5rem)_minmax(0,1fr)] items-start gap-0"
    >
      {steps.map((step, index) => (
        <div key={step.number} className="contents">
          <div className="flex min-w-0 flex-col items-center text-center">
            <div
              className={cn(
                'flex size-9 items-center justify-center rounded-full border text-sm font-semibold transition-colors',
                current > step.number
                  ? 'border-primary bg-primary text-primary-foreground'
                  : current === step.number
                  ? 'border-primary bg-primary/10 text-primary'
                  : 'border-border bg-background text-muted-foreground'
              )}
              aria-current={current === step.number ? 'step' : undefined}
            >
              {current > step.number ? <Check className="size-4" aria-hidden="true" /> : step.number}
            </div>
            <div className="mt-2 min-w-0">
              <p className={cn('truncate text-sm font-semibold', current >= step.number ? 'text-foreground' : 'text-muted-foreground')}>
                {step.label}
              </p>
              <p className="mt-1 hidden text-xs leading-5 text-muted-foreground sm:block">{step.caption}</p>
            </div>
          </div>
          {index < steps.length - 1 && (
            <div className={cn('mt-4 h-px w-full', current > step.number ? 'bg-primary' : 'bg-border')} />
          )}
        </div>
      ))}
    </nav>
  )
}

The current > step.number check keeps the entire stepper in sync with a single comparison. It determines the circle’s fill color, decides when the step number should be replaced by a checkmark, and controls whether the connecting line to the next step is filled.

This is important because the stepper only needs one piece of state, step, from the parent component. It doesn’t need to know why the user is on step 2, it only needs to know which step is currently active and update its visual state accordingly.

The "continue" logic that actually advances step lives outside the stepper itself:

const continueStep = () => {
  if (step === 1) setStep(2)
  else if (step === 2 && file) {
    setStep(3)
    setChecking(true)
    window.setTimeout(() => {
      setChecking(false)
      setVerified(true)
    }, 1400)
  }
}

Keeping this in the page component, not inside the shadcn stepper itself, is what keeps the stepper reusable. It only renders progress. Whether the user is allowed to move forward, a file is required on step 2, or whether nothing is required on step 1, is a decision for the flow around it to make.

Small Details that Make it Feel Finished

A few things in this build are easy to skip but change how the flow feels in practice:

• A dark mode toggle in the header, wired to a darkMode state that toggles a class on document.documentElement. It's small, but it means the flow doesn't fight a user's system theme preference.

• A security note in the sidebar, stating documents are encrypted and deleted after verification. This is copy, not code, but it answers the question a corporate user is quietest about and most worried by: what happens to the file after I upload it.

• A "Selected" confirmation line under the radio group, restating the chosen tier in plain text. A small detail, but it removes any doubt about what was actually selected before moving on.

If you're looking to wrap a flow like this inside a full application shell, with navigation and a dashboard around it, the Shadcn Dashboard starter uses the same component set. It's a reasonable base to extend from rather than building a shell from scratch.

Live Preview:

https://onboarding-kyc-flow.vercel.app/

This project is open source, and you can easily download the zip and if you like. Please consider giving it a star.

• Github Repo

Accessibility notes

• The stepper's nav element has an aria-label, and the current step carries aria-current="step", so assistive technology can identify progress without relying on visual position alone.

• The radio group sits inside a fieldset with a screen-reader-only legend, grouping the three account types as one decision.

• Icons throughout (Check, AlertCircle, UploadCloud, and so on) carry aria-hidden="true", since they're decorative next to text that already states the same information. This stops screen readers from announcing redundant icon labels.

• The remove-file button has an explicit aria-label, since its visible content is an icon only, with no text.

Key Concepts Recap

• Account type comes first because it's the one decision most likely to affect the rest of the flow, and it's kept in state at the page level, not inside the radio group itself.

• File validation runs in a shared function used by both the drag-and-drop path and the click-to-browse path, so both paths apply the same PDF/CSV validation.

• Verification status is communicated with both words (the alert) and a rough sense of progress (the progress bar), since either one alone leaves out part of the picture.

• The stepper is a pure display component driven by a single step value from its parent. The logic for whether the user can advance lives outside it, not inside it.

• Small, non-technical details (like a security note, a confirmation line, or a theme toggle) do as much for how finished a flow feels as any of the four core components.

Conclusion

None of the four components in this flow are complicated individually. What makes a KYC flow work is the decisions around them: which choice comes first, where validation actually runs, and how honestly the interface talks to the user while something outside their control is being checked.

Whether the first draft comes from typing every line by hand or from scaffolding it with an MCP server and v0, that's the part worth spending time getting right before it ships.

Resources

• Shadcn Components, the free component set used in this flow

• ShadcnSpace, the base library these components extend

• MCP server walkthrough

• MDN: ARIA current attribute

• Model Context Protocol specification