The answer is multi-tenancy: an architecture where one application dynamically serves isolated experiences to many different users, often through subdomains.

In this tutorial, you'll build a multi-tenant portfolio SaaS platform from scratch using Next.js, Express, and Prisma. Each user who signs up gets their own portfolio site, served on their own subdomain — generated instantly, powered by a single backend, and stored in a single database.

Here's what you'll build:

• A landing page where users fill out a form to create their portfolio

• An Express + Prisma backend that stores each user as a "tenant"

• A Next.js middleware layer that detects subdomains and routes requests dynamically

• A JSON-driven template system that controls which sections appear on each portfolio

• A production-ready portfolio page served at name.localhost:3000 in development and name.yourdomain.com in production

You can find the complete source code in the GitHub repositories linked at the end of this tutorial.

Table of Contents

• Prerequisites

• What is Multi-Tenancy?

• How to Set Up the Backend

  • How to Install Dependencies

  • How to Configure TypeScript for ESM

  • How to Initialize Prisma

• How to Define the Prisma Schema

• How to Run Your First Migration

• How to Generate and Instantiate the Prisma Client

  • How to Generate the Client

  • How to Instantiate the Client

• How to Seed a Template

• How to Build the Express API

  • How to Install Express

  • How to Create the Server Entry Point

  • How to Create the Express App

  • How to Create the Tenant Controller

  • How to Create the Tenant Routes

  • How to Start the Server

• How to Create the Next.js Frontend

• How to Add Subdomain Routing with Middleware

• How to Build the Landing Page

  • How to Update the Layout

  • How to Create the Home Page

• How to Build the Tenant Portfolio Page

• How to Test the Full Flow

• Next Steps

• Conclusion

• Source Code

Prerequisites

Before you begin, make sure you have the following:

• Node.js (version 18 or higher) installed on your machine

• A basic understanding of React, TypeScript, and REST APIs

• Familiarity with Prisma ORM (you don't need to be an expert)

• A code editor like VS Code

You'll use Prisma Postgres as your database, so you won't need to set up a separate database server locally. Prisma handles the connection string and adapter configuration for you.

What is Multi-Tenancy?

Multi-tenancy is an architectural pattern where a single application serves multiple users — called tenants — each with isolated data and often their own URL.

Here's how the flow works in this tutorial:

1. A user visits your landing page and fills out a form with their name, bio, and skills.

2. Your Express backend creates a new tenant record in the database and generates a slug from their name.

3. The browser redirects the user to their-name.localhost:3000.

4. Your Next.js middleware detects the subdomain, extracts the slug, and rewrites the request to /tenant/their-name internally.

5. The tenant page fetches that user's data from the API and renders their portfolio.

The key insight is that the URL in the browser never changes — the rewrite is invisible to the user. One Next.js app serves every tenant dynamically.

How to Set Up the Backend

Start by creating a project folder with separate directories for the backend and frontend:

mkdir portfolio-saas && cd portfolio-saas
mkdir portfolio-api portfolio-client

Navigate into the backend directory and initialize a new Node.js project:

cd portfolio-api
npm init -y

How to Install Dependencies

Install TypeScript, Prisma, and the supporting packages:

npm install typescript tsx @types/node --save-dev
npx tsc --init
npm install prisma @types/node @types/pg --save-dev
npm install @prisma/client @prisma/adapter-pg pg dotenv

How to Configure TypeScript for ESM

Open tsconfig.json and replace its contents with:

{
  "compilerOptions": {
    "module": "ESNext",
    "moduleResolution": "bundler",
    "target": "ES2023",
    "strict": true,
    "esModuleInterop": true,
    "ignoreDeprecations": "6.0",
    "types": ["node"]
  }
}

Then open package.json and add "type": "module" to enable ESM:

{
  "type": "module"
}

How to Initialize Prisma

Run the following command to initialize Prisma and generate your schema setup:

npx prisma init --db --output ../generated/prisma

This command creates a prisma/schema.prisma file, a .env file with your database connection string, and a generated Prisma configuration folder.

How to Define the Prisma Schema

Open prisma/schema.prisma and replace its contents with the following:

generator client {
  provider = "prisma-client"
  output   = "../generated/prisma"
}

datasource db {
  provider = "postgresql"
}

model Tenant {
  id         String   @id @default(uuid())
  slug       String   @unique
  name       String
  bio        String
  skills     String[]
  templateId String
  createdAt  DateTime @default(now())
}

model Template {
  id     String @id @default(uuid())
  name   String
  config Json
}

model Post {
  id         String   @id @default(uuid())
  title      String
  content    String
  tenantSlug String
  createdAt  DateTime @default(now())
}

Let's look at what each model does.

The Tenant model represents a user who has signed up and created a portfolio. The slug field is generated from their name (for example, "John Doe" becomes john-doe) and is used as their subdomain. The templateId links each tenant to a template that controls their portfolio's layout.

The Template model stores layout configuration as JSON. Instead of hardcoding sections like "hero" or "skills" into your components, you store them in the database. This means you can add or remove sections for different templates without touching any component code.

The Post model is included for future extensibility — you can use it to let tenants publish blog posts on their portfolio.

How to Run Your First Migration

Run the following command to create your database tables based on the schema:

npx prisma migrate dev --name init

This command creates the database tables, generates a migration file, and applies the migration to your database. After it runs, your Postgres database structure matches your Prisma schema exactly.

How to Generate and Instantiate the Prisma Client

How to Generate the Client

Run this command to generate a fully type-safe Prisma Client based on your schema:

npx prisma generate

You only need to run this once after each schema change. The generated client lives in the ../generated/prisma folder you configured earlier.

How to Instantiate the Client

Create a new file at lib/prisma.ts and add the following:

import 'dotenv/config';
import { PrismaPg } from '@prisma/adapter-pg';
import { PrismaClient } from '../generated/prisma/client';

const connectionString = process.env.DATABASE_URL as string;

const adapter = new PrismaPg({ connectionString });
const prisma = new PrismaClient({ adapter });

export default prisma;

This file creates a single shared Prisma Client instance that your entire backend will import. The PrismaPg adapter connects Prisma to your Postgres database using the connection string from your .env file.

How to Seed a Template

Your platform needs at least one template in the database before any tenant can sign up. Instead of hardcoding layout decisions into your components, you'll store the template configuration as JSON and read it at runtime.

Create a new file at prisma/seed.ts and add the following:

import prisma from '../lib/prisma';

async function main() {
  await prisma.template.create({
    data: {
      name: 'minimal',
      config: {
        theme: {
          primaryColor: '#6366f1',
          background: 'dark',
        },
        sections: {
          hero: true,
          about: true,
          skills: true,
          projects: true,
          blog: true,
          contact: true,
        },
      },
    },
  });

  console.log('Template seeded successfully.');
}

main()
  .catch((e) => {
    console.error(e);
    process.exit(1);
  })
  .finally(async () => {
    await prisma.$disconnect();
  });

The config field is stored as JSON in Postgres. When a tenant's portfolio loads, your frontend reads this JSON to decide which sections to show. Setting hero: false on a template would hide the hero section for every tenant using that template — no code changes needed.

Now add a seed script to your package.json:

{
  "scripts": {
    "seed": "tsx prisma/seed.ts"
  }
}

Run it:

npm run seed

Your database now has a default template ready to attach to new tenants.

How to Build the Express API

Now you'll build the backend API that creates tenants and retrieves their data.

How to Install Express

npm install express cors
npm install -D @types/express @types/cors

How to Create the Server Entry Point

Create src/index.ts:

import app from './app';

const PORT = 8080;

app.listen(PORT, () => {
  console.log('Server is running on port 8080');
});

How to Create the Express App

Create src/app.ts:

import express from 'express';
import cors from 'cors';
import tenantRoutes from './routes/tenant.routes';

const app = express();

app.use(cors());
app.use(express.json());
app.use('/api', tenantRoutes);

export default app;

This file sets up CORS so your Next.js frontend can communicate with the API, parses JSON request bodies, and mounts all tenant routes under the /api prefix.

How to Create the Tenant Controller

Create src/controllers/tenant.controller.ts:

import { Request, Response } from 'express';
import prisma from '../../lib/prisma';

export async function createTenant(req: Request, res: Response) {
  const { name, bio, skills } = req.body;

  if (!name || !bio || !skills) {
    return res.status(400).json({ error: 'Missing required fields' });
  }

  const slug = name.toLowerCase().replace(/\s+/g, '-');

  const template = await prisma.template.findFirst();
  if (!template) {
    return res.status(500).json({ error: 'No template found' });
  }

  const tenant = await prisma.tenant.create({
    data: {
      slug,
      name,
      bio,
      skills,
      templateId: template.id,
    },
  });

  res.json({ slug: tenant.slug });
}

export async function getTenant(req: Request, res: Response) {
  const slug = req.params.slug;

  if (!slug || typeof slug !== 'string') {
    return res.status(400).json({ error: 'Invalid slug parameter' });
  }

  const tenant = await prisma.tenant.findUnique({
    where: { slug },
  });

  if (!tenant) {
    return res.status(404).json({ error: 'Tenant not found' });
  }

  const template = await prisma.template.findUnique({
    where: { id: tenant.templateId },
  });

  res.json({ tenant, template });
}

Let's break down what this controller does.

createTenant takes the user's name, bio, and skills from the request body. It generates a slug by lowercasing the name and replacing spaces with hyphens — so "Jane Smith" becomes jane-smith. It then finds the first available template and creates the tenant record in the database, linking the template to the new tenant via templateId.

getTenant looks up a tenant by their slug and also fetches the template attached to them. Both pieces of data are returned together so the frontend can render the portfolio and apply the correct layout configuration in a single API call.

How to Create the Tenant Routes

Create src/routes/tenant.routes.ts:

import { Router } from 'express';
import { createTenant, getTenant } from '../controllers/tenant.controller';

const router = Router();

router.post('/tenants', createTenant);
router.get('/tenants/:slug', getTenant);

export default router;

Your API now exposes two endpoints:

POST   /api/tenants        — creates a new tenant
GET    /api/tenants/:slug  — retrieves a tenant and their template

How to Start the Server

Add the dev script to your package.json:

{
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "seed": "tsx prisma/seed.ts"
  }
}

Run it:

npm run dev

You should see Server is running on port 8080 in your terminal. Your backend is ready.

How to Create the Next.js Frontend

Navigate to the portfolio-client directory and create a new Next.js project. Make sure to select Yes when the installer asks if you want to use Tailwind CSS:

cd ../portfolio-client
npx create-next-app@latest . --typescript --app --tailwind --eslint
npm install

Since create-next-app sets up Tailwind for you automatically, no extra configuration is needed. The tailwind.config.ts and the @tailwind directives in globals.css are already in place.

How to Add Subdomain Routing with Middleware

This is the heart of the multi-tenant architecture. You need a piece of code that runs before every request, reads the subdomain from the URL, and rewrites the request to the correct internal route — all without the user ever seeing the URL change.

Create a file called proxy.ts in the root directory:

import { NextRequest, NextResponse } from 'next/server';

export function proxy(request: NextRequest) {
  const { pathname } = request.nextUrl;
  const host = request.headers.get('host') ?? '';

  const hostname = host.split(':')[0];
  const parts = hostname.split('.');

  // Local development: john.localhost:3000
  if (hostname.endsWith('localhost')) {
    const subdomain = parts[0];

    // Root localhost — load the landing page normally
    if (subdomain === 'localhost') {
      return NextResponse.next();
    }

    // Already rewritten — don't rewrite again
    if (pathname.startsWith('/tenant')) {
      return NextResponse.next();
    }

    return NextResponse.rewrite(new URL(`/tenant/${subdomain}`, request.url));
  }

  // Production: john.yourdomain.com
  if (parts.length > 2) {
    const subdomain = parts[0];

    if (subdomain !== 'www') {
      if (pathname.startsWith('/tenant')) {
        return NextResponse.next();
      }

      return NextResponse.rewrite(new URL(`/tenant/${subdomain}`, request.url));
    }
  }

  return NextResponse.next();
}

export const config = {
  matcher: [
    '/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)',
  ],
};

Here's exactly what this proxy does, step by step.

It reads the host header from every incoming request and splits it on . to extract the subdomain. For a request to john.localhost:3000, the subdomain is john. For a request directly to localhost:3000, the subdomain is localhost itself — in which case the proxy lets the request through unchanged so the landing page loads normally.

When a subdomain is detected, the proxy rewrites the request URL from / to /tenant/john internally. This rewrite is invisible to the browser — the user still sees john.localhost:3000 in their address bar, but Next.js routes the request to your /tenant/[slug] page.

The if (pathname.startsWith('/tenant')) guard prevents infinite rewrite loops. Without it, the already-rewritten request would be rewritten again on the next pass through the middleware.

How to Build the Landing Page

How to Update the Layout

Open app/layout.tsx and update it:

import type { Metadata } from 'next';
import { Geist, Geist_Mono } from 'next/font/google';
import './globals.css';

const geistSans = Geist({
  variable: '--font-geist-sans',
  subsets: ['latin'],
});

const geistMono = Geist_Mono({
  variable: '--font-geist-mono',
  subsets: ['latin'],
});

export const metadata: Metadata = {
  title: 'Portfolio SaaS App',
  description: 'Create and host your portfolio with subdomains',
};

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body className={`\({geistSans.variable} \){geistMono.variable} antialiased`}>
        {children}
      </body>
    </html>
  );
}

How to Create the Home Page

Create app/page.tsx:

'use client';

import { useState } from 'react';

export default function Home() {
  const [name, setName] = useState('');
  const [bio, setBio] = useState('');
  const [skills, setSkills] = useState('');

  const handleSubmit = async () => {
    const res = await fetch('http://localhost:8080/api/tenants', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        name,
        bio,
        skills: skills.split(',').map((s) => s.trim()),
      }),
    });

    const data = await res.json();
    window.location.href = `http://${data.slug}.localhost:3000`;
  };

  return (
    <div className="min-h-screen bg-gradient-to-br from-slate-900 to-slate-800 text-white flex flex-col">
      {/* Header */}
      <header className="flex items-center justify-between px-10 py-5">
        <span className="text-xl font-semibold tracking-wide">PortfolioSaaS</span>
        <nav className="flex gap-6 text-sm text-slate-400">
          <a href="#features" className="hover:text-white transition-colors">Features</a>
          <a href="#pricing" className="hover:text-white transition-colors">Pricing</a>
          <a href="#docs" className="hover:text-white transition-colors">Docs</a>
        </nav>
      </header>

      {/* Main */}
      <main className="flex flex-1 items-center justify-center px-6 py-10">
        <div className="w-full max-w-md bg-slate-800 rounded-2xl p-10 shadow-2xl">
          <h1 className="text-3xl font-bold mb-3">Create Your Portfolio</h1>
          <p className="text-slate-400 text-sm mb-8">
            Launch your personal portfolio instantly with your own subdomain.
          </p>

          <input
            type="text"
            placeholder="Your Name"
            onChange={(e) => setName(e.target.value)}
            className="w-full bg-slate-900 border border-slate-700 rounded-lg px-4 py-3 text-sm text-white placeholder-slate-500 focus:outline-none focus:border-indigo-500 mb-4"
          />

          <textarea
            placeholder="Short Bio"
            rows={4}
            onChange={(e) => setBio(e.target.value)}
            className="w-full bg-slate-900 border border-slate-700 rounded-lg px-4 py-3 text-sm text-white placeholder-slate-500 focus:outline-none focus:border-indigo-500 mb-4 resize-none"
          />

          <input
            type="text"
            placeholder="Skills (comma separated)"
            onChange={(e) => setSkills(e.target.value)}
            className="w-full bg-slate-900 border border-slate-700 rounded-lg px-4 py-3 text-sm text-white placeholder-slate-500 focus:outline-none focus:border-indigo-500 mb-6"
          />

          <button
            onClick={handleSubmit}
            className="w-full bg-indigo-600 hover:bg-indigo-500 text-white font-semibold py-3 rounded-lg transition-colors"
          >
            Create Portfolio
          </button>
        </div>
      </main>

      {/* Footer */}
      <footer className="text-center text-xs text-slate-500 py-5 border-t border-slate-800">
        © {new Date().getFullYear()} PortfolioSaaS. All rights reserved.
      </footer>
    </div>
  );
}

When a user submits the form, three things happen in sequence. First, a POST request creates a new tenant in your database. Second, the API returns the generated slug. Third, the browser redirects the user to their subdomain — their-name.localhost:3000 — where the middleware takes over and renders their portfolio.

How to Build the Tenant Portfolio Page

Create app/tenant/[slug]/page.tsx:

import type { Metadata } from 'next';

async function getTenant(slug: string) {
  const res = await fetch(`http://localhost:8080/api/tenants/${slug}`, {
    cache: 'no-store',
  });
  return res.json();
}

export async function generateMetadata({
  params,
}: {
  params: Promise<{ slug: string }>;
}): Promise<Metadata> {
  const { slug } = await params;
  const { tenant } = await getTenant(slug);

  if (!tenant) {
    return {
      title: 'Portfolio Not Found',
      description: 'This portfolio does not exist.',
      robots: { index: false, follow: false },
    };
  }

  return {
    title: tenant.name,
    description:
      tenant.bio?.slice(0, 160) ||
      `Explore ${tenant.name}'s professional portfolio.`,
    openGraph: {
      title: tenant.name,
      description: tenant.bio,
      type: 'website',
    },
  };
}

function initials(name: string) {
  return name
    .split(' ')
    .filter(Boolean)
    .slice(0, 2)
    .map((n) => n[0]?.toUpperCase())
    .join('');
}

export default async function TenantPage({
  params,
}: {
  params: Promise<{ slug: string }>;
}) {
  const { slug } = await params;
  const data = await getTenant(slug);

  const tenant = data?.tenant;
  const template = data?.template;

  if (!tenant) {
    return (
      <div className="min-h-screen bg-slate-900 text-white flex items-center justify-center">
        <h1 className="text-2xl font-bold text-slate-400">Portfolio not found</h1>
      </div>
    );
  }

  const primary = template?.config?.theme?.primaryColor || '#6366f1';

  // Template-driven section toggles with safe defaults
  const sections = {
    hero: true,
    about: true,
    skills: true,
    projects: true,
    blog: true,
    contact: true,
    ...(template?.config?.sections ?? {}),
  };

  const avatarUrl = tenant.avatarUrl as string | undefined;

  return (
    <div className="min-h-screen bg-gradient-to-br from-slate-900 via-slate-800 to-slate-900 text-slate-100">

      {/* Header */}
      <header className="sticky top-0 z-20 backdrop-blur-md bg-slate-900/60 border-b border-white/5">
        <div className="max-w-5xl mx-auto px-5 py-4 flex items-center justify-between gap-4">
          <div className="flex items-center gap-3 min-w-0">
            <span
              className="w-2.5 h-2.5 rounded-full shrink-0"
              style={{ backgroundColor: primary }}
            />
            <span className="font-semibold truncate">{tenant.name}</span>
          </div>

          <nav className="hidden md:flex items-center gap-5 text-sm text-slate-400">
            {(sections.hero || sections.about) && (
              <a href="#about" className="hover:text-white transition-colors">About</a>
            )}
            {sections.skills && (
              <a href="#skills" className="hover:text-white transition-colors">Skills</a>
            )}
            {sections.projects && (
              <a href="#projects" className="hover:text-white transition-colors">Projects</a>
            )}
            {sections.blog && (
              <a href="#blog" className="hover:text-white transition-colors">Blog</a>
            )}
            {sections.contact && (
              <a href="#contact" className="hover:text-white transition-colors">Contact</a>
            )}
          </nav>

          {sections.contact && (
            <a
              href="#contact"
              className="text-sm font-semibold px-4 py-2 rounded-full transition-transform hover:-translate-y-px"
              style={{ backgroundColor: primary, color: '#0b1020' }}
            >
              Hire me
            </a>
          )}
        </div>
      </header>

      {/* Hero / About */}
      {(sections.hero || sections.about) && (
        <section className="px-5 pt-20 pb-14" id="about">
          <div className="max-w-5xl mx-auto bg-white/[0.04] border border-white/[0.08] rounded-2xl p-7 shadow-2xl grid grid-cols-[110px_1fr] gap-6 items-center">

            {/* Avatar */}
            <div className="w-[110px] h-[110px] rounded-full overflow-hidden border border-white/10 bg-white/5 flex items-center justify-center shrink-0">
              {avatarUrl ? (
                // eslint-disable-next-line @next/next/no-img-element
                <img src={avatarUrl} alt={`${tenant.name} avatar`} className="w-full h-full object-cover" />
              ) : (
                <span className="text-2xl font-extrabold text-slate-200 tracking-tight">
                  {initials(tenant.name)}
                </span>
              )}
            </div>

            {/* Text */}
            <div className="min-w-0">
              <h1
                className="text-5xl font-extrabold tracking-tight leading-tight mb-3"
                style={{ color: primary }}
              >
                {tenant.name}
              </h1>
              <p className="text-slate-400 text-base leading-relaxed max-w-2xl">
                {tenant.bio}
              </p>
              <div className="flex flex-wrap gap-3 mt-5">
                {sections.contact && (
                  <a
                    href="#contact"
                    className="inline-flex items-center px-5 py-2.5 rounded-full text-sm font-semibold transition-transform hover:-translate-y-px"
                    style={{ backgroundColor: primary, color: '#0b1020' }}
                  >
                    Let&apos;s connect
                  </a>
                )}
                {sections.skills && (
                  <a
                    href="#skills"
                    className="inline-flex items-center px-5 py-2.5 rounded-full text-sm font-semibold text-slate-200 border border-white/10 bg-white/5 hover:border-white/20 transition-all hover:-translate-y-px"
                  >
                    View skills
                  </a>
                )}
              </div>
            </div>
          </div>
        </section>
      )}

      {/* Skills */}
      {sections.skills && (
        <section className="px-5 py-14 max-w-5xl mx-auto" id="skills">
          <h2 className="text-2xl font-bold text-center tracking-tight mb-7">Skills</h2>
          <div className="flex flex-wrap gap-3 justify-center">
            {tenant.skills.map((skill: string) => (
              <span
                key={skill}
                className="px-4 py-2 rounded-full text-sm font-semibold bg-white/5 backdrop-blur-sm hover:-translate-y-0.5 hover:shadow-lg transition-all"
                style={{ border: `1px solid ${primary}` }}
              >
                {skill}
              </span>
            ))}
          </div>
        </section>
      )}

      {/* Projects */}
      {sections.projects && (
        <section className="px-5 py-14 max-w-5xl mx-auto" id="projects">
          <h2 className="text-2xl font-bold text-center tracking-tight mb-7">Projects</h2>
          <div className="flex flex-wrap gap-3 justify-center">
            {['Portfolio SaaS', 'Multi-tenant Routing', 'Template Builder'].map((p) => (
              <span
                key={p}
                className="px-4 py-2 rounded-full text-sm font-semibold bg-white/5 backdrop-blur-sm hover:-translate-y-0.5 hover:shadow-lg transition-all"
                style={{ border: `1px solid ${primary}` }}
              >
                {p}
              </span>
            ))}
          </div>
        </section>
      )}

      {/* Blog */}
      {sections.blog && (
        <section className="px-5 py-14 max-w-5xl mx-auto" id="blog">
          <h2 className="text-2xl font-bold text-center tracking-tight mb-7">Blog</h2>
          <div className="flex flex-wrap gap-3 justify-center">
            {[
              'How I built this portfolio',
              'Next.js Middleware Tips',
              'Designing Templates',
            ].map((post) => (
              <span
                key={post}
                className="px-4 py-2 rounded-full text-sm font-semibold bg-white/5 backdrop-blur-sm hover:-translate-y-0.5 hover:shadow-lg transition-all"
                style={{ border: `1px solid ${primary}` }}
              >
                {post}
              </span>
            ))}
          </div>
        </section>
      )}

      {/* Contact */}
      {sections.contact && (
        <section className="px-5 pt-2 pb-16" id="contact">
          <div className="max-w-3xl mx-auto bg-white/[0.04] border border-white/[0.08] rounded-2xl p-7">
            <h2 className="text-xl font-bold mb-2">Contact</h2>
            <p className="text-slate-400 leading-relaxed mb-5">
              Want to work together? Send a message and I&apos;ll reply quickly.
            </p>
            <div className="flex flex-wrap gap-3">
              <a
                href={`mailto:hello@${tenant.slug}.com`}
                className="inline-flex items-center px-5 py-2.5 rounded-full text-sm font-semibold transition-transform hover:-translate-y-px"
                style={{ backgroundColor: primary, color: '#0b1020' }}
              >
                Email me
              </a>
              <a
                href="#about"
                className="inline-flex items-center px-5 py-2.5 rounded-full text-sm font-semibold text-slate-200 border border-white/10 bg-white/5 hover:border-white/20 transition-all hover:-translate-y-px"
              >
                Back to top
              </a>
            </div>
          </div>
        </section>
      )}

      {/* Footer */}
      <footer className="text-center text-xs text-slate-500 py-5 border-t border-white/5">
        © {new Date().getFullYear()} {tenant.name}
      </footer>
    </div>
  );
}

There are a few important details in this page worth calling out.

The generateMetadata function runs server-side before the page renders and sets the page title, description, and Open Graph tags for each tenant individually. This means every portfolio gets its own SEO metadata — important for a real SaaS product.

The sections object merges safe defaults (hero: true, skills: true, and so on) with whatever the tenant's template specifies. This means even if the template JSON is missing a key, the page won't break — the section will simply fall back to being shown.

The initials helper generates a two-letter avatar placeholder from the tenant's name when no profile image is available. "Jane Smith" produces "JS" — a small detail that makes the portfolio look polished even before a user adds a photo.

Notice that the primary color from the template JSON (theme.primaryColor) is applied using the style prop rather than a Tailwind class. This is intentional. Tailwind generates class names at build time and cannot know the dynamic color value stored in your database. Inline styles are the correct approach whenever a CSS value is truly dynamic.

How to Test the Full Flow

Start both servers in separate terminal windows:

# Terminal 1 — Backend
cd portfolio-api
npm run dev

# Terminal 2 — Frontend
cd portfolio-client
npm run dev

Now test the complete flow:

1. Visit http://localhost:3000 and fill out the form with your name, a short bio, and a comma-separated list of skills.

2. Click Create Portfolio. The form submits to your Express API, which creates the tenant record and returns the slug.

3. Your browser redirects to http://your-name.localhost:3000.

4. The Next.js middleware detects the subdomain, rewrites the request to /tenant/your-name, and your portfolio page fetches and renders your data.

You should see a fully rendered portfolio page with your name, bio, skills, and the placeholder projects and blog sections — all styled with Tailwind utility classes.

Next Steps

You now have a working multi-tenant SaaS foundation. Here are some extensions worth considering for a production build:

You could add authentication with NextAuth.js so tenants can log in and update their portfolio without losing their data between sessions.

You could also add custom domain support so tenants can point their own domain (for example, janedoe.com) to their portfolio by adding a CNAME record. You would need to handle wildcard SSL certificates on your hosting provider.

You could add image uploads for avatars using Cloudinary or AWS S3, then store the URL in the tenant record and replace the initials fallback with a real photo.

You could add real blog post management using the Post model already defined in your schema. Tenants could write and publish posts that appear on their portfolio.

And you could add Stripe subscriptions so tenants pay a monthly fee to keep their portfolio live. The architecture from the Stripe Connect tutorial maps directly onto this.

Finally, you could deploy the backend to Railway or Render and the frontend to Vercel. Just make sure to update your API URLs from localhost:8080 to your production URL before deploying.

Conclusion

In this tutorial, you built a complete multi-tenant SaaS platform where users can sign up, get their own subdomain, and have a portfolio site generated instantly — all from a single codebase.

You learned how to use Next.js middleware to detect subdomains and rewrite requests dynamically, model multi-tenant data in Prisma with a slug-based routing system, build a JSON-driven template system that controls page layout without code changes, and style a production-ready Next.js frontend entirely with Tailwind CSS utility classes.

The core insight is that multi-tenancy isn't magic. It's subdomain detection plus dynamic routing plus isolated data. Once you understand those three moving parts, you can apply this pattern to any SaaS product you build.

If you found this tutorial helpful, share it with someone who is learning to build full-stack applications. Happy coding!

Source Code

You can find the complete source code for both parts of this project on GitHub:

• Frontend (Next.js multi-tenant app) + Backend (Express + Prisma API): https://github.com/michaelokolo/portfolio-saas-v1

Message Queuing Telemetry Transport (MQTT) is a lightweight messaging protocol that excels at this. Combined with a broker like Mosquitto and a web framework like Express, you can build a production-ready real-time system in a single afternoon.

In this tutorial, you'll build a complete real-time football (soccer) sports update system from scratch. You'll create an admin interface for uploading scores and match details, a viewer interface for watching live updates, and a backend that uses MQTT to broadcast changes instantly to every connected client.

By the end of this guide, you'll understand how to integrate MQTT with Express, set up the Mosquitto broker, and deliver real-time data to web browsers using Server-Sent Events. You will have a working system that you can extend for production use.

Table of Contents

1. What You Will Learn

2. Prerequisites

3. Understanding the Architecture

4. What is MQTT and Why Use It?

   • MQTT Topic Design

   • Why Server-Sent Events Instead of WebSockets?

5. Project Setup

6. How to Set Up the MQTT Broker

   • Option 1: Docker (Recommended)

   • Option 2: Local Install

   • Option 3: Public Test Broker

7. How to Build the Express Server

8. How to Implement the Match Routes

9. How to Bridge MQTT to the Browser with Server-Sent Events

10. How to Build the Admin Upload Interface

    • Admin HTML Structure and Styles

    • Admin JavaScript Logic

11. How to Build the Live Viewer Interface

    • Viewer JavaScript Logic

12. How to Build the Home Page

13. How to Run and Test the System

14. How to Extend the System for Production

15. API Reference

16. Troubleshooting

17. Conclusion

What You Will Learn

During this tutorial, you'll learn how to:

• Connect an Express server to an MQTT broker using the MQTT.js library

• Publish and subscribe to MQTT topics for real-time messaging

• Use Server-Sent Events to push MQTT messages to web browsers

• Build a REST application programming interface (API) for match and score management

• Create a simple admin interface for uploading match data

• Create a viewer interface that updates in real time without page refreshes

Prerequisites

Before you start, you should have:

• Node.js version 18 or later installed on your machine

• Basic familiarity with JavaScript, Express, and HTML

• A terminal or command line for running commands

• Docker installed (optional, for running Mosquitto in a container)

If you don't have Node.js installed, you can download it from the official Node.js website.

Understanding the Architecture

The system has three main parts:

1. Admin interface – A web page where you create matches, update scores, and add events such as goals and cards.

2. Express server – Receives HyperText Transfer Protocol (HTTP) requests from the admin, publishes data to MQTT, subscribes to MQTT topics, and streams updates to viewers via Server-Sent Events.

3. Viewer interface – A web page that connects to the server and displays live scores and events as they arrive.

How the flow works
When you submit a score update in the admin panel, the Express server publishes a message to the MQTT broker. The server also subscribes to those same topics. When a message arrives, the server forwards it to all connected viewers through Server-Sent Events. The viewers update their display without refreshing the page.

What is MQTT and Why Use It?

MQTT stands for Message Queuing Telemetry Transport. It's a lightweight, publish-subscribe messaging protocol designed for low bandwidth and unreliable networks. It's widely used in Internet of Things (IoT) applications, but it works well for any real-time system where you need to broadcast updates to many subscribers.

Here are some reasons to use MQTT for a sports update system:

• Low overhead: Messages are small and efficient, which helps when you have many clients.

• Built-in Quality of Service (QoS): You can choose how many times a message is delivered (at most once, at least once, or exactly once).

• Topic-based routing: You organize messages by topic (for example, sports/football/match/123) so subscribers receive only what they need.

• Broker-based: A central broker (Mosquitto) handles all message distribution, so your application logic stays simple.

Mosquitto is a popular, open-source MQTT broker that's easy to install and configure.

MQTT Topic Design

MQTT uses a hierarchical topic structure. For this project, the topics are:

• sports/football/match/{id}: One topic per match. When you publish the full match object here, any subscriber receives the complete state. This makes it easy to add new match fields later without changing the topic structure.

• sports/football/scores: A single topic for score-related notifications. Messages include a type field (match_created or score_update) so subscribers can handle them differently.

• sports/football/events: A topic for match events such as goals and cards. Subscribers receive { type: 'match_event', matchId, event }.

The # wildcard in a subscription means "match this level and all levels below." So sports/football/# subscribes to every topic under sports/football, including sports/football/match/abc123 and sports/football/scores. The + wildcard matches exactly one level. For example, sports/+/match/# would match any sport, not just football.

Why Server-Sent Events Instead of WebSockets?

You might wonder why this tutorial uses Server-Sent Events (SSE) instead of WebSockets. Both can push data to the browser. The main difference:

• Server-Sent Events: One-way (server to client). Built on HTTP. Automatic reconnection in the browser. Simpler to implement. No extra libraries.

• WebSockets: Two-way. Requires a different protocol. More flexible but more complex.

For a sports score viewer, you only need server-to-client updates. The viewer never sends messages back over the same channel. Server-Sent Events is a better fit. If you later need the client to send commands (for example, to filter by league), you can add a separate HTTP API or switch to WebSockets.

Project Setup

Start by creating a new folder for your project and initialize it with npm. The mkdir command creates the directory, cd moves into it, and npm init -y creates a package.json file with default values without prompting you for input.

mkdir mqtt-football-scores
cd mqtt-football-scores
npm init -y

Install the required dependencies. Each package serves a specific role in the application. Run this command in the project root directory.

npm install express cors mqtt uuid

• express: Web framework for the HTTP server and API. It provides routing, middleware, and static file serving.

• cors: Enables Cross-Origin Resource Sharing so your frontend can call the API from a different origin (for example, if you serve the HTML from a different port or domain).

• mqtt: MQTT client for Node.js. It handles connection, publish, subscribe, reconnection, and Quality of Service (QoS) flows.

• uuid: Generates unique identifiers (Universally Unique Identifiers) for matches and events. Each ID is practically unique across all systems.

Next, create the following folder structure. The server folder holds the Node.js backend code. The public folder holds the HTML, CSS, and client-side JavaScript that the browser loads. The routes subfolder keeps the match-related route handlers separate from the main server file.

mqtt-football-scores/
├── server/
│   ├── index.js
│   ├── sse.js
│   └── routes/
│       └── matches.js
├── public/
│   ├── index.html
│   ├── admin.html
│   └── viewer.html
├── mosquitto.conf
├── docker-compose.yml
└── package.json

Add "type": "module" to your package.json so you can use JavaScript modules (import and export). The type field tells Node.js to treat .js files as ES modules, which allows you to use import and export syntax instead of CommonJS require and module.exports.

{
  "name": "mqtt-football-scores",
  "version": "1.0.0",
  "type": "module",
  "main": "server/index.js"
}

How to Set Up the MQTT Broker

You need an MQTT broker running before the server can connect. You have three options.

Option 1: Docker (Recommended)

Create a docker-compose.yml file. This file defines a single service named mosquitto that runs the Eclipse Mosquitto 2 image. The ports directive maps port 1883 on your host to port 1883 in the container so your Express server can connect. The volumes directive mounts your local mosquitto.conf into the container so the broker uses your configuration. The restart: unless-stopped option ensures the container restarts automatically if it crashes or if you reboot your machine.

version: "3.8"

services:
  mosquitto:
    image: eclipse-mosquitto:2
    container_name: mqtt-football-mosquitto
    ports:
      - "1883:1883"
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
    restart: unless-stopped

Create a mosquitto.conf file. The listener 1883 directive tells Mosquitto to listen on port 1883 for MQTT connections. The protocol mqtt specifies the standard MQTT protocol (as opposed to WebSocket). The allow_anonymous true setting permits connections without a username and password, which is fine for local development but should be disabled in production. The log_dest stdout and log_type all directives send all log output to the console so you can debug connection issues.

listener 1883
protocol mqtt
allow_anonymous true
log_dest stdout
log_type all

Start the broker:

docker-compose up -d

Option 2: Local Install

On macOS with Homebrew:

brew install mosquitto
mosquitto -c mosquitto.conf

On Ubuntu or Debian:

sudo apt install mosquitto mosquitto-clients
sudo systemctl start mosquitto

Option 3: Public Test Broker

You can use the public test broker at test.mosquitto.org without installing anything. Set the environment variable when you start the server:

MQTT_BROKER=mqtt://test.mosquitto.org npm start

Note: The public broker is shared and not suitable for production. Use it only for development and testing.

How to Build the Express Server

In this step, you'll create the main server that powers the real-time application. The Express server handles HTTP requests, serves static files like HTML and JavaScript, and acts as the bridge between the browser and the MQTT broker. It also provides endpoints that allow data to be sent and received in real time. Essentially, this server is the backbone of the application, enabling communication between all components.

Begin by creating the main server file at server/index.js:

import express from 'express';
import cors from 'cors';
import mqtt from 'mqtt';
import { fileURLToPath } from 'url';
import { dirname, join } from 'path';
import { v4 as uuidv4 } from 'uuid';

import { matchRoutes } from './routes/matches.js';
import { setupSSE, addSSEClient } from './sse.js';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

const MQTT_BROKER = process.env.MQTT_BROKER || 'mqtt://localhost:1883';
const PORT = process.env.PORT || 3000;

const app = express();
app.use(cors());
app.use(express.json());
app.use(express.static(join(__dirname, '../public')));

let mqttClient = null;

function connectMQTT() {
  mqttClient = mqtt.connect(MQTT_BROKER, {
    clientId: `football-scores-${uuidv4().slice(0, 8)}`,
    reconnectPeriod: 3000,
    connectTimeout: 10000,
  });

  mqttClient.on('connect', () => {
    console.log('Connected to MQTT broker at', MQTT_BROKER);
    mqttClient.subscribe('sports/football/#', { qos: 1 }, (err) => {
      if (err) console.error('Subscribe error:', err);
    });
  });

  mqttClient.on('error', (err) => {
    console.error('MQTT error:', err.message);
  });

  mqttClient.on('close', () => {
    console.log('MQTT connection closed');
  });

  mqttClient.on('reconnect', () => {
    console.log('MQTT reconnecting...');
  });

  return mqttClient;
}

const mqttClientInstance = connectMQTT();
const { publishMatch, publishScoreUpdate, publishEvent, getMatches } = matchRoutes(mqttClientInstance);
setupSSE(mqttClientInstance);

app.get('/api/events', (req, res) => addSSEClient(res));
app.post('/api/matches', publishMatch);
app.patch('/api/matches/:id/score', publishScoreUpdate);
app.post('/api/matches/:id/events', publishEvent);
app.get('/api/matches', getMatches);

app.listen(PORT, () => {
  console.log(`Football Scores Server running at http://localhost:${PORT}`);
  console.log(`Admin (upload):  http://localhost:${PORT}/admin.html`);
  console.log(`Viewer:          http://localhost:${PORT}/viewer.html`);
});

Here is what each part does:

• Imports: The fileURLToPath and dirname utilities replicate the __dirname variable that CommonJS provides, since ES modules don't have it. You need __dirname to build the path to the public folder.

• Environment variables: MQTT_BROKER defaults to mqtt://localhost:1883 so the server connects to a local Mosquitto instance. You can override it for Docker or a remote broker. PORT defaults to 3000.

• Middleware: cors() allows requests from any origin, which is useful during development. express.json() parses JSON request bodies. express.static() serves files from public so /admin.html and /viewer.html are available.

• connectMQTT: Creates an MQTT client with a unique client ID (required by the broker), a 3-second reconnect interval, and a 10-second connection timeout. On connect, it subscribes to sports/football/# with QoS 1. The # wildcard means "all topics under sports/football."

• matchRoutes: Returns the route handlers that create matches, update scores, and add events. Each handler publishes to MQTT and responds with JSON.

• setupSSE: Registers a listener on the MQTT client's message event. When a message arrives, it forwards the payload to all connected Server-Sent Events clients.

• addSSEClient: Called when a viewer opens /api/events. It sets the response headers for Server-Sent Events, flushes the headers so the connection stays open, and adds the response object to a set of active clients.

• Routes: The GET /api/events route establishes the Server-Sent Events stream. The POST, PATCH, and GET routes for matches delegate to the handlers from matchRoutes.

The server serves static files from the public folder, so your HTML pages are available at the root.

How to Implement the Match Routes

In this step, you'll create the route handlers that manage matches, scores, and events. These routes allow the server to receive requests from the admin interface, update match data, and publish real-time updates to MQTT. They also store match information in memory so it can be retrieved and updated during the session.

Begin by creating the file at server/routes/matches.js:

import { v4 as uuidv4 } from 'uuid';

const TOPIC_MATCH = 'sports/football/match';
const TOPIC_SCORES = 'sports/football/scores';
const TOPIC_EVENTS = 'sports/football/events';

const matches = new Map();

function publish(client, topic, payload, qos = 1) {
  if (!client?.connected) {
    console.warn('MQTT not connected, message not published');
    return false;
  }
  client.publish(topic, JSON.stringify(payload), { qos, retain: false });
  return true;
}

export function matchRoutes(mqttClient) {
  return {
    publishMatch: (req, res) => {
      const { homeTeam, awayTeam, league, venue, kickoff } = req.body;
      if (!homeTeam || !awayTeam) {
        return res.status(400).json({ error: 'homeTeam and awayTeam are required' });
      }

      const match = {
        id: uuidv4(),
        homeTeam,
        awayTeam,
        homeScore: 0,
        awayScore: 0,
        league: league || 'Premier League',
        venue: venue || 'TBD',
        kickoff: kickoff || new Date().toISOString(),
        status: 'scheduled',
        minute: 0,
        events: [],
        createdAt: new Date().toISOString(),
      };

      matches.set(match.id, match);

      const topic = `\({TOPIC_MATCH}/\){match.id}`;
      publish(mqttClient, topic, match);
      publish(mqttClient, TOPIC_SCORES, { type: 'match_created', match });

      res.status(201).json(match);
    },

    publishScoreUpdate: (req, res) => {
      const { id } = req.params;
      const { homeScore, awayScore, minute, status } = req.body;

      const match = matches.get(id);
      if (!match) {
        return res.status(404).json({ error: 'Match not found' });
      }

      if (homeScore !== undefined) match.homeScore = homeScore;
      if (awayScore !== undefined) match.awayScore = awayScore;
      if (minute !== undefined) match.minute = minute;
      if (status !== undefined) match.status = status;

      const topic = `\({TOPIC_MATCH}/\){id}`;
      publish(mqttClient, topic, match);
      publish(mqttClient, TOPIC_SCORES, {
        type: 'score_update',
        matchId: id,
        homeScore: match.homeScore,
        awayScore: match.awayScore,
        minute: match.minute,
        status: match.status,
      });

      res.json(match);
    },

    publishEvent: (req, res) => {
      const { id } = req.params;
      const { type, team, player, minute, description } = req.body;

      const match = matches.get(id);
      if (!match) {
        return res.status(404).json({ error: 'Match not found' });
      }

      const event = {
        id: uuidv4().slice(0, 8),
        type: type || 'goal',
        team,
        player: player || 'Unknown',
        minute: minute ?? match.minute,
        description: description || `\({type}: \){player}`,
        timestamp: new Date().toISOString(),
      };

      match.events.push(event);
      if (type === 'goal') {
        if (team === match.homeTeam) match.homeScore++;
        else if (team === match.awayTeam) match.awayScore++;
      }

      const topic = `\({TOPIC_MATCH}/\){id}`;
      publish(mqttClient, topic, match);
      publish(mqttClient, TOPIC_EVENTS, { type: 'match_event', matchId: id, event });

      res.status(201).json({ match, event });
    },

    getMatches: (req, res) => {
      const list = Array.from(matches.values()).sort(
        (a, b) => new Date(b.createdAt) - new Date(a.createdAt)
      );
      res.json(list);
    },
  };
}

The routes use an in-memory Map to store matches. In production, you would replace this with a database such as PostgreSQL or MongoDB.

Explanation of the key logic:

• publish: A helper that checks if the MQTT client is connected before publishing. If the broker is down, it logs a warning instead of throwing. The retain: false option means the broker doesn't store the last message for new subscribers.

• publishMatch: Validates that homeTeam and awayTeam are present. Creates a match object with default values for league, venue, kickoff, status, and events. Stores it in the Map, publishes to the match-specific topic and the scores topic, and returns the match with status 201 (Created).

• publishScoreUpdate: Looks up the match by ID. If not found, returns 404. Updates only the fields that are provided (using !== undefined so you can set a score to 0). Publishes the full match and a score_update notification.

• publishEvent: Creates an event object with a short unique ID. Pushes it to the match's events array. If the event type is goal, increments the home or away score based on the team. Publishes the updated match and an event notification.

• getMatches: Converts the Map to an array, sorts by createdAt descending (newest first), and returns the list as JSON.

MQTT topics used:

• sports/football/match/{id}: Full match state. Used when a match is created or updated.

• sports/football/scores: Score change notifications. Used for match creation and score updates.

• sports/football/events: Match events such as goals and cards.

The publish function sends JavaScript Object Notation (JSON) payloads with QoS 1 (at least once delivery). MQTT defines three QoS levels: 0 (at most once), 1 (at least once), and 2 (exactly once). Level 1 ensures the broker will retry until the subscriber acknowledges the message, which reduces the chance of losing score updates if the connection drops briefly.

How to Bridge MQTT to the Browser with Server-Sent Events

In this step, you'll create the Server-Sent Events (SSE) module that bridges MQTT messages to the browser. MQTT works over TCP and browsers cannot connect to it directly, so we use SSE as a lightweight HTTP-based streaming channel. SSE keeps an open connection and allows the server to push updates to connected browsers in real time. This is what enables viewers to see match updates instantly without refreshing.

Now create the file at server/sse.js:

const clients = new Set();

export function setupSSE(mqttClient) {
  if (!mqttClient) return;

  mqttClient.on('message', (topic, message) => {
    try {
      const payload = JSON.parse(message.toString());
      const data = JSON.stringify({ topic, ...payload });
      clients.forEach((res) => {
        try {
          res.write(`data: ${data}\n\n`);
        } catch (e) {
          clients.delete(res);
        }
      });
    } catch (e) {
      console.error('SSE parse error:', e.message);
    }
  });
}

export function addSSEClient(res) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no');
  res.flushHeaders();

  clients.add(res);

  res.on('close', () => {
    clients.delete(res);
  });
}

Explanation of each part:

• clients Set: A Set holds all active Server-Sent Events response objects. Using a Set makes it easy to add and remove clients without duplicates.

• setupSSE: Attaches a message listener to the MQTT client. When any message arrives on a subscribed topic, the callback runs. It parses the message payload (which is JSON), merges the topic into the payload with { topic, ...payload }, and sends the result to every client. The Server-Sent Events format requires each message to be data: {content}\n\n (two newlines). The forEach loop catches write errors (for example, if a client disconnected) and removes the client from the set.

• addSSEClient: Sets the Content-Type header to text/event-stream so the browser treats the response as an event stream. The Cache-Control: no-cache and Connection: keep-alive headers prevent the browser or proxy from caching or closing the connection. The X-Accel-Buffering: no header disables buffering in Nginx, which can delay or block Server-Sent Events. The flushHeaders call sends the headers immediately so the connection is established. The close event handler removes the client when the client disconnects (closes the tab or navigates away).

Server-Sent Events is one-way (server to client). For this use case, that is enough because viewers only need to receive updates, not send messages back over the same channel.

How to Build the Admin Upload Interface

The admin interface is a single HTML page where match creators can create new matches, update scores, and add events such as goals or cards. It uses standard HTML forms so data can be submitted to the server, and JavaScript will later handle form submissions and dynamic updates. All the markup, styles, and script live in public/admin.html, which the Express server serves as a static page.

Admin HTML Structure and Styles

The document starts with the standard HTML5 boilerplate. The charset and viewport meta tags ensure proper character encoding and responsive layout on mobile devices. The page loads the Outfit font from Google Fonts for a clean, modern look.

The Cascading Style Sheets (CSS) uses custom properties (variables) in the :root block so you can change the color scheme in one place. The --bg variable holds the dark background color, --surface for card backgrounds, --accent for the green accent color, and --text-muted for secondary text. The .grid class creates a two-column layout for form fields that collapses to one column on screens under 600 pixels wide. The .toast class positions the notification at the bottom-right and uses transform and opacity for a slide-in animation when the .show class is added.

Now create the file at public/admin.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Admin - Football Scores Upload</title>
  <link rel="preconnect" href="https://fonts.googleapis.com">
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
  <link href="https://fonts.googleapis.com/css2?family=Outfit:wght@400;500;600;700&display=swap" rel="stylesheet">
  <style>
    :root {
      --bg: #0f1419;
      --surface: #1a2332;
      --surface-hover: #243044;
      --accent: #00d26a;
      --accent-dim: #00a854;
      --text: #e8edf2;
      --text-muted: #8b9aab;
      --border: #2d3a4d;
      --danger: #ff4757;
    }
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body {
      font-family: 'Outfit', sans-serif;
      background: var(--bg);
      color: var(--text);
      min-height: 100vh;
      padding: 2rem;
    }
    .container { max-width: 900px; margin: 0 auto; }
    header {
      display: flex;
      align-items: center;
      gap: 1rem;
      margin-bottom: 2rem;
      padding-bottom: 1rem;
      border-bottom: 1px solid var(--border);
    }
    section {
      background: var(--surface);
      border-radius: 12px;
      padding: 1.5rem;
      margin-bottom: 1.5rem;
      border: 1px solid var(--border);
    }
    .grid {
      display: grid;
      grid-template-columns: 1fr 1fr;
      gap: 1rem;
    }
    @media (max-width: 600px) { .grid { grid-template-columns: 1fr; } }
    .form-group { display: flex; flex-direction: column; gap: 0.4rem; }
    .form-group.full { grid-column: 1 / -1; }
    input, select {
      padding: 0.65rem 1rem;
      border: 1px solid var(--border);
      border-radius: 8px;
      background: var(--bg);
      color: var(--text);
      font-family: inherit;
    }
    button {
      padding: 0.75rem 1.5rem;
      border: none;
      border-radius: 8px;
      font-family: inherit;
      font-weight: 600;
      cursor: pointer;
    }
    .btn-primary { background: var(--accent); color: var(--bg); }
    .btn-secondary { background: var(--surface-hover); color: var(--text); border: 1px solid var(--border); }
    .actions { display: flex; gap: 0.75rem; flex-wrap: wrap; margin-top: 1rem; }
    .badge { background: var(--accent); color: var(--bg); font-size: 0.75rem; padding: 0.25rem 0.6rem; border-radius: 999px; font-weight: 600; }
    .viewer-link { margin-left: auto; color: var(--accent); text-decoration: none; font-weight: 500; }
    section h2 { font-size: 1rem; font-weight: 600; margin-bottom: 1rem; color: var(--text-muted); }
    label { font-size: 0.85rem; font-weight: 500; color: var(--text-muted); }
    .toast {
      position: fixed;
      bottom: 2rem;
      right: 2rem;
      padding: 1rem 1.5rem;
      border-radius: 8px;
      font-weight: 500;
      color: var(--bg);
      background: var(--accent);
      transform: translateY(100px);
      opacity: 0;
      transition: transform 0.3s, opacity 0.3s;
      z-index: 100;
    }
    .toast.show { transform: translateY(0); opacity: 1; }
    .toast.error { background: var(--danger); }
    .match-card {
      display: flex;
      align-items: center;
      justify-content: space-between;
      padding: 1rem;
      background: var(--bg);
      border-radius: 8px;
      margin-bottom: 0.5rem;
      border: 1px solid var(--border);
    }
    .match-score { font-size: 1.5rem; font-weight: 700; color: var(--accent); margin: 0 1rem; }
    .match-info { flex: 1; }
    .match-teams { font-weight: 600; font-size: 1rem; }
    .match-meta { font-size: 0.8rem; color: var(--text-muted); margin-top: 0.25rem; }
    .match-actions { display: flex; gap: 0.5rem; }
    .match-actions button { padding: 0.5rem 1rem; font-size: 0.85rem; }
    .match-list { margin-top: 1rem; }
  </style>
</head>
<body>
  <div class="container">
    <header>
      <h1>⚽ Football Scores</h1>
      <span class="badge">Admin</span>
      <a href="/viewer.html" class="viewer-link">→ Open Viewer</a>
    </header>

    <section>
      <h2>Create New Match</h2>
      <form id="createMatch">
        <div class="grid">
          <div class="form-group">
            <label for="homeTeam">Home Team</label>
            <input type="text" id="homeTeam" placeholder="e.g. Manchester United" required>
          </div>
          <div class="form-group">
            <label for="awayTeam">Away Team</label>
            <input type="text" id="awayTeam" placeholder="e.g. Liverpool" required>
          </div>
          <div class="form-group">
            <label for="league">League</label>
            <input type="text" id="league" placeholder="e.g. Premier League" value="Premier League">
          </div>
          <div class="form-group">
            <label for="venue">Venue</label>
            <input type="text" id="venue" placeholder="e.g. Old Trafford">
          </div>
          <div class="form-group full">
            <label for="kickoff">Kickoff (ISO)</label>
            <input type="datetime-local" id="kickoff">
          </div>
        </div>
        <div class="actions">
          <button type="submit" class="btn-primary">Create Match</button>
        </div>
      </form>
    </section>

    <section>
      <h2>Update Score</h2>
      <form id="updateScore">
        <div class="grid">
          <div class="form-group full">
            <label for="scoreMatchId">Select Match</label>
            <select id="scoreMatchId" required>
              <option value="">-- Select match --</option>
            </select>
          </div>
          <div class="form-group">
            <label for="homeScore">Home Score</label>
            <input type="number" id="homeScore" min="0" value="0">
          </div>
          <div class="form-group">
            <label for="awayScore">Away Score</label>
            <input type="number" id="awayScore" min="0" value="0">
          </div>
          <div class="form-group">
            <label for="minute">Minute</label>
            <input type="number" id="minute" min="0" placeholder="e.g. 67">
          </div>
          <div class="form-group">
            <label for="status">Status</label>
            <select id="status">
              <option value="scheduled">Scheduled</option>
              <option value="live">Live</option>
              <option value="halftime">Halftime</option>
              <option value="finished">Finished</option>
            </select>
          </div>
        </div>
        <div class="actions">
          <button type="submit" class="btn-primary">Update Score</button>
        </div>
      </form>
    </section>

    <section>
      <h2>Add Match Event (Goal, Card, etc.)</h2>
      <form id="addEvent">
        <div class="grid">
          <div class="form-group full">
            <label for="eventMatchId">Select Match</label>
            <select id="eventMatchId" required>
              <option value="">-- Select match --</option>
            </select>
          </div>
          <div class="form-group">
            <label for="eventType">Event Type</label>
            <select id="eventType">
              <option value="goal">Goal</option>
              <option value="yellow_card">Yellow Card</option>
              <option value="red_card">Red Card</option>
              <option value="substitution">Substitution</option>
              <option value="penalty">Penalty</option>
            </select>
          </div>
          <div class="form-group">
            <label for="eventTeam">Team</label>
            <input type="text" id="eventTeam" placeholder="e.g. Manchester United">
          </div>
          <div class="form-group">
            <label for="eventPlayer">Player</label>
            <input type="text" id="eventPlayer" placeholder="e.g. Marcus Rashford">
          </div>
          <div class="form-group">
            <label for="eventMinute">Minute</label>
            <input type="number" id="eventMinute" min="0" placeholder="e.g. 23">
          </div>
          <div class="form-group full">
            <label for="eventDesc">Description</label>
            <input type="text" id="eventDesc" placeholder="Optional description">
          </div>
        </div>
        <div class="actions">
          <button type="submit" class="btn-primary">Add Event</button>
        </div>
      </form>
    </section>

    <section>
      <h2>Active Matches</h2>
      <div class="match-list" id="matchList"></div>
    </section>
  </div>

  <div class="toast" id="toast"></div>

The three forms use id attributes (createMatch, updateScore, addEvent) so the JavaScript can attach submit handlers. The match dropdowns (scoreMatchId and eventMatchId) are populated dynamically when the page loads. The matchList div is the container for the list of active matches. The toast element sits outside the main container so it can be fixed to the viewport.

Admin JavaScript Logic

The script block handles all interaction between the admin page and the server. It defines helper functions for showing notifications, fetching matches, updating the UI, and submitting data. When the page loads or after any action (create, update, or event submission), the UI refreshes so the latest match data is always visible.

  <script>
    const API = '/api';
    const toast = document.getElementById('toast');

    function showToast(msg, isError = false) {
      toast.textContent = msg;
      toast.className = 'toast show' + (isError ? ' error' : '');
      setTimeout(() => toast.classList.remove('show'), 3000);
    }

    async function fetchMatches() {
      const res = await fetch(`${API}/matches`);
      return res.json();
    }

    function populateSelects(matches) {
      const opts = matches.map(m => `<option value="\({m.id}">\){m.homeTeam} vs ${m.awayTeam}</option>`).join('');
      const html = '<option value="">-- Select match --</option>' + opts;
      document.getElementById('scoreMatchId').innerHTML = html;
      document.getElementById('eventMatchId').innerHTML = html;
    }

    function renderMatchList(matches) {
      const list = document.getElementById('matchList');
      if (!matches.length) {
        list.innerHTML = '<p style="color: var(--text-muted);">No matches yet. Create one above.</p>';
        return;
      }
      list.innerHTML = matches.map(m => `
        <div class="match-card" data-id="${m.id}">
          <div class="match-info">
            <div class="match-teams">\({m.homeTeam} vs \){m.awayTeam}</div>
            <div class="match-meta">\({m.league} • \){m.status} • ${m.minute}'</div>
          </div>
          <div class="match-score">\({m.homeScore} - \){m.awayScore}</div>
          <div class="match-actions">
            <button class="btn-secondary" onclick="quickScore('\({m.id}', \){m.homeScore}, ${m.awayScore})">Update</button>
          </div>
        </div>
      `).join('');
    }

    function quickScore(id, h, a) {
      document.getElementById('scoreMatchId').value = id;
      document.getElementById('homeScore').value = h;
      document.getElementById('awayScore').value = a;
    }

    async function loadMatches() {
      const matches = await fetchMatches();
      populateSelects(matches);
      renderMatchList(matches);
    }

    document.getElementById('createMatch').onsubmit = async (e) => {
      e.preventDefault();
      const body = {
        homeTeam: document.getElementById('homeTeam').value.trim(),
        awayTeam: document.getElementById('awayTeam').value.trim(),
        league: document.getElementById('league').value.trim() || 'Premier League',
        venue: document.getElementById('venue').value.trim() || 'TBD',
        kickoff: document.getElementById('kickoff').value
          ? new Date(document.getElementById('kickoff').value).toISOString()
          : new Date().toISOString(),
      };
      try {
        const res = await fetch(`${API}/matches`, {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(body),
        });
        const data = await res.json();
        if (res.ok) {
          showToast('Match created!');
          document.getElementById('createMatch').reset();
          loadMatches();
        } else showToast(data.error || 'Failed', true);
      } catch (err) {
        showToast('Network error', true);
      }
    };

    document.getElementById('updateScore').onsubmit = async (e) => {
      e.preventDefault();
      const id = document.getElementById('scoreMatchId').value;
      const body = {
        homeScore: parseInt(document.getElementById('homeScore').value, 10),
        awayScore: parseInt(document.getElementById('awayScore').value, 10),
        minute: parseInt(document.getElementById('minute').value, 10) || undefined,
        status: document.getElementById('status').value,
      };
      try {
        const res = await fetch(`\({API}/matches/\){id}/score`, {
          method: 'PATCH',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(body),
        });
        const data = await res.json();
        if (res.ok) {
          showToast('Score updated!');
          loadMatches();
        } else showToast(data.error || 'Failed', true);
      } catch (err) {
        showToast('Network error', true);
      }
    };

    document.getElementById('addEvent').onsubmit = async (e) => {
      e.preventDefault();
      const id = document.getElementById('eventMatchId').value;
      const body = {
        type: document.getElementById('eventType').value,
        team: document.getElementById('eventTeam').value.trim(),
        player: document.getElementById('eventPlayer').value.trim(),
        minute: parseInt(document.getElementById('eventMinute').value, 10) || undefined,
        description: document.getElementById('eventDesc').value.trim() || undefined,
      };
      try {
        const res = await fetch(`\({API}/matches/\){id}/events`, {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(body),
        });
        const data = await res.json();
        if (res.ok) {
          showToast('Event added!');
          loadMatches();
        } else showToast(data.error || 'Failed', true);
      } catch (err) {
        showToast('Network error', true);
      }
    };

    loadMatches();
  </script>
</body>
</html>

Explanation of each part:

• showToast: updates the toast text and adds the show class to trigger the CSS animation. The isError parameter switches the toast to a red background for error messages. The setTimeout removes the show class after 3 seconds so the toast slides back out.

• fetchMatches: calls the GET /api/matches endpoint and returns the parsed JSON so the UI can display the latest data.

• populateSelects: builds option elements from the matches array and injects them into both dropdowns, so the same match list appears in the Update Score and Add Event forms.

• renderMatchList: either shows a placeholder when there are no matches or renders each match as a card with team names, scores, league, status, and an Update button.

• quickScore: pre-fills the Update Score form when you click the Update button on a match card, so you can adjust the score without re-selecting the match.

• loadMatches: fetches matches, populates the dropdowns, and renders the list. It runs on page load and after every successful create, update, or event submission.

• onsubmit: calls e.preventDefault() to stop the default form submission, builds a request body from the form values, sends a fetch request to the appropriate endpoint, and on success shows a toast and calls loadMatches to refresh the UI.

How to Build the Live Viewer Interface

The viewer interface displays real-time match updates and connects to the Server-Sent Events endpoint so it can receive data the moment the server pushes it.

Unlike the admin page, the viewer is read-only: it shows live scores, match events, and status updates without requiring user input. The page uses a dark theme and a connection indicator so users can see whether the real-time stream is active.

Now create the file at public/viewer.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Live Football Scores</title>
  <link rel="preconnect" href="https://fonts.googleapis.com">
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
  <link href="https://fonts.googleapis.com/css2?family=Outfit:wght@400;500;600;700&display=swap" rel="stylesheet">
  <style>
    :root {
      --bg: #0a0e14;
      --surface: #131a24;
      --accent: #00d26a;
      --accent-glow: rgba(0, 210, 106, 0.3);
      --text: #e8edf2;
      --text-muted: #8b9aab;
      --border: #2d3a4d;
      --live: #ff4757;
    }
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body {
      font-family: 'Outfit', sans-serif;
      background: var(--bg);
      color: var(--text);
      min-height: 100vh;
      padding: 2rem;
    }
    .container { max-width: 700px; margin: 0 auto; }
    header { text-align: center; margin-bottom: 2rem; }
    .status {
      display: inline-flex;
      align-items: center;
      gap: 0.5rem;
      font-size: 0.85rem;
      color: var(--text-muted);
    }
    .status-dot {
      width: 8px;
      height: 8px;
      border-radius: 50%;
      background: var(--text-muted);
      animation: pulse 2s infinite;
    }
    .status-dot.connected {
      background: var(--accent);
      box-shadow: 0 0 0 3px var(--accent-glow);
    }
    @keyframes pulse {
      0%, 100% { opacity: 1; }
      50% { opacity: 0.5; }
    }
    .match-card {
      background: var(--surface);
      border-radius: 16px;
      padding: 1.5rem;
      margin-bottom: 1rem;
      border: 1px solid var(--border);
      transition: border-color 0.2s, box-shadow 0.2s;
    }
    .match-card.live {
      border-color: var(--live);
      box-shadow: 0 0 0 1px rgba(255, 71, 87, 0.2);
    }
    .match-header {
      display: flex;
      align-items: center;
      justify-content: space-between;
      margin-bottom: 1rem;
    }
    .league { 
      font-size: 0.8rem; 
      color: var(--text-muted); 
      margin-bottom: 0.25rem; 
    }
    .match-teams {
      display: flex;
      align-items: center;
      justify-content: space-between;
      gap: 1rem;
      margin: 1rem 0;
    }
    .team { flex: 1; text-align: center; font-weight: 600; font-size: 1.1rem; }
    .team.home { text-align: left; }
    .team.away { text-align: right; }
    .score-box {
      display: flex;
      align-items: center;
      justify-content: center;
      min-width: 80px;
      gap: 0.5rem;
    }
    .score { 
      font-size: 2rem; 
      font-weight: 700; 
      color: var(--accent); 
     }
    .status-badge { 
      font-size: 0.7rem; 
      padding: 0.2rem 0.5rem; 
      border-radius: 4px; 
      font-weight: 600; 
     }
    .status-badge.live { background: var(--live); color: white; }
    .status-badge.finished { background: var(--border); color: var(--text-muted); }
    .status-badge.scheduled { background: var(--accent); color: var(--bg); }
    .match-meta { font-size: 0.85rem; color: var(--text-muted); margin-top: 0.5rem; }
    .events { margin-top: 1rem; padding-top: 1rem; border-top: 1px solid var(--border); }
    .events h4 { font-size: 0.8rem; color: var(--text-muted); margin-bottom: 0.5rem; }
    .event { display: flex; align-items: center; gap: 0.5rem; font-size: 0.85rem; padding: 0.35rem 0; border-bottom: 1px solid var(--border); }
    .event:last-child { border-bottom: none; }
    .event-icon { width: 24px; text-align: center; font-size: 1rem; }
    .event.goal .event-icon { color: var(--accent); }
    .event.yellow_card .event-icon { color: #ffd93d; }
    .event.red_card .event-icon { color: var(--live); }
    .feed { margin-top: 2rem; padding-top: 1.5rem; border-top: 1px solid var(--border); }
    .feed h3 { font-size: 1rem; margin-bottom: 1rem; color: var(--text-muted); }
    .feed-item { font-size: 0.85rem; padding: 0.5rem 0; color: var(--text-muted); border-bottom: 1px solid var(--border); }
    .feed-item:last-child { border-bottom: none; }
    .feed-item strong { color: var(--text); }
    .empty { text-align: center; padding: 3rem 2rem; color: var(--text-muted); }
    .empty-icon { font-size: 3rem; margin-bottom: 1rem; opacity: 0.5; }
  </style>
</head>
<body>
  <div class="container">
    <header>
      <h1>⚽ Live Football Scores</h1>
      <div class="status" id="status">
        <span class="status-dot" id="statusDot"></span>
        <span id="statusText">Connecting...</span>
      </div>
    </header>

    <div id="matches"></div>

    <div class="feed" id="feedSection">
      <h3>Live Feed</h3>
      <div id="feed"></div>
    </div>
  </div>

Explanation of each part:

• header: displays the page title and a connection indicator so users know whether the real-time stream is active.

• status dot: a small circle that pulses while disconnected and turns green with a glow when the SSE connection is established.

• matches container: the div with id="matches" is where match cards will be rendered dynamically by JavaScript as data arrives.

• feed section: shows a chronological list of live updates so users can see recent events at a glance.

• CSS theme: uses dark colors and custom properties so the look can be adjusted in one place. The live badge and border styles highlight matches that are in progress.

• Server-Sent Events integration: JavaScript (added in the next step) will connect to /api/events and update this page whenever new data arrives.

The header shows the title and a connection status indicator. The matches div is the container for match cards, populated by JavaScript. The feed div displays the live feed of recent updates.

Viewer JavaScript Logic

The script powers the live viewer interface by maintaining an in-memory collection of matches and a feed of recent updates. It connects to the Server-Sent Events endpoint so data flows from the server to the browser in real time. When messages arrive, the page updates automatically to show the latest scores, events, and match details.

  <script>
    const API = '/api';
    const matches = new Map();
    const feed = [];
    const MAX_FEED = 20;

    const statusDot = document.getElementById('statusDot');
    const statusText = document.getElementById('statusText');
    const matchesEl = document.getElementById('matches');
    const feedEl = document.getElementById('feed');

    function setStatus(connected) {
      statusDot.classList.toggle('connected', connected);
      statusText.textContent = connected ? 'Live' : 'Reconnecting...';
    }

    function renderMatches() {
      const list = Array.from(matches.values()).sort(
        (a, b) => new Date(b.createdAt) - new Date(a.createdAt)
      );
      if (!list.length) {
        matchesEl.innerHTML = `
          <div class="empty">
            <div class="empty-icon">⚽</div>
            <p>No matches yet. Updates will appear here in real-time.</p>
          </div>
        `;
        return;
      }
      matchesEl.innerHTML = list.map(m => `
        <div class="match-card \({m.status === 'live' ? 'live' : ''}" data-id="\){m.id}">
          <div class="match-header">
            <div>
              <div class="league">${m.league}</div>
              <div class="match-meta">\({m.venue} • \){m.kickoff ? new Date(m.kickoff).toLocaleString() : ''}</div>
            </div>
            <span class="status-badge \({m.status}">\){m.status}</span>
          </div>
          <div class="match-teams">
            <div class="team home">${m.homeTeam}</div>
            <div class="score-box">
              <span class="score">${m.homeScore}</span>
              <span>-</span>
              <span class="score">${m.awayScore}</span>
            </div>
            <div class="team away">${m.awayTeam}</div>
          </div>
          \({m.minute ? `<div class="match-meta">\){m.minute}'</div>` : ''}
          ${m.events?.length ? `
            <div class="events">
              <h4>Events</h4>
              ${m.events.map(e => `
                <div class="event ${e.type}">
                  <span class="event-icon">${getEventIcon(e.type)}</span>
                  <span>\({e.minute}' \){e.player} (\({e.team}) - \){e.description || e.type}</span>
                </div>
              `).join('')}
            </div>
          ` : ''}
        </div>
      `).join('');
    }

    function getEventIcon(type) {
      const icons = { goal: '⚽', yellow_card: '🟨', red_card: '🟥', substitution: '🔄', penalty: '⚽' };
      return icons[type] || '•';
    }

    function renderFeed() {
      const items = feed.slice(-MAX_FEED).reverse();
      feedEl.innerHTML = items.length
        ? items.map(f => `<div class="feed-item">${f}</div>`).join('')
        : '<div class="feed-item">Waiting for updates...</div>';
    }

    function addFeedItem(type, msg) {
      const time = new Date().toLocaleTimeString();
      feed.push(`<strong>\({time}</strong> | \){msg}`);
      if (feed.length > MAX_FEED) feed.shift();
      renderFeed();
    }

    function handleMessage(data) {
      if (data.match) {
        matches.set(data.match.id, data.match);
        renderMatches();
      }
      if (data.id && data.homeTeam && data.awayTeam) {
        matches.set(data.id, data);
        renderMatches();
      }
      if (data.type === 'match_created' && data.match) {
        addFeedItem('match', `New match: \({data.match.homeTeam} vs \){data.match.awayTeam}`);
      }
      if (data.type === 'score_update') {
        const m = matches.get(data.matchId);
        if (m) {
          m.homeScore = data.homeScore;
          m.awayScore = data.awayScore;
          m.minute = data.minute;
          m.status = data.status;
          matches.set(data.matchId, m);
          addFeedItem('score', `\({m.homeTeam} \){data.homeScore}-\({data.awayScore} \){m.awayTeam} (${data.minute || '?'}')`);
          renderMatches();
        }
      }
      if (data.type === 'match_event' && data.event) {
        const m = matches.get(data.matchId);
        if (m) {
          m.events = m.events || [];
          m.events.push(data.event);
          if (data.event.type === 'goal') {
            if (data.event.team === m.homeTeam) m.homeScore++;
            else if (data.event.team === m.awayTeam) m.awayScore++;
          }
          matches.set(data.matchId, m);
          addFeedItem('event', `\({data.event.type}: \){data.event.player} (\({data.event.team}) - \){data.event.minute}'`);
          renderMatches();
        }
      }
    }

    function handleSSEMessage(msg) {
      try {
        const data = JSON.parse(msg);
        if (data.topic && /^sports\/football\/match\/([^/]+)$/.test(data.topic) && data.id) {
          matches.set(data.id, data);
        }
        handleMessage(data);
      } catch (e) {}
    }

    function connectSSE() {
      const es = new EventSource(`${API}/events`);
      es.onopen = () => setStatus(true);
      es.onerror = () => {
        setStatus(false);
        es.close();
        setTimeout(connectSSE, 3000);
      };
      es.onmessage = (e) => handleSSEMessage(e.data);
    }

    async function loadInitial() {
      try {
        const res = await fetch(`${API}/matches`);
        const list = await res.json();
        list.forEach(m => matches.set(m.id, m));
        renderMatches();
      } catch (e) {
        matchesEl.innerHTML = '<div class="empty"><p>Could not load matches. Is the server running?</p></div>';
      }
    }

    loadInitial();
    connectSSE();
  </script>
</body>
</html>

Explanation of each part:

• The matches Map: stores match objects keyed by ID so updates can be applied efficiently without searching arrays.

• The feed array: keeps a small history of recent events (limited to MAX_FEED) so the live feed remains lightweight.

• setStatus: toggles the connected class on the status dot and updates the status text to “Live” or “Reconnecting…” so users know connection status.

• renderMatches: converts the Map to a sorted array (newest first). If there are no matches, it displays an empty state. Otherwise it renders cards showing league, venue, teams, scores, status badge, minute, and events.

• getEventIcon: returns an emoji for each event type so events are visually identifiable (goal, card, substitution, and so on).

• renderFeed: displays the live feed items or a placeholder message when no updates exist.

• addFeedItem: appends a timestamped message to the feed, keeps only the latest items, and re-renders the feed.

• handleMessage: processes incoming data. It updates the matches Map for full match objects, score updates, and match events. For score updates and goals it adjusts scores and adds feed items so the viewer reflects real-time changes.

• handleSSEMessage: parses the Server-Sent Events payload and forwards it to handleMessage. If the message contains a match topic and full match data, it stores it in the Map.

• connectSSE: creates an EventSource connection to /api/events. On open it marks the connection as live. On error it closes and retries after three seconds so transient network issues do not break the stream.

• loadInitial: fetches existing matches on page load so the viewer displays data even before real-time updates arrive.

How to Build the Home Page

The home page (public/index.html) is a simple landing page that links to the viewer and admin interfaces. It uses a centered card layout with two buttons. The primary button (green) goes to the viewer, and the secondary button (outlined) goes to the admin. The page uses the same dark theme and Outfit font for consistency. There is no JavaScript – it's purely static HTML and CSS.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Football Scores - MQTT Real-Time</title>
  <link rel="preconnect" href="https://fonts.googleapis.com">
  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
  <link href="https://fonts.googleapis.com/css2?family=Outfit:wght@400;500;600;700&display=swap" rel="stylesheet">
  <style>
    :root {
      --bg: #0a0e14;
      --surface: #131a24;
      --accent: #00d26a;
      --text: #e8edf2;
      --text-muted: #8b9aab;
    }
    * { box-sizing: border-box; margin: 0; padding: 0; }
    body {
      font-family: 'Outfit', sans-serif;
      background: var(--bg);
      color: var(--text);
      min-height: 100vh;
      display: flex;
      align-items: center;
      justify-content: center;
      padding: 2rem;
    }
    .card {
      background: var(--surface);
      border-radius: 16px;
      padding: 2rem;
      max-width: 400px;
      text-align: center;
      border: 1px solid rgba(255,255,255,0.06);
    }
    h1 { font-size: 1.5rem; margin-bottom: 0.5rem; }
    p { color: var(--text-muted); font-size: 0.95rem; margin-bottom: 1.5rem; }
    .links { display: flex; flex-direction: column; gap: 0.75rem; }
    a {
      display: block;
      padding: 1rem 1.5rem;
      background: var(--accent);
      color: var(--bg);
      text-decoration: none;
      font-weight: 600;
      border-radius: 10px;
      transition: opacity 0.2s;
    }
    a:hover { opacity: 0.9; }
    a.secondary {
      background: transparent;
      color: var(--text);
      border: 1px solid rgba(255,255,255,0.15);
    }
  </style>
</head>
<body>
  <div class="card">
    <h1>⚽ Football Scores</h1>
    <p>Real-time updates via MQTT & Mosquitto</p>
    <div class="links">
      <a href="/viewer.html">View Live Scores</a>
      <a href="/admin.html" class="secondary">Admin - Upload Scores</a>
    </div>
  </div>
</body>
</html>

The Express server serves index.html when you visit the root URL (http://localhost:3000/) because the express.static middleware serves files from the public folder, and Express automatically serves index.html when the request path is /.

How to Run and Test the System

Start the MQTT broker (Docker or local install).

Then start the Express server:

npm start

Next, open http://localhost:3000/admin.html in the admin panel. Create a new match (for example, Manchester United vs Liverpool).

Open http://localhost:3000/viewer.html in another tab or window. Then update the score or add an event in the admin panel. The viewer should update within a second without refreshing.

How to Extend the System for Production

The current implementation uses in-memory storage. For production, you should:

• Add a database: Store matches in PostgreSQL, MongoDB, or another database. Load matches on startup and persist every create, update, and event.

• Add authentication: Protect the admin routes with JSON Web Tokens (JWT) or session-based auth so only authorized users can upload scores.

• Add validation: Validate request bodies with a library such as Joi or Zod to prevent invalid data.

• Enable TLS: Use HTTPS for the Express server and secure WebSockets or MQTTS for the broker in production.

• Scale horizontally: If you run multiple server instances, each will have its own MQTT connection and SSE clients. The MQTT broker will deliver messages to all subscribers, so each instance will receive updates and forward them to its connected viewers.

API Reference

For quick reference, here are the endpoints your server exposes:

The status field can be scheduled, live, halftime, or finished. The type field for events can be goal, yellow_card, red_card, substitution, or penalty.

Troubleshooting

You might come across various issues as you build this. Here are some common ones:

The server cannot connect to the MQTT broker. Check that Mosquitto is running. If you use Docker, run docker ps to verify the container is up. If you use the public broker, ensure you have internet connectivity and that your firewall allows outbound connections on port 1883.

The viewer does not update when you change scores. Open the browser developer tools and check the Network tab. The /api/events request should show a pending state (it stays open). If it fails or closes, check the server logs for errors. Ensure you are not behind a proxy that buffers or closes long-lived connections.

Matches disappear when you restart the server. The current implementation stores matches in memory. Restarting the server clears the data. Add a database as described in the production section to persist matches across restarts.

Conclusion

In this tutorial, you built a real-time football sports update system using MQTT, Mosquitto, and Express. You've learned how to:

• Connect an Express server to an MQTT broker

• Publish match and score updates to MQTT topics

• Subscribe to topics and forward messages to browsers via Server-Sent Events

• Build an admin interface for creating matches and updating scores

• Build a viewer interface that displays live updates without page refreshes

The same pattern applies to other real-time systems: IoT dashboards, live notifications, collaborative editing, and more. MQTT gives you a reliable, scalable messaging layer, and Server-Sent Events gives you a simple way to push updates to web clients.

The code in this tutorial gives you a solid foundation. Try adding features such as match filtering by league, historical event logs, or push notifications to make the system your own.

Building a payroll system is an excellent way to practice real-world backend development skills. Unlike simple CRUD applications, payroll systems require you to think about:

• Asynchronous processing: When you need to pay hundreds of employees, processing payments synchronously can cause your server to timeout. Background jobs with Bull and Redis allow you to handle long-running operations without blocking your API.

• Payment gateway integration: Working with payment APIs like Monnify teaches you how to handle external service integrations, authentication flows, webhook verification, and error handling in production systems.

• Data consistency: Payroll systems need to maintain accurate records. You'll learn about transaction reconciliation, idempotency, and how to handle partial failures gracefully.

• Production-ready patterns: This tutorial covers patterns you'll use in real applications: job queues, webhook handlers, database migrations, and proper error handling.

Whether you're building a fintech application, an HR system, or just want to understand how payment processing works, the concepts in this tutorial will serve you well. The combination of Express, TypeScript, background jobs, and payment APIs represents a common stack in modern backend development.

In this tutorial, you’ll learn how to build a production-grade payroll engine using Express.js, TypeScript, and Monnify's payment API. You'll implement background job processing with Bull and Redis to handle bulk disbursements efficiently.

By the end, you will have a fully functional payroll system that can:

• Manage employee records with bank account details

• Create and process payroll batches

• Process bulk payments using Monnify's disbursement API

• Handle payment status updates via webhooks

• Reconcile transactions to ensure data consistency

Table of Contents

1. Prerequisites

2. Project Architecture Overview

3. Setting Up the Project

4. Configuring Docker for PostgreSQL and Redis

5. Setting Up the Database

6. Creating Database Models

   • Employee Model

   • Employee Data Structure (Employee Interface)

   • Employee Model Class

   • Auto-Generating Employee IDs (generateEmployeeId)

   • Creating an Employee (create)

   • Retrieving All Active Employees (findAll)

   • Retrieving an Employee by Database ID (findById)

   • Retrieving an Employee by Employee Identifier (findByEmployeeId)

   • Updating an Employee (update)

   • Soft-Deleting an Employee (delete)

   • Payroll Model

   • Payroll Status Lifecycle

   • Payroll Entity

   • Payroll Item Entity

   • Creating a Payroll (PayrollModel.create)

   • Fetching Payroll Records

   • Updating Payroll Status (PayrollModel.updateStatus)

7. PayrollItemModel

   • Fetching Payroll Items (PayrollItemModel.findByPayrollId)

   • Fetching a Single Payroll Item (PayrollItemModel.findById)

   • Updating Payroll Item Status (PayrollItemModel.updateStatus)

   • Overall Payroll Flow

8. Building the Monnify Client

   • Configuration and Environment Setup

   • Create the MonnifyClient Class

   • Axios Client and Request Interceptor

   • Authenticate with Monnify

   • Automatic Token Refresh (ensureAuthenticated)

   • Initiating Bulk Transfers

   • Authorizing Bulk Transfers (OTP Validation)

   • Transaction Status Lookup

   • Batch Details Retrieval

   • Wallet Balance Check

9. Implementing Background Job Processing

   • Set Up the Payroll Processing Queue

   • Queue Processor Registration

   • Bulk Payroll Processing Flow (processBulkPayroll)

   • Building the Bulk Transfer Payload

   • Initiating Bulk Disbursement via Monnify

   • Storing Transaction References

   • Payroll Statistics Reconciliation (updatePayrollStats)

   • Queue Entry Point (processPayrollItems)

   • Role in the Overall Payroll Architecture

10. Creating the API Controllers

    • Controller Responsibilities

    • Creating an Employee (createEmployee)

    • Fetching All Employees (getAllEmployees)

    • Fetching a Single Employee (getEmployeeById)

    • Updating an Employee (updateEmployee)

    • Deleting an Employee (deleteEmployee)

    • Error Handling Strategy

    • Role in the Overall Payroll System

    • Payroll Controller

    • Controller Responsibilities

    • Creating a Payroll (createPayroll)

    • Fetching All Payrolls (getAllPayrolls)

    • Fetching a Payroll with Items (getPayrollById)

    • Processing a Payroll (processPayroll)

    • Reconciling Payroll Payments (reconcilePayroll)

    • Payroll Statistics Update (Internal Helper)

    • Fetching Payroll Status Summary (getPayrollStatus)

    • Authorizing Bulk Transfers (authorizeBulkTransfer)

    • Checking Transaction Status (checkTransactionStatus)

    • Checking Wallet Balance (getAccountBalance)

    • Error Handling and Resilience

    • Role in the Overall Payroll Architecture

11. Setting Up Webhook Handlers

12. Wiring Up Routes

    • Employee Routes

    • Payroll Routes

    • Main Application Entry Point

13. Testing the System

14. Setting Up Webhooks for Production

15. Conclusion

    • Key Takeaways

    • References:

Prerequisites

Before you begin, make sure you have the following:

• Node.js (v18 or higher)

• Docker and Docker Compose installed

• A Monnify merchant account with API credentials

• Basic knowledge of TypeScript and Express.js

• Familiarity with REST APIs

You'll also need to obtain these credentials from your Monnify dashboard:

• API Key

• Secret Key

• Contract Code

• Webhook Secret (for verifying webhook signatures)

Project Architecture Overview

Here's how the payroll system works:

Key components:

1. Express API: A minimal and flexible Node.js web framework that handles HTTP requests for managing employees and payrolls. Express provides routing, middleware support, and makes it easy to build RESTful APIs.

2. Bull Queue: A Redis-based queue library for Node.js that processes payroll jobs asynchronously in the background. Bull handles job retries, scheduling, and provides a reliable way to process long-running tasks without blocking your main application thread.

3. Redis: An in-memory data structure store that serves as the backend for Bull queues. Redis stores job data, manages job states (pending, active, completed, failed), and enables distributed job processing across multiple workers.

4. PostgreSQL: A relational database that persists employee records, payrolls, and payment items. PostgreSQL's ACID compliance ensures data integrity, and its support for complex queries makes it ideal for financial applications.

5. Monnify API: A payment gateway service that handles actual money transfers to employee bank accounts. Monnify provides bulk disbursement capabilities, allowing you to process multiple payments in a single API call, which is essential for payroll systems.

6. Webhooks: HTTP callbacks that receive real-time payment status updates from Monnify. When a payment completes or fails, Monnify sends a webhook to your server, allowing you to update your database immediately without polling.

Setting Up the Project

In this section, we'll initialize a new Node.js project with TypeScript and install all the necessary dependencies. We'll configure TypeScript for type safety and set up the project structure that will support our payroll system.

First, create a new directory and initialize your project:

mkdir monnify-payroll-system
cd monnify-payroll-system
npm init -y

Next, install the required dependencies:

npm install express cors helmet dotenv axios bull ioredis pg swagger-jsdoc swagger-ui-express express-validator

Then install the development dependencies:

npm install -D typescript ts-node-dev @types/node @types/express @types/cors @types/pg @types/bull

Create a tsconfig.json file:

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020"],
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*", "scripts/**/*"],
  "exclude": ["node_modules", "dist"]
}

And update your package.json scripts:

{
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js",
    "dev": "ts-node-dev --respawn --transpile-only src/index.ts",
    "migrate": "ts-node scripts/run-migrations.ts"
  }
}

Now, create a .env file for your environment variables. All the Monnify env details can be gotten in this route:

# Server
PORT=3008
NODE_ENV=development

# Database
DB_HOST=localhost
DB_PORT=5433
DB_NAME=payroll_db
DB_USER=payroll_user
DB_PASSWORD=payroll_password

# Redis
REDIS_HOST=localhost
REDIS_PORT=6379

# Monnify
MONNIFY_API_KEY=your_api_key
MONNIFY_SECRET_KEY=your_secret_key
MONNIFY_BASE_URL=https://sandbox.monnify.com
MONNIFY_CONTRACT_CODE=your_contract_code
MONNIFY_WEBHOOK_SECRET=your_webhook_secret

Configuring Docker for PostgreSQL and Redis

Before we can start building our application, we need to set up the infrastructure services: PostgreSQL for data persistence and Redis for job queue management. Using Docker Compose makes it easy to run these services locally with a single command. This approach ensures consistency across development environments and simplifies deployment.

Create a docker-compose.yml file to set up PostgreSQL and Redis:

services:
  postgres:
    image: postgres:15-alpine
    container_name: monnify-payroll-db
    environment:
      POSTGRES_USER: payroll_user
      POSTGRES_PASSWORD: payroll_password
      POSTGRES_DB: payroll_db
    ports:
      - '5433:5432'
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U payroll_user']
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    container_name: monnify-payroll-redis
    ports:
      - '6379:6379'
    volumes:
      - redis_data:/data
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  postgres_data:
  redis_data:

Start the services:

docker-compose up -d

And verify that both services are running:

docker-compose ps

Setting Up the Database

Now we'll configure the database connection and create the necessary tables. We'll use a connection pool for efficient database access and create migration files to set up our schema. This approach ensures our database structure is version-controlled and can be easily reproduced.

Create the src/config/database.ts file to configure the PostgreSQL connection:

import { Pool, PoolConfig } from 'pg';
import dotenv from 'dotenv';

dotenv.config();

const dbName = (process.env.DB_NAME || 'payroll_db').trim();
if (!dbName) {
  throw new Error('Database name (DB_NAME) must be set and cannot be empty');
}

const config: PoolConfig = {
  host: process.env.DB_HOST || 'localhost',
  port: parseInt(process.env.DB_PORT || '5433'),
  database: dbName,
  user: process.env.DB_USER || 'payroll_user',
  password: process.env.DB_PASSWORD || 'payroll_password',
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
};

export const pool = new Pool(config);

pool.on('error', (err: Error) => {
  console.error('Unexpected error on idle client', err);
  process.exit(-1);
});

export const query = async (text: string, params?: any[]) => {
  const start = Date.now();
  try {
    const res = await pool.query(text, params);
    return res;
  } catch (error) {
    console.error('Database query error', error);
    throw error;
  }
};

Now create the migration files. First, create a migrations folder:

mkdir migrations

Then create migrations/001_create_employees_table.sql:

-- Create employees table
CREATE TABLE IF NOT EXISTS employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  email VARCHAR(255) NOT NULL UNIQUE,
  employee_id VARCHAR(100) NOT NULL UNIQUE,
  salary DECIMAL(15, 2) NOT NULL,
  account_number VARCHAR(50) NOT NULL,
  bank_code VARCHAR(20) NOT NULL,
  bank_name VARCHAR(255),
  is_active BOOLEAN DEFAULT true,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create indexes for faster lookups
CREATE INDEX IF NOT EXISTS idx_employees_employee_id ON employees(employee_id);
CREATE INDEX IF NOT EXISTS idx_employees_is_active ON employees(is_active);

Now, create migrations/002_create_payrolls_table.sql:

-- Create payrolls table
CREATE TABLE IF NOT EXISTS payrolls (
  id SERIAL PRIMARY KEY,
  payroll_period VARCHAR(100) NOT NULL,
  total_amount DECIMAL(15, 2) NOT NULL,
  total_employees INTEGER NOT NULL,
  status VARCHAR(50) NOT NULL DEFAULT 'pending',
  processed_count INTEGER DEFAULT 0,
  failed_count INTEGER DEFAULT 0,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  processed_at TIMESTAMP
);

-- Create indexes for faster queries
CREATE INDEX IF NOT EXISTS idx_payrolls_status ON payrolls(status);
CREATE INDEX IF NOT EXISTS idx_payrolls_period ON payrolls(payroll_period);

And next, create migrations/003_create_payroll_items_table.sql:

-- Create payroll_items table
CREATE TABLE IF NOT EXISTS payroll_items (
  id SERIAL PRIMARY KEY,
  payroll_id INTEGER NOT NULL REFERENCES payrolls(id) ON DELETE CASCADE,
  employee_id INTEGER NOT NULL REFERENCES employees(id) ON DELETE CASCADE,
  amount DECIMAL(15, 2) NOT NULL,
  status VARCHAR(50) NOT NULL DEFAULT 'pending',
  transaction_reference VARCHAR(255),
  error_message TEXT,
  processed_at TIMESTAMP,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create indexes for faster queries
CREATE INDEX IF NOT EXISTS idx_payroll_items_payroll_id ON payroll_items(payroll_id);
CREATE INDEX IF NOT EXISTS idx_payroll_items_employee_id ON payroll_items(employee_id);
CREATE INDEX IF NOT EXISTS idx_payroll_items_status ON payroll_items(status);
CREATE INDEX IF NOT EXISTS idx_payroll_items_transaction_ref ON payroll_items(transaction_reference);

Then create a migration runner script at scripts/run-migrations.ts:

import fs from 'fs';
import path from 'path';
import { pool } from '../src/config/database';

async function runMigrations() {
  const migrationsDir = path.join(__dirname, '../migrations');
  const files = fs.readdirSync(migrationsDir).sort();

  for (const file of files) {
    if (file.endsWith('.sql')) {
      console.log(`Running migration: ${file}`);
      const sql = fs.readFileSync(path.join(migrationsDir, file), 'utf-8');
      await pool.query(sql);
      console.log(`Completed: ${file}`);
    }
  }

  console.log('All migrations completed');
  await pool.end();
}

runMigrations().catch((err) => {
  console.error('Migration failed:', err);
  process.exit(1);
});

Run the migrations:

npm run migrate

Creating Database Models

In this section, we'll create the data access layer for our payroll system. Models encapsulate all database operations, providing a clean interface for the rest of the application. We'll build two main models: one for managing employees and another for handling payrolls and payroll items.

For each model, I’ll first explain its purpose and key methods, then show you the complete code implementation. This approach helps you understand what each model does before you implement it.

Employee Model

The EmployeeModel serves as the data-access layer for employee records. It handles creating, reading, updating, and soft-deleting employees. The model includes automatic employee ID generation (format: EMP001, EMP002, and so on) and ensures that each employee has the banking details required for payroll disbursement.

Start by creating a new file at src/models/employee.ts where we’ll implement all employee-related database logic.

After creating the file, import a shared database query helper to execute parameterized SQL safely.

import { query } from '../config/database';

This keeps raw SQL isolated from controllers and ensures protection against SQL injection.

Employee Data Structure (Employee Interface)

Next, we’ll define the employee interface.

The Employee interface represents a row in the employees database table and captures both operational and audit fields. It includes identifying fields (`id`, employee_id), personal fields (`name`, email), payroll fields (`salary`), banking details (`account_number`, bank_code, bank_name), operational state (`is_active`), and timestamps (`created_at`, updated_at). The is_active flag is used to support soft deletion and employee deactivation without permanently removing historical payroll relationships.

export interface Employee {
  id: number;
  name: string;
  email: string;
  employee_id: string;
  salary: number;
  account_number: string;
  bank_code: string;
  bank_name: string;
  is_active: boolean;
  created_at: Date;
  updated_at: Date;
}

Now, we’ll define the CreateEmployeeInput interface which represent the expected payload for creating an employee. It includes required fields such as name, email, salary, and bank details.

export interface CreateEmployeeInput {
  name: string;
  email: string;
  employee_id?: string;
  salary: number;
  account_number: string;
  bank_code: string;
  bank_name: string;
}

The employee_id field is optional, allowing the system to auto-generate a unique identifier if one is not provided. This flexibility supports both automated workflows and manual HR data imports.

Employee Model Class

Next, we’ll define the EmployeeModel class.

export class EmployeeModel {
  // Class methods will go here
}

This class encapsulates all database operations related to employee records. It centralizes logic for creating, retrieving, updating, and deleting employees, as well as generating unique sequential employee IDs.

Auto-Generating Employee IDs (generateEmployeeId)

We start by creating the generateEmployeeId method inside the EmployeeModel class.

  private static async generateEmployeeId(): Promise<string> {
    // Get the highest existing employee_id number that matches EMP### pattern
    const result = await query(
      `SELECT employee_id FROM employees
       WHERE employee_id LIKE 'EMP%'
       AND LENGTH(employee_id) >= 4
       AND SUBSTRING(employee_id FROM 4) ~ '^[0-9]+$'
       ORDER BY CAST(SUBSTRING(employee_id FROM 4) AS INTEGER) DESC
       LIMIT 1`
    );

    if (result.rows.length === 0) {
      return 'EMP001';
    }

    const lastId = result.rows[0].employee_id;
    const numberPart = lastId.substring(3);
    const lastNumber = parseInt(numberPart, 10);

    if (isNaN(lastNumber)) {
      return 'EMP001';
    }

    const nextNumber = lastNumber + 1;
    // Format as EMP001, EMP002, etc. (3 digits minimum)
    return `EMP${nextNumber.toString().padStart(3, '0')}`;
  }

The private generateEmployeeId method generates a unique employee identifier in a readable sequential format such as EMP001, EMP002, and so on. It queries the database for the highest existing employee ID that matches the expected pattern (EMP prefix followed by numeric digits), orders by the numeric suffix in descending order, and increments the latest number to produce the next ID.

If no matching record exists, it starts from EMP001. The method also protects against malformed data by returning EMP001 if parsing fails.

Finally, it ensures formatting consistency by padding the number portion to at least three digits using padStart(3, '0'), which keeps IDs aligned and easy to sort visually.

Creating an Employee (create)

Next, we’ll define the create method, which inserts a new employee record into the database. If the caller does not supply an employee_id, the method generates one automatically using generateEmployeeId.

  static async create(data: CreateEmployeeInput): Promise<Employee> {
    // Auto-generate employee_id if not provided
    let employeeId = data.employee_id;
    if (!employeeId) {
      employeeId = await this.generateEmployeeId();
    }

    // Check if employee_id already exists (if manually provided)
    if (data.employee_id) {
      const existing = await this.findByEmployeeId(data.employee_id);
      if (existing) {
        throw new Error('Employee ID already exists');
      }
    }

    const result = await query(
      `INSERT INTO employees (name, email, employee_id, salary, account_number, bank_code, bank_name)
       VALUES ($1, $2, $3, $4, $5, $6, $7)
       RETURNING *`,
      [
        data.name,
        data.email,
        employeeId,
        data.salary,
        data.account_number,
        data.bank_code,
        data.bank_name,
      ]
    );
    return result.rows[0];
  }

Here’s what’s happening in the code:

If an employee_id is manually provided, it validates uniqueness by checking if that ID already exists among active employees, preventing collisions and ensuring each employee has a distinct identifier. After validations, the employee is inserted into the employees table and the new record is returned. This method ensures every employee created has complete banking details required for payroll disbursement.

Retrieving All Active Employees (findAll)

The findAll method fetches all active employees from the database.

static async findAll(): Promise<Employee[]> {
  const result = await query(
    'SELECT * FROM employees WHERE is_active = true ORDER BY created_at DESC'
  );
  return result.rows;
}

The findAll method returns all active employees (is_active = true) ordered by most recent creation date. This behavior supports common UI patterns such as HR dashboards and payroll selection screens, where only active employees should be visible by default.

Retrieving an Employee by Database ID (findById)

The findById method retrieves a single employee by the internal numeric primary key (id).

static async findById(id: number): Promise<Employee | null> {
  const result = await query('SELECT * FROM employees WHERE id = $1', [id]);
  return result.rows[0] || null;
}

If the employee does not exist, it returns null. This is typically used for internal operations such as payroll processing, updates, or admin detail views.

Retrieving an Employee by Employee Identifier (findByEmployeeId)

The findByEmployeeId method retrieves an active employee using the business-friendly employee_id (for example, EMP014).

static async findByEmployeeId(employeeId: string): Promise<Employee | null> {
    const result = await query(
      'SELECT * FROM employees WHERE employee_id = $1 AND is_active = true',
      [employeeId]
    );
    return result.rows[0] || null;
}

The method filters by is_active = true to prevent selecting deactivated employees during operations like payroll runs or HR searches.

Updating an Employee (update)

The update method supports partial updates by dynamically building the SQL SET clause based on the fields present in the update payload. It iterates through the provided properties, includes only those with defined values, and constructs a parameterized query to prevent SQL injection and preserve correctness.

static async update(
    id: number,
    data: Partial<CreateEmployeeInput>
  ): Promise<Employee> {
    const fields: string[] = [];
    const values: any[] = [];
    let paramCount = 1;

    // Build dynamic update query based on provided fields
    Object.entries(data).forEach(([key, value]) => {
      if (value !== undefined) {
        fields.push(`${key} = $${paramCount}`);
        values.push(value);
        paramCount++;
      }
    });

    if (fields.length === 0) {
      throw new Error('No fields to update');
    }

    // Always update the updated_at timestamp
    fields.push(`updated_at = $${paramCount}`);
    values.push(new Date());
    values.push(id);

    const result = await query(
      `UPDATE employees SET ${fields.join(', ')} WHERE id = $${
        paramCount + 1
      } RETURNING *`,
      values
    );
    return result.rows[0];
  }

Here’s what’s happening in the code:

If no fields are provided, it throws an error to avoid performing a meaningless update. It also explicitly updates the updated_at timestamp to ensure accurate audit tracking. Finally, it returns the updated database record, making it easy for controllers to respond with the latest employee state.

Soft-Deleting an Employee (delete)

Finally, instead of permanently removing the employee record, the delete method performs a soft delete by setting is_active = false and updating the updated_at timestamp.

This approach preserves historical payroll references and audit trails while excluding inactive employees from standard queries like findAll. It’s especially important in payroll systems where historical payment records must remain valid and traceable even after an employee leaves the organization.

static async delete(id: number): Promise<void> {
  await query(
    'UPDATE employees SET is_active = false, updated_at = NOW() WHERE id = $1',
    [id]
  );
}

Key features of the employee model:

• Auto-generates sequential employee IDs if not provided

• Validates employee ID uniqueness

• Supports soft deletion to preserve historical payroll records

• Provides methods for finding employees by database ID or employee identifier

Payroll Model

The PayrollModel manages payroll batches and individual payroll items. A payroll represents a single payment cycle (for example, "December 2024"), while payroll items represent individual employee payments within that cycle. This separation allows us to track the status of each payment independently.

Key features:

• Creates payroll batches with automatic calculation of totals

• Supports filtering employees for selective payroll runs

• Tracks status at both batch and item levels

• Provides methods for reconciliation and status updates

Let's implement the Payroll Model.

We’ll begin by creating a new file at src/models/payroll.ts, where we’ll implement the payroll models that encapsulate payroll batch creation, employee payment tracking, and payroll status management.

First, import a shared database query helper to execute parameterized SQL safely.

import { query } from '../config/database';

This keeps raw SQL isolated from controllers and ensures protection against SQL injection.

Payroll Status Lifecycle

Next, we’ll define the PayrollStatus enum.

export enum PayrollStatus {
 PENDING = 'pending',
 PROCESSING = 'processing',
 COMPLETED = 'completed',
 FAILED = 'failed',
 PARTIALLY_COMPLETED = 'partially_completed',
}

The PayrollStatus enum defines all possible states for both payroll batches and individual payroll items:

• PENDING – Created but not yet processed

• PROCESSING – Currently being processed by background workers

• COMPLETED – Successfully processed

• FAILED – Processing failed

• PARTIALLY_COMPLETED – Some items succeeded while others failed

Payroll Entity

With the payroll status lifecycle defined, we can now define the payroll entity.

The Payroll interface represents a single payroll run, such as a monthly salary payout. It stores aggregate and audit information including the payroll period, total salary amount, total number of employees, processing status, counts of successful and failed payments, and timestamps for creation, updates, and completion.

Add the following interface to src/models/payroll.ts:

export interface Payroll {
 id: number;
 payroll_period: string;
 total_amount: number;
 total_employees: number;
 status: PayrollStatus;
 processed_count: number;
 failed_count: number;
 created_at: Date;
 updated_at: Date;
 processed_at?: Date;
}

This entity acts as the parent record for all employee payments within a payroll cycle and is used to track overall payroll progress and outcomes.

Payroll Item Entity

Next, we’ll define the payroll item entity, which represents an individual employee payment within a payroll.

The PayrollItem tracks the employee being paid, the payment amount, its processing status, any transaction reference returned by the payment provider, error messages in case of failure, and relevant timestamps.

Add the following interface just below the Payroll interface:

export interface PayrollItem {
  id: number;
  payroll_id: number;
  employee_id: number;
  amount: number;
  status: PayrollStatus;
  transaction_reference?: string;
  error_message?: string;
  processed_at?: Date;
  created_at: Date;
  updated_at: Date;
}

This structure allows individual employee payments to be retried, audited, or reconciled independently without affecting the rest of the payroll batch.

Creating a Payroll (PayrollModel.create)

Now that we’ve defined the Payroll and PayrollItem entities, we can move on to creating a payroll batch.

To keep our business logic organized, we’ll introduce a PayrollModel class. This class will be responsible for creating payroll records, calculating aggregates, and generating individual payroll items for each employee.

Before writing the model itself, let’s define the input required to create a payroll.

Add the following interface below the PayrollItem interface:

export interface CreatePayrollInput {
  payroll_period: string;
  employee_ids?: number[];
}

• payroll_period identifies the payroll run (for example, 2025-01)

• employee_ids is optional and allows us to run payroll for a subset of employees, enabling selective payouts or retries

Next, create the PayrollModel class. This class will encapsulate all payroll-related database operations.

export class PayrollModel {
// Payroll model class methods will go here
}

We’ll start by implementing the create method, which is responsible for creating a new payroll batch.

The method performs the following steps:

1. Optionally filters employees if specific employee IDs are provided

2. Calculates aggregate payroll statistics from the employees table

3. Creates a payroll record with a PENDING status

4. Creates a payroll item for each eligible employee

Here’s the implementation:

  static async create(data: CreatePayrollInput): Promise<Payroll> {
    let employeeFilter = '';
    let queryParams: any[] = [];

    // Build filter for selective employee payrolls
    if (data.employee_ids && data.employee_ids.length > 0) {
      employeeFilter = `AND id = ANY($1::int[])`;
      queryParams = [data.employee_ids];
    }

    // Calculate aggregate statistics from employees table
    const employeeStats = await query(
      `SELECT COUNT(*) as count, COALESCE(SUM(salary), 0) as total
       FROM employees
       WHERE is_active = true ${employeeFilter}`,
      queryParams
    );

    const totalEmployees = parseInt(employeeStats.rows[0].count);
    const totalAmount = parseFloat(employeeStats.rows[0].total);

    // Create the payroll record
    const result = await query(
      `INSERT INTO payrolls (payroll_period, total_amount, total_employees, status)
       VALUES ($1, $2, $3, $4)
       RETURNING *`,
      [data.payroll_period, totalAmount, totalEmployees, PayrollStatus.PENDING]
    );

    const payroll = result.rows[0];

    // Create payroll items for each employee
    // Each item starts with PENDING status and will be processed asynchronously
    const employees = await query(
      `SELECT id, salary FROM employees WHERE is_active = true ${employeeFilter}`,
      queryParams
    );

    for (const employee of employees.rows) {
      await query(
        `INSERT INTO payroll_items (payroll_id, employee_id, amount, status)
         VALUES ($1, $2, $3, $4)`,
        [payroll.id, employee.id, employee.salary, PayrollStatus.PENDING]
      );
    }

    return payroll;
  }

The payroll creation process begins by determining which employees should be included. If specific employee IDs are provided, only those employees are selected – otherwise, all active employees are included. This allows the system to support both full payroll runs and selective payouts.

Next, the system calculates aggregate payroll statistics directly from the employees table by counting eligible employees and summing their salaries. These values are stored in a new payroll record created with a PENDING status.

Finally, a payroll item is generated for each eligible employee, with each item also initialized in a PENDING state. This design separates payroll setup from payment execution, allowing employee payments to be processed asynchronously and in parallel in later stages of the system.

Fetching Payroll Records

After creating payrolls, we often need to retrieve them for administrative dashboards, reporting, and audit trails.

The PayrollModel provides two simple methods:

1. findById – Retrieves a single payroll by its unique identifier

2. findAll – Retrieves all payroll records, ordered by creation date (newest first)

These methods should be added below the create method in the PayrollModel class:

static async findById(id: number): Promise<Payroll | null> {
  const result = await query('SELECT * FROM payrolls WHERE id = $1', [id]);
  return result.rows[0] || null;
}

static async findAll(): Promise<Payroll[]> {
  const result = await query(
    'SELECT * FROM payrolls ORDER BY created_at DESC'
  );
  return result.rows;
}

The findById method retrieves a single payroll by its identifier, while findAll returns all payroll records ordered by creation date.

Updating Payroll Status (PayrollModel.updateStatus)

Once payroll processing begins, we need a way to track the overall status of a payroll batch. The updateStatus method updates the payroll record with:

• The current status (PENDING, PROCESSING, COMPLETED, and so on)

• Optional counts of processed and failed payments

• A processed_at timestamp automatically set for terminal states (COMPLETED or PARTIALLY_COMPLETED)

Add the following method below the fetch methods in your PayrollModel class:

  static async updateStatus(
    id: number,
    status: PayrollStatus,
    processedCount?: number,
    failedCount?: number
  ): Promise<Payroll> {
    const updates: string[] = ['status = $2', 'updated_at = NOW()'];
    const values: any[] = [id, status];

    // Dynamically add processed_count if provided
    if (processedCount !== undefined) {
      updates.push(`processed_count = $${values.length + 1}`);
      values.push(processedCount);
    }

    // Dynamically add failed_count if provided
    if (failedCount !== undefined) {
      updates.push(`failed_count = $${values.length + 1}`);
      values.push(failedCount);
    }

    // Set processed_at timestamp for terminal states
    if (
      status === PayrollStatus.COMPLETED ||
      status === PayrollStatus.PARTIALLY_COMPLETED
    ) {
      updates.push(`processed_at = NOW()`);
    }

    const result = await query(
      `UPDATE payrolls SET ${updates.join(', ')} WHERE id = $1 RETURNING *`,
      values
    );
    return result.rows[0];
  }
}

As payroll processing progresses, this method updates the overall payroll status along with optional counts of processed and failed payments. When a payroll reaches a terminal state such as COMPLETED or PARTIALLY_COMPLETED, the system automatically records a completion timestamp. This ensures accurate tracking of payroll execution and supports reconciliation workflows.

PayrollItemModel

After handling payroll batches with PayrollModel, we need a way to manage individual employee payments. This is where the PayrollItemModel comes in. It encapsulates database operations related to payroll items, including fetching, and updating records with employee details.

Start by adding a new class below PayrollModel:

export class PayrollItemModel {
  // Methods will go here
}

Fetching Payroll Items (PayrollItemModel.findByPayrollId)

Often, we want to get all payroll items for a specific payroll batch. For example, to display them on a dashboard or process them in a background worker.

This findByPayrollId method does that exactly. It retrieves all payroll items associated with a specific payroll and enriches them with employee details such as name, bank account number, and bank information through a database join.

static async findByPayrollId(payrollId: number): Promise<PayrollItem[]> {
  const result = await query(
    `SELECT
       pi.id, pi.payroll_id, pi.employee_id, pi.amount, pi.status,
       pi.transaction_reference, pi.error_message, pi.processed_at,
       pi.created_at, pi.updated_at,
       e.name as employee_name, e.employee_id as employee_identifier,
       e.account_number, e.bank_code, e.bank_name
     FROM payroll_items pi
     JOIN employees e ON pi.employee_id = e.id
     WHERE pi.payroll_id = $1
     ORDER BY pi.created_at`,
      [payrollId]
    );
    // Normalize numeric fields from PostgreSQL (which returns them as strings)
    return result.rows.map((row) => ({
      ...row,
      employee_id: parseInt(row.employee_id, 10),
      id: parseInt(row.id, 10),
      payroll_id: parseInt(row.payroll_id, 10),
      amount: parseFloat(row.amount),
    }));
  }

Here’s what’s happening in the code:

1. We use a JOIN with the employees table so each payroll item includes the employee’s name, account number, and bank information.

2. Some numeric fields may come as strings, so we convert them to proper JavaScript numbers (parseInt / parseFloat) for accurate calculations and display.

3. The results are ordered by creation date, which helps when rendering items in a UI or processing them sequentially.

This method makes it easy to work with all items in a payroll batch while keeping the data enriched and consistent.

Fetching a Single Payroll Item (PayrollItemModel.findById)

Sometimes, you need to look at one specific employee’s payment (for example, to retry a failed transaction or investigate an issue). The findById method helps in fetching a single payroll item along with the employee’s details, so you have everything you need in one place.

Here’s how we implement it:

static async findById(id: number): Promise<PayrollItem | null> {
  const result = await query(
    `SELECT
       pi.id, pi.payroll_id, pi.employee_id, pi.amount, pi.status,
       pi.transaction_reference, pi.error_message, pi.processed_at,
       pi.created_at, pi.updated_at,
       e.name as employee_name, e.employee_id as employee_identifier,
       e.account_number, e.bank_code, e.bank_name
     FROM payroll_items pi
     JOIN employees e ON pi.employee_id = e.id
     WHERE pi.id = $1`,
    [id]
  );

  if (result.rows.length === 0) return null;

  const row = result.rows[0];

  // Convert numeric fields to proper JavaScript numbers for easier calculations and display
  return {
    ...row,
    employee_id: parseInt(row.employee_id, 10),
    id: parseInt(row.id, 10),
    payroll_id: parseInt(row.payroll_id, 10),
    amount: parseFloat(row.amount),
  };
}

Here’s what’s happening in the code:

• We use a JOIN with the employees table to include employee info such as name, account number, and bank details.

• If the ID doesn’t exist, the method returns null so you can handle missing records gracefully.

• Numeric fields are converted to JavaScript numbers, making it easy to calculate totals or display amounts in the UI.

This method ensures that whenever you need a single payroll item, you get a complete, ready-to-use record.

Updating Payroll Item Status (PayrollItemModel.updateStatus)

As individual employee payments are processed, this method updates the payroll item’s status, stores transaction references from external payment providers, captures error messages on failure, and timestamps completion or failure events. This fine-grained tracking enables reliable retries, audits, and reconciliation with external payment systems.

Here’s the implementation:

static async updateStatus(
  id: number,
  status: PayrollStatus,
  transactionReference?: string,
  errorMessage?: string
): Promise<PayrollItem> {
  const updates: string[] = ['status = $2', 'updated_at = NOW()'];
  const values: any[] = [id, status];

  // Add transaction reference if provided (from Monnify API response)
  if (transactionReference) {
    updates.push(`transaction_reference = $${values.length + 1}`);
    values.push(transactionReference);
  }

  // Add error message if provided (from failed payment)
  if (errorMessage) {
    updates.push(`error_message = $${values.length + 1}`);
    values.push(errorMessage);
  }

  // Set processed_at timestamp for terminal states
  if (status === PayrollStatus.COMPLETED || status === PayrollStatus.FAILED) {
    updates.push(`processed_at = NOW()`);
  }

  const result = await query(
    `UPDATE payroll_items SET ${updates.join(
      ', '
     )} WHERE id = $1 RETURNING *`,
     values
   );
   return result.rows[0];
 }
}

Here’s what’s happening in the code:

• We build a dynamic SET clause to update only the fields provided – status is required, while transaction reference and error message are optional.

• Terminal states (COMPLETED or FAILED) trigger an automatic timestamp on processed_at, so we always know when a payment finished.

• The method returns the updated payroll item, ready for further processing, logging, or UI display.

This ensures each payroll item is tracked accurately throughout its lifecycle, enabling reliable retries and complete audit trails.

Overall Payroll Flow

In this payroll flow, an administrator creates a payroll batch, which generates individual payroll items for each employee. The payroll is then handed off to background workers that process each payroll item independently via an external payment service.

As each payment succeeds or fails, payroll items are updated accordingly. Once processing concludes, the payroll batch status is updated to reflect the overall outcome, whether fully successful, partially successful, or failed.

This architecture provides scalability, resilience, and strong auditability for real-world payroll systems.

Building the Monnify Client

The Monnify client is the bridge between our application and Monnify's payment API. In this section, we'll build a reusable client that handles authentication, bulk transfers, and transaction tracking. The client automatically manages API tokens, retries failed requests, and provides a clean interface for the rest of our application.

This module implements a reusable Monnify API client responsible for handling authentication, bulk payroll disbursements, authorization, transaction tracking, and balance checks in a secure and production-ready manner. It abstracts all Monnify-specific logic behind a single class, making it easy to integrate into background jobs, payroll processors, or service layers.

We’ll begin by creating a new file at src/config/monnify.ts where we’ll implement the Monnify client.

Configuration and Environment Setup

Start by loading the configuration from environment variables using dotenv, ensuring that sensitive credentials are never hardcoded. These include the Monnify API key, secret key, base URL, and contract code (wallet account number). This setup allows the same client to be safely used across development, staging, and production environments.

import axios, { AxiosInstance } from 'axios';
import dotenv from 'dotenv';

dotenv.config();

export interface MonnifyConfig {
  apiKey: string;
  secretKey: string;
  baseUrl: string;
  contractCode: string;
}

Create the MonnifyClient Class

Next, you’ll define the MonnifyClient class. This class encapsulates all communication with the Monnify API. It internally manages API credentials, an Axios HTTP client, an access token, and token expiry tracking.

export class MonnifyClient {
  private readonly apiKey: string;
  private readonly secretKey: string;
  private baseUrl: string;
  private contractCode: string;
  private client: AxiosInstance;

  private accessToken: string | null = null;
  private tokenExpiry: number = 0;

This design ensures authentication is handled transparently and automatically for every request.

Axios Client and Request Interceptor

Inside the constructor, initialize the Monnify client with credentials from environment variables. The Axios instance is created with the Monnify base URL and JSON headers.

  constructor() {
    this.apiKey = process.env.MONNIFY_API_KEY || '';
    this.secretKey = process.env.MONNIFY_SECRET_KEY || '';
    this.baseUrl = process.env.MONNIFY_BASE_URL || 'https://api.monnify.com';
    this.contractCode = process.env.MONNIFY_CONTRACT_CODE || '';

    this.client = axios.create({
      baseURL: this.baseUrl,
      headers: {
        'Content-Type': 'application/json',
      },
    });

We attach the request interceptor to this client to automatically inject a valid Bearer token into every outgoing request (except the authentication endpoint). Before each request, the interceptor ensures the client is authenticated, preventing unauthorized requests and eliminating token-related boilerplate across the codebase.

    this.client.interceptors.request.use(async (config: any) => {
      // Skip auth for the login endpoint itself
      if (config.url?.includes('/auth/login')) {
        return config;
      }

      // Ensure a valid token exists before every request
      await this.ensureAuthenticated();

      if (this.accessToken) {
        config.headers.Authorization = `Bearer ${this.accessToken}`;
      }

      return config;
    });
  }

Authenticate with Monnify

Authentication is handled using Monnify’s Basic Auth mechanism, where the API key and secret key are base64-encoded and sent to the /auth/login endpoint. Upon successful authentication, the client stores the returned access token and sets an internal expiry timestamp slightly below the official token lifetime to avoid edge-case expirations. Any authentication failure is logged and surfaced as a controlled error to prevent silent failures.

  private async authenticate(): Promise<void> {
    try {
      // Encode credentials as Base64 for Basic Auth
      const credentials = Buffer.from(
        `${this.apiKey}:${this.secretKey}`
      ).toString('base64');

      const response = await axios.post(
        `${this.baseUrl}/api/v1/auth/login`,
        {},
        {
          headers: {
            Authorization: `Basic ${credentials}`,
            'Content-Type': 'application/json',
          },
        }
      );

      this.accessToken = response.data.responseBody.accessToken;
      // Set expiry to 23 hours (Monnify tokens typically last 24 hours)
      // This prevents edge cases where token expires mid-request
      this.tokenExpiry = Date.now() + 23 * 60 * 60 * 1000;
    } catch (error: any) {
      console.error(
        'Monnify authentication error:',
        error.response?.data || error.message
      );
      throw new Error('Failed to authenticate with Monnify');
    }
  }

Automatic Token Refresh (ensureAuthenticated)

Before any API call, the client verifies whether a valid access token exists or if the token has expired. If so, it transparently re-authenticates.

  private async ensureAuthenticated(): Promise<void> {
    if (!this.accessToken || Date.now() >= this.tokenExpiry) {
      await this.authenticate();
    }
  }

This ensures that long-running processes such as payroll queues or background workers can safely make Monnify requests without manual token handling.

Initiating Bulk Transfers

The initiateBulkTransfer method handles the creation of a bulk disbursement batch, typically used for payroll payments. It validates input transfers to ensure each payment has a valid amount, destination account number, and bank code.

A structured batch request is then constructed, including a unique batch reference, source account (contract code), narration, and a list of transactions. The request is logged for traceability and sent to Monnify’s batch disbursement endpoint. Any API error is normalized and returned with meaningful messaging to aid debugging and retries.

  async initiateBulkTransfer(
    transfers: Array<{
      amount: number;
      recipientAccountNumber: string;
      recipientBankCode: string;
      recipientName: string;
      narration: string;
      reference: string;
    }>
  ): Promise<any> {
    await this.ensureAuthenticated();

We validate inputs early to fail fast:

    if (!transfers || transfers.length === 0) {
      throw new Error('No transfers provided');
    }

    if (!this.contractCode) {
      throw new Error('Monnify contract code is not configured');
    }

Each transfer is validated individually:

    for (const transfer of transfers) {
      if (!transfer.amount || transfer.amount <= 0) {
        throw new Error(`Invalid amount for transfer: ${transfer.reference}`);
      }
      if (!transfer.recipientAccountNumber) {
        throw new Error(`Missing account number for transfer: ${transfer.reference}`);
      }
      if (!transfer.recipientBankCode) {
        throw new Error(`Missing bank code for transfer: ${transfer.reference}`);
      }
    }

We then construct the batch payload:

    const requestBody = {
      title: 'Bulk Payroll Transfers',
      batchReference: `BATCH_${Date.now()}`,
      narration: 'Payroll batch disbursement',
      sourceAccountNumber: this.contractCode,
      onValidationFailure: 'CONTINUE',
      notificationInterval: 50,
      transactionList: transfers.map((t) => ({
        amount: t.amount,
        reference: t.reference,
        narration: t.narration,
        destinationBankCode: t.recipientBankCode,
        destinationAccountNumber: t.recipientAccountNumber,
        currency: 'NGN',
      })),
    };

Finally, we send the request and normalize errors:

    try {
      const response = await this.client.post(
        '/api/v2/disbursements/batch',
        requestBody
      );
      return response.data;
    } catch (error: any) {
      const errorData = error.response?.data;
      const message =
        errorData?.responseMessage ||
        errorData?.message ||
        `Monnify API error (${error.response?.status})`;
      throw new Error(message);
    }
  }

Authorizing Bulk Transfers (OTP Validation)

Some bulk transfers require OTP authorization. The authorizeBulkTransfer method validates the presence of a batch reference and authorization code before submitting them to Monnify’s OTP validation endpoint. This step finalizes the batch disbursement and allows processing to continue. Errors are logged and surfaced clearly for operational visibility.

async authorizeBulkTransfer(
reference: string,
authorizationCode: string
): Promise<any> {
await this.ensureAuthenticated();
    if (!reference) {
      throw new Error('Batch reference is required');
    }

    if (!authorizationCode) {
      throw new Error('Authorization code (OTP) is required');
    }

    const requestBody = {
      reference,
      authorizationCode,
    };

    try {
      const response = await this.client.post(
        '/api/v2/disbursements/batch/validate-otp',
        requestBody
      );

      return response.data;
    } catch (error: any) {
      const errorDetails = error.response?.data || error.message;
      console.error(
        'Monnify authorization error:',
        JSON.stringify(errorDetails, null, 2)
      );

      if (error.response) {
        const errorData = error.response.data;
        const errorMessage =
          errorData?.responseMessage ||
          errorData?.message ||
          `Monnify API error (${error.response.status})`;
        throw new Error(errorMessage);
      }
      throw error;
    }
}

Transaction Status Lookup

The getTransactionStatus method retrieves the real-time status of an individual transaction using its reference.

async getTransactionStatus(transactionReference: string): Promise<any> {
await this.ensureAuthenticated();
    try {
      const response = await this.client.get(
        `/api/v2/disbursements/${transactionReference}/status`
      );
      return response.data;
    } catch (error: any) {
      console.error(
        'Monnify status check error:',
        error.response?.data || error.message
      );
      throw error;
    }
}

This is useful for reconciliation, webhook fallbacks, or manual verification of disbursement outcomes.

Batch Details Retrieval

The getBatchDetails method fetches detailed information about an entire disbursement batch, including the state of individual transactions.

async getBatchDetails(batchReference: string): Promise<any> {
await this.ensureAuthenticated();
    if (!batchReference) {
      throw new Error('Batch reference is required');
    }

    try {
      const response = await this.client.get(
        `/api/v2/disbursements/batch/${batchReference}`
      );
      return response.data;
    } catch (error: any) {
      console.error(
        'Monnify batch details error:',
        error.response?.data || error.message
      );
      throw error;
    }
}

This is particularly useful when reconciling payroll runs or recovering from partial failures.

Wallet Balance Check

Finally, we can query the available balance of the Monnify wallet.

The getAccountBalance method retrieves the available balance of the configured Monnify wallet (contract account).

Create src/config/monnify.ts:

async getAccountBalance(): Promise<any> {
await this.ensureAuthenticated();

    try {
      const response = await this.client.get(
        `/api/v2/disbursements/wallet-balance?accountNumber=${this.contractCode}`
      );
      return response.data;
    } catch (error: any) {
      console.error(
        'Monnify balance check error:',
        error.response?.data || error.message
      );
      throw error;
    }
}

export const monnifyClient = new MonnifyClient();

Key features of this client:

1. Automatic token management: The client automatically handles authentication and refreshes tokens before they expire.

2. Request interceptor: Every API request automatically includes the authentication token.

3. Bulk transfers: Uses Monnify's batch disbursement API for efficient payroll processing.

4. Error handling: Comprehensive error handling with meaningful error messages.

Implementing Background Job Processing

To avoid blocking HTTP requests and to ensure reliable retries, payroll execution is handled asynchronously using a background job processor. This worker is responsible for orchestrating bulk payroll disbursements, coordinating with Monnify, updating payroll and payroll item states, and handling retries safely.

Begin by creating a new file at src/jobs/payroll.processor.ts. All background payroll execution logic will live in this file.

Set Up the Payroll Processing Queue

We’ll create a Bull queue named payroll-processing and a backed by Redis. Redis connection details are loaded from environment variables, allowing flexibility across environments.

Default job options are configured to retry failed jobs up to three times using an exponential backoff strategy. This ensures resilience against transient failures such as network issues or temporary payment gateway downtime. Completed jobs are automatically removed from the queue to keep Redis storage clean.

import Queue from 'bull';
import { monnifyClient } from '../config/monnify';
import {
 PayrollItemModel,
 PayrollModel,
 PayrollStatus,
} from '../models/payroll';
import { EmployeeModel } from '../models/employee';

export const payrollQueue = new Queue('payroll-processing', {
 redis: {
 host: process.env.REDIS_HOST || 'localhost',
 port: Number(process.env.REDIS_PORT || 6379),
},
defaultJobOptions: {
 attempts: 3,
 backoff: { 
  type: 'exponential', 
  delay: 2000 
},
 removeOnComplete: true,
},
});

Queue Processor Registration

The queue registers a processor function using payrollQueue.process, which receives jobs containing a payrollId. Each job triggers the processBulkPayroll function, making the queue responsible for executing one payroll batch at a time.

payrollQueue.process(async (job) => {
 return processBulkPayroll(job.data.payrollId);
});

This design decouples payroll execution from HTTP requests and allows processing to happen asynchronously in background workers.

Bulk Payroll Processing Flow (processBulkPayroll)

When a payroll job is picked up, the system first fetches all payroll items associated with the given payroll ID. It filters out only items that are eligible for processing: those still in a PENDING state or previously marked as PROCESSING but missing a transaction reference.

async function processBulkPayroll(payrollId: number) {

const items = await PayrollItemModel.findByPayrollId(payrollId);

Also, it filters payroll items to include only those that still require processing. This prevents duplicate payments when jobs are retried.

const payable = items.filter(
  (i) =>
    i.status === PayrollStatus.PENDING ||
    (i.status === PayrollStatus.PROCESSING && !i.transaction_reference)
);

if (payable.length === 0) return;

If no payable items remain, the function exits early, avoiding unnecessary API calls.

Once we confirm there are payable items, we update the overall payroll status:

  await PayrollModel.updateStatus(payrollId, PayrollStatus.PROCESSING);

This provides immediate visibility that disbursement is underway.

Building the Bulk Transfer Payload

Create a variable to store the transfer list that will be sent to Monnify.

  const transfers = [];

For each payable payroll item, the corresponding employee record is fetched to retrieve bank and account details. A unique payment reference is generated using the payroll ID and payroll item ID, ensuring traceability across systems. Each payroll item is immediately marked as PROCESSING before initiating payment to prevent concurrent workers from attempting to process the same item.

A transfer object is then constructed containing the payment amount, recipient bank details, narration, and unique reference. These transfer objects are accumulated into a single batch request.

for (const item of payable) {
const employee = await EmployeeModel.findById(item.employee_id);
if (!employee) throw new Error('Employee not found');

    const reference = `PAYROLL_${payrollId}_${item.id}`;

    await PayrollItemModel.updateStatus(item.id, PayrollStatus.PROCESSING);

    transfers.push({
      amount: Number(item.amount),
      reference,
      recipientAccountNumber: employee.account_number,
      recipientBankCode: employee.bank_code,
      recipientName: employee.name,
      narration: `Payroll payment`,
    });

}

Initiating Bulk Disbursement via Monnify

Once all transfers are prepared, the system initiates a bulk transfer through the Monnify client.

const response = await monnifyClient.initiateBulkTransfer(transfers);

if (!response?.requestSuccessful) {
  throw new Error('Bulk transfer initiation failed');
}

If Monnify doesn’t confirm successful initiation, the job throws an error, allowing Bull’s retry mechanism to take over. This ensures failed initiation attempts are retried safely without manual intervention.

Storing Transaction References

After a successful bulk transfer initiation, Monnify returns a list of transactions containing unique transaction references. The system matches each response entry to its corresponding payroll item using the generated reference and updates the payroll item record with the Monnify transaction reference while keeping its status as PROCESSING.

const results = response.responseBody?.transactionList || [];

for (const item of payable) {
const ref = `PAYROLL_${payrollId}_${item.id}`;
const match = results.find((r: any) => r.reference === ref);

    if (match?.transactionReference) {
      await PayrollItemModel.updateStatus(
        item.id,
        PayrollStatus.PROCESSING,
        match.transactionReference
      );
    }

}

await updatePayrollStats(payrollId);
}

This step is critical for later reconciliation through webhooks or status polling.

Payroll Statistics Reconciliation (updatePayrollStats)

After initiating payments, the system recalculates payroll-level statistics by refetching all payroll items.

async function updatePayrollStats(payrollId: number) {
const items = await PayrollItemModel.findByPayrollId(payrollId);

const completed = items.filter(
  (i) => i.status === PayrollStatus.COMPLETED
).length;

The overall payroll status is derived from these counts:

const failed = items.filter((i) => i.status === PayrollStatus.FAILED).length;

let status = PayrollStatus.PROCESSING;

if (completed === items.length) {
  status = PayrollStatus.COMPLETED;
} else if (failed === items.length) {
  status = PayrollStatus.FAILED;
} else if (completed > 0) {
  status = PayrollStatus.PARTIALLY_COMPLETED;
}

 await PayrollModel.updateStatus(payrollId, status, completed, failed);
}

If all items are completed, the payroll is marked as COMPLETED. If all failed, it’s marked as FAILED. If some succeeded and some failed, it’s marked as PARTIALLY_COMPLETED. Otherwise, it remains in PROCESSING. The payroll record is then updated with the new status and aggregate counts, providing an accurate real-time snapshot of payroll execution.

Queue Entry Point (processPayrollItems)

The processPayrollItems function serves as the public entry point for triggering payroll execution.

export async function processPayrollItems(payrollId: number) {
  await payrollQueue.add({ payrollId, type: 'bulk' });
}

It simply enqueues a payroll job with the relevant payroll ID, allowing controllers or services to initiate payroll processing without coupling themselves to queue logic or payment execution details.

Role in the Overall Payroll Architecture

This queue worker acts as the execution engine of the payroll system. It:

• Bridges payroll domain models with the Monnify payment gateway

• Ensures safe retries through Bull’s job management and maintains idempotency

• Continuously synchronizes payroll and payroll item states

By offloading payment execution to background workers, the system achieves scalability, reliability, and operational resilience required for real-world payroll processing.

Key features of the job processor:

1. Exponential backoff: Failed jobs are retried with increasing delays (2s, 4s, 8s).

2. Bulk processing: All payroll items are processed as a single batch transfer.

3. Status tracking: Each item's status is updated throughout the process.

4. Automatic cleanup: Completed jobs are automatically removed from the queue.

Creating the API Controllers

Next, we’ll build the HTTP controller layer for managing employees in the payroll system using Express.js. It exposes RESTful API endpoints that handle incoming requests, perform validation, interact with the employee data model, and return appropriate HTTP responses.

The controller acts as the bridge between client-facing APIs and the underlying business logic encapsulated in the EmployeeModel.

Controller Responsibilities

The EmployeeController is responsible for:

• Validating incoming request data

• Calling the appropriate model methods

• Handling errors gracefully

• Returning meaningful HTTP status codes and JSON responses

Each method follows a consistent structure using try–catch blocks to ensure reliability and simplify error handling.

Start by creating a new file at src/controllers/employee.controller.ts. This file will contain all the endpoints needed to manage employees in the payroll system.

At the top of the file, import the required Express types and the employee model:

import { Request, Response } from 'express';
import { EmployeeModel, CreateEmployeeInput } from '../models/employee';

export class EmployeeController {
  // Controller methods will go here
}

Each method inside this class will map to a specific API endpoint.

Creating an Employee (createEmployee)

We’ll start with an endpoint for creating a new employee.

This endpoint handles the creation of a new employee record. It extracts the request body and validates the presence of required fields such as name, email, salary, bank account number, and bank code.

static async createEmployee(req: Request, res: Response): Promise<void> {
try {
const data: CreateEmployeeInput = req.body;

      if (
        !data.name ||
        !data.email ||
        !data.salary ||
        !data.account_number ||
        !data.bank_code
      ) {
        res.status(400).json({
          error:
            'Missing required fields: name, email, salary, account_number, bank_code',
        });
        return;
      }

      const employee = await EmployeeModel.create(data);
      res.status(201).json({
        message: 'Employee created successfully',
        data: employee,
      });
    } catch (error: any) {
      console.error('Error creating employee:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to create employee' });
    }

}

If any required field is missing, the request is rejected with a 400 Bad Request response.

Upon successful validation, the controller delegates employee creation to the EmployeeModel.create method and returns a 201 Created response containing the newly created employee. Any unexpected error during the process results in a 500 Internal Server Error.

Fetching All Employees (getAllEmployees)

Next, we’ll add an endpoint for retrieving all employee records from the system.

This endpoint simply calls EmployeeModel.findAll and returns the result as a JSON response. This API is typically used for administrative dashboards, payroll preparation, or reporting purposes.

static async getAllEmployees(req: Request, res: Response): Promise<void> {
  try {
    const employees = await EmployeeModel.findAll();
    res.json({ data: employees });
  } catch (error: any) {
    console.error('Error fetching employees:', error);
    res
      .status(500)
      .json({ error: error.message || 'Failed to fetch employees' });
  }
}

If the retrieval is successful, the controller responds with the full list of employees. If something goes wrong, such as a database or unexpected runtime error, the error is logged and a 500 Internal Server Error is returned to the client.

Fetching a Single Employee (getEmployeeById)

After listing all employees, the next logical step is being able to fetch a single employee by their ID.

This endpoint retrieves a specific employee by ID, which is parsed from the URL parameters. If the employee doesn’t exist, the controller responds with a 404 Not Found. Otherwise, the employee data is returned in a successful response. This endpoint is useful for viewing or editing individual employee details.

  static async getEmployeeById(req: Request, res: Response): Promise<void> {
    try {
      const { id } = req.params;
      const employee = await EmployeeModel.findById(parseInt(id));

      if (!employee) {
        res.status(404).json({ error: 'Employee not found' });
        return;
      }

      res.json({ data: employee });
    } catch (error: any) {
      console.error('Error fetching employee:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to fetch employee' });
    }
  }

Updating an Employee (updateEmployee)

Now that we can retrieve individual employees, the next step is allowing their details to be updated.

This endpoint allows partial updates to an existing employee record. It first checks whether the employee exists before attempting an update.

If the employee isn’t found, a 404 Not Found response is returned. If the employee exists, the controller forwards the update payload to EmployeeModel.update and returns the updated employee record. This approach ensures data integrity and prevents silent failures.

  static async updateEmployee(req: Request, res: Response): Promise<void> {
    try {
      const { id } = req.params;
      const data: Partial<CreateEmployeeInput> = req.body;

      const employee = await EmployeeModel.findById(parseInt(id));
      if (!employee) {
        res.status(404).json({ error: 'Employee not found' });
        return;
      }

      const updated = await EmployeeModel.update(parseInt(id), data);
      res.json({
        message: 'Employee updated successfully',
        data: updated,
      });
    } catch (error: any) {
      console.error('Error updating employee:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to update employee' });
    }
  }

Deleting an Employee (deleteEmployee)

Finally, the last endpoint in the EmployeeController handles employee deletion.

Before deleting, it verifies that the employee exists to avoid invalid delete operations. If found, the employee record is removed using EmployeeModel.delete, and a success message is returned. If the employee doesn’t exist, the controller responds with a 404 Not Found.

 static async deleteEmployee(req: Request, res: Response): Promise<void> {
    try {
      const { id } = req.params;

      const employee = await EmployeeModel.findById(parseInt(id));
      if (!employee) {
        res.status(404).json({ error: 'Employee not found' });
        return;
      }

      await EmployeeModel.delete(parseInt(id));
      res.json({ message: 'Employee deleted successfully' });
    } catch (error: any) {
      console.error('Error deleting employee:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to delete employee' });
    }
  }

Error Handling Strategy

All controller methods use structured error handling to log errors internally while returning clean and user-friendly error messages to API consumers. This separation ensures sensitive implementation details are not leaked while still providing useful feedback for debugging and client-side handling.

Role in the Overall Payroll System

The EmployeeController provides the foundational APIs required for managing employee records, which are essential inputs for payroll processing. By cleanly separating HTTP concerns from business logic and persistence layers, this controller supports maintainability, scalability, and clear system boundaries within the payroll architecture.

Payroll Controller

This module defines the PayrollController, which serves as the primary HTTP-facing orchestration layer for payroll operations in the system. It exposes RESTful APIs that allow clients to create payrolls, retrieve payroll data, trigger payroll processing, reconcile payment results, authorize bulk transfers, and monitor transaction and account statuses.

Controller Responsibilities

The PayrollController is responsible for:

• Accepting and validating client requests related to payrolls

• Managing payroll lifecycle transitions (creation → processing → completion)

• Triggering background job execution for bulk payroll disbursement

• Reconciling payment results with Monnify

• Providing real-time payroll and transaction status visibility

• Acting as a safe boundary between external clients and internal services

To get started, create a new file src/controllers/payroll.controller.ts. This is where we’ll define all payroll-related endpoints.

At the top of src/controllers/payroll.controller.ts, we start with the following imports:

import { Request, Response } from 'express';
import {
  PayrollModel,
  PayrollItemModel,
  PayrollStatus,
} from '../models/payroll';
import { processPayrollItems } from '../jobs/payroll.processor';
import { monnifyClient } from '../config/monnify';

Here’s what each of these is responsible for:

• Request and Response (from Express): These types give us strongly typed access to incoming HTTP requests and outgoing responses.

• PayrollModel: This model handles payroll batch operations such as creating payrolls, fetching them, and updating their overall status.

• PayrollItemModel: This model lets us fetch and update those items, especially during processing and reconciliation.

• PayrollStatus: This is an enum that defines the valid states of a payroll or payroll item (for example: PENDING, PROCESSING, COMPLETED, FAILED). Using an enum helps keep state transitions explicit and consistent across the system.

• processPayrollItems: This function is responsible for handing off payroll processing to background workers. Instead of processing payrolls synchronously in the HTTP request, we queue the work and let workers handle it asynchronously.

• monnifyClient: This is our gateway to the external payment service. We use it to authorize bulk transfers, check transaction statuses, reconcile payments, and fetch account balances.

Together, these imports give the controller everything it needs to process payroll operations.

With our imports in place, we can now define the controller class itself. This class will serve as the single home for all payroll-related endpoints.

export class PayrollController {
  // Payroll endpoints will live here
}

Creating a Payroll (createPayroll)

With the controller in place, we’ll begin by implementing the endpoint create payroll. This endpoint initializes a new payroll batch, allowing us to either process all employees or a subset by their IDs.

static async createPayroll(req: Request, res: Response): Promise<void> {
try {
const { payroll_period, employee_ids } = req.body;

      if (!payroll_period) {
        res.status(400).json({ error: 'payroll_period is required' });
        return;
      }

      const processedEmployeeIds = employee_ids
        ? employee_ids
            .map((id: any) => parseInt(id, 10))
            .filter((id: number) => !isNaN(id))
        : undefined;

      const payroll = await PayrollModel.create({
        payroll_period,
        employee_ids: processedEmployeeIds,
      });

      res.status(201).json({
        message: 'Payroll created successfully',
        data: payroll,
      });
    } catch (error: any) {
      console.error('Error creating payroll:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to create payroll' });
    }

}

Here’s what’s happening in the code:

• The endpoint requires a payroll_period and optionally accepts a list of employee IDs to support partial payroll runs.

• Incoming employee IDs are normalized and validated to ensure they are valid integers before being passed to the payroll model.

• The controller delegates the actual creation logic to PayrollModel.create, which computes totals and creates payroll items.

• On success, the API responds with a 201 Created status and the newly created payroll record.

Fetching All Payrolls (getAllPayrolls)

This endpoint retrieves all payroll batches in the system. It’s typically used for administrative dashboards and payroll history views. The controller simply delegates to PayrollModel.findAll and returns the results in a structured JSON response.

static async getAllPayrolls(req: Request, res: Response): Promise<void> {
try {
const payrolls = await PayrollModel.findAll();
res.json({ data: payrolls });
} catch (error: any) {
console.error('Error fetching payrolls:', error);
res
.status(500)
.json({ error: error.message || 'Failed to fetch payrolls' });
}
}

static async getPayrollById(req: Request, res: Response): Promise<void> {
try {
const { id } = req.params;
const payroll = await PayrollModel.findById(parseInt(id));

      if (!payroll) {
        res.status(404).json({ error: 'Payroll not found' });
        return;
      }

      const items = await PayrollItemModel.findByPayrollId(payroll.id);

      res.json({
        data: {
          ...payroll,
          items,
        },
      });
    } catch (error: any) {
      console.error('Error fetching payroll:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to fetch payroll' });
    }
}

Fetching a Payroll with Items (getPayrollById)

Next, we’ll implement an endpoint to retrieve a single payroll by its ID along with all associated payroll items. This is useful for administrative dashboards and payroll history views.

static async getPayrollById(req: Request, res: Response): Promise<void> {
try {
const { id } = req.params;
const payroll = await PayrollModel.findById(parseInt(id));

      if (!payroll) {
        res.status(404).json({ error: 'Payroll not found' });
        return;
      }

      const items = await PayrollItemModel.findByPayrollId(payroll.id);

      res.json({
        data: {
          ...payroll,
          items,
        },
      });
    } catch (error: any) {
      console.error('Error fetching payroll:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to fetch payroll' });
    }

}

In the code, we read the id parameter from the URL and convert it to an integer.

If the payroll does not exist, a 404 Not Found response is returned. When found, the controller aggregates payroll metadata and its child payroll items into a single response object, making it convenient for detailed payroll inspection and UI rendering.

Processing a Payroll (processPayroll)

Next, we implement the processPayroll endpoint. This endpoint initiates payroll execution. Before queuing the payroll for processing, the controller enforces important state checks to prevent duplicate or invalid execution, ensuring payrolls that are already PROCESSING or COMPLETED cannot be reprocessed.

static async processPayroll(req: Request, res: Response): Promise<void> {
try {
const { id } = req.params;

      const payroll = await PayrollModel.findById(Number(id));

      if (!payroll) {
        res.status(404).json({ error: 'Payroll not found' });
        return;
      }

      if (
        payroll.status === PayrollStatus.COMPLETED ||
        payroll.status === PayrollStatus.PROCESSING
      ) {
        res.status(400).json({
          error: `Payroll already ${payroll.status}`,
        });
        return;
      }

      // Queue the payroll for processing
      await processPayrollItems(payroll.id);

      res.json({
        message: 'Payroll queued for bulk processing',
        data: {
          payroll_id: payroll.id,
          processing_mode: 'bulk',
        },
      });
    } catch (error: any) {
      console.error('Error processing payroll:', error);
      res.status(500).json({
        error: error.message || 'Failed to process payroll',
      });
    }
}

Here’s what’s happening in the code:

• We get the id parameter from the URL and convert it to a number.

• If no payroll is found with the given ID, we return a 404 Not Found response.

• Before queuing, we check the payroll’s current status. Payrolls that are already PROCESSING or COMPLETED cannot be reprocessed.

• Valid payrolls are handed off to processPayrollItems, which runs the bulk execution in background workers (Bull jobs).

• Once queued, we respond with a JSON object confirming the payroll is ready for bulk processing.

Reconciling Payroll Payments (reconcilePayroll)

Next, we’ll implement the endpoint that reconciles payroll payments. This ensures that the statuses of payroll items in our system match the actual payment outcomes from Monnify.

static async reconcilePayroll(req: Request, res: Response): Promise<void> {
try {
const { id } = req.params;

      const payroll = await PayrollModel.findById(Number(id));
      if (!payroll) {
        res.status(404).json({ error: 'Payroll not found' });
        return;
      }

      const items = await PayrollItemModel.findByPayrollId(Number(id));

      const itemsToReconcile = items.filter(
        (item) => item.transaction_reference
      );

      if (itemsToReconcile.length === 0) {
        res.json({
          message: 'No items to reconcile (no transaction references found)',
          reconciled: 0,
        });
        return;
      }

      let updated = 0;
      let errors = 0;

      for (const item of itemsToReconcile) {
        try {
          const txStatus = await monnifyClient.getTransactionStatus(
            item.transaction_reference!
          );

          const responseBody = txStatus.responseBody || txStatus;
          const paymentStatus =
            responseBody.paymentStatus || responseBody.status;

          if (
            paymentStatus === 'PAID' &&
            item.status !== PayrollStatus.COMPLETED
          ) {
            await PayrollItemModel.updateStatus(
              item.id,
              PayrollStatus.COMPLETED,
              item.transaction_reference
            );
            updated++;
          } else if (
            paymentStatus === 'FAILED' &&
            item.status !== PayrollStatus.FAILED
          ) {
            const errorMessage =
              responseBody.paymentDescription ||
              responseBody.failureReason ||
              'Transaction failed';
            await PayrollItemModel.updateStatus(
              item.id,
              PayrollStatus.FAILED,
              item.transaction_reference,
              errorMessage
            );
            updated++;
          }
        } catch (error: any) {
          errors++;
          console.error(`Error reconciling item ${item.id}:`, error.message);
        }
      }

      // Update payroll stats
      await PayrollController.updatePayrollStats(Number(id));

      res.json({
        message: 'Payroll reconciled successfully',
        reconciled: updated,
        errors,
        total: itemsToReconcile.length,
      });
    } catch (error: any) {
      console.error('Error reconciling payroll:', error);
      res.status(500).json({
        error: error.message || 'Failed to reconcile payroll',
      });
    }
}

The endpoint retrieves all payroll items with transaction references and queries Monnify for each transaction’s status. Based on the response, payroll items are updated to either COMPLETED or FAILED, with failure reasons captured where applicable.

Errors during reconciliation are tracked and logged without aborting the entire reconciliation process. After reconciliation, payroll-level statistics are recalculated to ensure consistency between item-level and batch-level states.

Payroll Statistics Update (Internal Helper)

The private updatePayrollStats method recalculates payroll status based on the aggregate states of its payroll items.

private static async updatePayrollStats(payrollId: number): Promise<void> {
const items = await PayrollItemModel.findByPayrollId(payrollId);

    const completed = items.filter(
      (i) => i.status === PayrollStatus.COMPLETED
    ).length;
    const failed = items.filter(
      (i) => i.status === PayrollStatus.FAILED
    ).length;
    const total = items.length;

    let status: PayrollStatus;
    if (completed === total) {
      status = PayrollStatus.COMPLETED;
    } else if (failed === total) {
      status = PayrollStatus.FAILED;
    } else if (completed > 0) {
      status = PayrollStatus.PARTIALLY_COMPLETED;
    } else {
      status = PayrollStatus.PROCESSING;
    }

    await PayrollModel.updateStatus(payrollId, status, completed, failed);

}

The endpoint determines whether a payroll is fully completed, fully failed, partially completed, or still processing, and updates the payroll record accordingly.

This logic guarantees that the payroll’s summary status always reflects the true execution state of its underlying payments.

Fetching Payroll Status Summary (getPayrollStatus)

Next, we’ll implement the getPayrollStatus endpoint. This endpoint provides a comprehensive status snapshot of a payroll. In addition to returning payroll metadata and items, it computes a summary breakdown of completed, failed, pending, and processing items. This endpoint is particularly useful for real-time dashboards, monitoring tools, and operational visibility.

static async getPayrollStatus(req: Request, res: Response): Promise<void> {
try {
const { id } = req.params;
const payroll = await PayrollModel.findById(parseInt(id));

      if (!payroll) {
        res.status(404).json({ error: 'Payroll not found' });
        return;
      }

      const items = await PayrollItemModel.findByPayrollId(payroll.id);

      res.json({
        data: {
          ...payroll,
          items,
          summary: {
            total: items.length,
            completed: items.filter((i) => i.status === PayrollStatus.COMPLETED)
              .length,
            failed: items.filter((i) => i.status === PayrollStatus.FAILED)
              .length,
            pending: items.filter((i) => i.status === PayrollStatus.PENDING)
              .length,
            processing: items.filter(
              (i) => i.status === PayrollStatus.PROCESSING
            ).length,
          },
        },
      });
    } catch (error: any) {
      console.error('Error fetching payroll status:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to fetch payroll status' });
    }
}

Authorizing Bulk Transfers (authorizeBulkTransfer)

Next, we’ll implement the authorizeBulkTransfer endpoint. Some bulk disbursements require OTP authorization from Monnify. This endpoint accepts a batch reference and authorization code, validates their presence, and forwards them to the Monnify client for verification. Successful authorization allows the bulk transfer to proceed, while failures are clearly reported to the client.

static async authorizeBulkTransfer(
req: Request,
res: Response
): Promise<void> {
try {
const { reference, authorizationCode, payrollId } = req.body;

      if (!reference) {
        res.status(400).json({ error: 'Batch reference is required' });
        return;
      }

      if (!authorizationCode) {
        res.status(400).json({ error: 'Authorization code (OTP) is required' });
        return;
      }

      const result = await monnifyClient.authorizeBulkTransfer(
        reference,
        authorizationCode
      );

      res.json({
        message: 'Bulk transfer authorized successfully',
        data: result,
      });
    } catch (error: any) {
      console.error('Error authorizing bulk transfer:', error);
      res.status(500).json({
        error: error.message || 'Failed to authorize bulk transfer',
      });
    }
}

Here is what’s happening in the code:

• Firstly, we get the batch reference, OTP, and optional payroll ID from the request body.

• We return a 400 Bad Request if the reference or OTP is missing.

• Next, we send the reference and OTP to Monnify to approve the bulk transfer.

• If successful, return a JSON confirmation with Monnify’s response.

Checking Transaction Status (checkTransactionStatus)

This endpoint allows clients or administrators to query the status of an individual transaction using its reference. It delegates the lookup to the Monnify client and returns the raw response, making it useful for debugging, audits, or manual verification workflows.

static async checkTransactionStatus(
req: Request,
res: Response
): Promise<void> {
try {
const { reference } = req.params;

      if (!reference) {
        res.status(400).json({ error: 'Transaction reference is required' });
        return;
      }

      const status = await monnifyClient.getTransactionStatus(reference);
      res.json({ data: status });
    } catch (error: any) {
      console.error('Error checking transaction status:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to check transaction status' });
    }
}

Checking Wallet Balance (getAccountBalance)

This endpoint retrieves the current balance of the Monnify wallet associated with the payroll contract code. It’s typically used for pre-disbursement checks, monitoring available funds, or administrative reporting.

  static async getAccountBalance(req: Request, res: Response): Promise<void> {
    try {
      const balance = await monnifyClient.getAccountBalance();
      res.json({ data: balance });
    } catch (error: any) {
      console.error('Error fetching account balance:', error);
      res
        .status(500)
        .json({ error: error.message || 'Failed to fetch account balance' });
    }
  }

Error Handling and Resilience

All controller methods use structured try–catch blocks to ensure unexpected failures are logged and surfaced as controlled HTTP error responses. This approach prevents sensitive internal errors from leaking while maintaining clarity and debuggability for API consumers.

Role in the Overall Payroll Architecture

The PayrollController acts as the central coordinator of the payroll system. It bridges client requests, domain models, background job processing, and external payment services into a cohesive workflow.

By enforcing state transitions, delegating heavy processing to background workers, and providing reconciliation and monitoring capabilities, this controller ensures payroll execution remains reliable, auditable, and scalable in real-world production environments.

Setting Up Webhook Handlers

Webhooks are essential for receiving real-time payment status updates from Monnify. When a payment completes or fails, Monnify sends a notification to your webhook endpoint.

Start by creating a new file src/routes/monnify.webhook.ts. This file will contain everything related to handling Monnify webhook events.

import { Router, Request, Response } from 'express';
import crypto from 'crypto';
import {
PayrollItemModel,
PayrollModel,
PayrollStatus,
} from '../models/payroll';

const router = Router();

function verifySignature(req: Request): boolean {
const signature = req.headers['monnify-signature'] as string;
if (!signature) return false;

const secret = process.env.MONNIFY_WEBHOOK_SECRET!;
const hash = crypto
.createHmac('sha512', secret)
.update(JSON.stringify(req.body))
.digest('hex');

return hash === signature;
}

router.post('/monnify/webhook', async (req: Request, res: Response) => {
try {
console.log('Monnify Webhook:', JSON.stringify(req.body, null, 2));

    const { eventType, eventData } = req.body;

    if (!eventData?.reference) {
      console.warn('Missing reference, ignoring webhook');
      return res.status(200).send('Ignored');
    }

    const paymentReference = eventData.reference;
    const transactionReference = eventData.transactionReference;
    const description = eventData.transactionDescription || '';

    // Parse our reference format: PAYROLL_{payrollId}_{itemId}
    const [prefix, payrollIdStr, itemIdStr] = paymentReference.split('_');

    if (prefix !== 'PAYROLL') {
      console.warn('Invalid payment reference format:', paymentReference);
      return res.status(200).send('Ignored');
    }

    const payrollId = Number(payrollIdStr);
    const itemId = Number(itemIdStr);

    if (isNaN(payrollId) || isNaN(itemId)) {
      console.warn('Invalid payroll/item IDs:', paymentReference);
      return res.status(200).send('Ignored');
    }

    const item = await PayrollItemModel.findById(itemId);

    if (!item) {
      console.warn('Payroll item not found:', itemId);
      return res.status(200).send('Ignored');
    }

    // Idempotency check - don't process already finalized items
    if (
      item.status === PayrollStatus.COMPLETED ||
      item.status === PayrollStatus.FAILED
    ) {
      console.log(`Item ${itemId} already finalized (${item.status})`);
      return res.status(200).send('Already processed');
    }

    // Update status based on event type
    if (
      eventType === 'SUCCESSFUL_DISBURSEMENT' ||
      eventData.status === 'SUCCESS'
    ) {
      await PayrollItemModel.updateStatus(
        itemId,
        PayrollStatus.COMPLETED,
        transactionReference
      );
      console.log(`✅ Payroll item ${itemId} COMPLETED`);
    } else if (
      eventType === 'FAILED_DISBURSEMENT' ||
      eventType === 'REVERSED_DISBURSEMENT' ||
      eventData.status === 'FAILED'
    ) {
      await PayrollItemModel.updateStatus(
        itemId,
        PayrollStatus.FAILED,
        transactionReference,
        description
      );
      console.log(`Payroll item ${itemId} FAILED`);
    } else {
      console.log(`Unhandled Monnify eventType: ${eventType}`);
    }

    // Update overall payroll stats
    await updatePayrollStats(payrollId);

    return res.status(200).send('OK');

} catch (error: any) {
console.error('Monnify webhook error:', error.message);
return res.status(200).send('OK'); // Always return 200 to prevent retries
}
});

export default router;

async function updatePayrollStats(payrollId: number) {
const items = await PayrollItemModel.findByPayrollId(payrollId);

const completed = items.filter(
(i) => i.status === PayrollStatus.COMPLETED
).length;

const failed = items.filter((i) => i.status === PayrollStatus.FAILED).length;

let status = PayrollStatus.PROCESSING;

if (completed === items.length) {
status = PayrollStatus.COMPLETED;
} else if (failed === items.length) {
status = PayrollStatus.FAILED;
} else if (completed > 0) {
status = PayrollStatus.PARTIALLY_COMPLETED;
}

await PayrollModel.updateStatus(payrollId, status, completed, failed);
}

Key webhook implementation details:

1. Signature verification: The verifySignature function validates that webhooks actually come from Monnify.

2. Idempotency: The handler checks if an item is already finalized before processing.

3. Always return 200: Even on errors, return 200 to prevent Monnify from retrying indefinitely.

4. Reference parsing: Our reference format PAYROLL_{payrollId}_{itemId} lets us identify which payment item to update.

Wiring Up Routes

Employee Routes

We’ll start by defining routes for employee management. These routes expose CRUD operations for employees and simply delegate the actual logic to the EmployeeController.

Create the file src/routes/employee.routes.ts:

import { Router } from 'express';
import { EmployeeController } from '../controllers/employee.controller';

const router = Router();

router.post('/', EmployeeController.createEmployee);
router.get('/', EmployeeController.getAllEmployees);
router.get('/:id', EmployeeController.getEmployeeById);
router.put('/:id', EmployeeController.updateEmployee);
router.delete('/:id', EmployeeController.deleteEmployee);

export default router;

What this gives us:

• A clean /api/employees entry point for all employee-related operations

• Clear separation between routing (URLs) and business logic (controllers)

• A predictable REST structure that’s easy to extend later

Payroll Routes

Next, we define routes for payroll operations. Payroll is more complex than employees, so this router exposes endpoints for creation, processing, reconciliation, authorization, and monitoring.

Create the file src/routes/payroll.routes.ts:

import { Router } from 'express';
import { PayrollController } from '../controllers/payroll.controller';

const router = Router();

router.post('/', PayrollController.createPayroll);
router.get('/', PayrollController.getAllPayrolls);
router.get('/:id', PayrollController.getPayrollById);
router.post('/:id/process', PayrollController.processPayroll);
router.post('/batch/authorize', PayrollController.authorizeBulkTransfer);
router.get('/:id/status', PayrollController.getPayrollStatus);
router.get(
  '/transaction/:reference/status',
  PayrollController.checkTransactionStatus
);
router.get('/account/balance', PayrollController.getAccountBalance);
router.post('/:id/reconcile', PayrollController.reconcilePayroll);

export default router;

What’s happening here:

• Each route maps directly to a well-defined payroll operation

• Long-running or sensitive actions (processing, reconciliation, authorization) are clearly separated

• Monitoring and operational endpoints (status, transaction lookup, balance checks) are first-class citizens

Main Application Entry Point

With all routes defined, we now bring everything together in the main application file. This is where we configure middleware, register routes, and start the server.

Create the file src/index.ts:

import express, { Application, Request, Response } from 'express';
import cors from 'cors';
import helmet from 'helmet';
import dotenv from 'dotenv';
import path from 'path';
import { pool } from './config/database';
import employeeRoutes from './routes/employee.routes';
import payrollRoutes from './routes/payroll.routes';
import monnifyWebhookRoutes from './routes/monnify.webhook';

dotenv.config();

const app: Application = express();
const PORT = process.env.PORT || 3008;

// Middleware
app.use(
  helmet({
    contentSecurityPolicy: false,
  })
);
app.use(
  cors({
    origin: '*',
    methods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS'],
    allowedHeaders: ['Content-Type', 'Authorization'],
  })
);
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

// Health check endpoint
app.get('/health', async (req: Request, res: Response) => {
  try {
    await pool.query('SELECT 1');
    res.json({ status: 'healthy', database: 'connected' });
  } catch (error) {
    res.status(500).json({ status: 'unhealthy', database: 'disconnected' });
  }
});

// Routes
app.use('/api/employees', employeeRoutes);
app.use('/api/payrolls', payrollRoutes);
app.use('/api', monnifyWebhookRoutes);

// 404 handler
app.use((req: Request, res: Response) => {
  res.status(404).json({ error: 'Route not found' });
});

// Error handler
app.use((err: any, req: Request, res: Response, next: any) => {
  console.error('Error:', err);
  res.status(err.status || 500).json({
    error: err.message || 'Internal server error',
  });
});

app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}`);
  console.log(`Environment: ${process.env.NODE_ENV || 'development'}`);
});

// Graceful shutdown
process.on('SIGTERM', async () => {
  console.log('SIGTERM signal received: closing HTTP server');
  await pool.end();
  process.exit(0);
});

process.on('SIGINT', async () => {
  console.log('SIGINT signal received: closing HTTP server');
  await pool.end();
  process.exit(0);
});

Testing the System

Now let's test the complete payroll flow.

Start the application:

docker-compose up -d
npm run dev

Create employees:

curl -X POST http://localhost:3008/api/employees \
  -H "Content-Type: application/json" \
  -d '{
    "name": "John Doe",
    "email": "john.doe@company.com",
    "salary": 50000,
    "account_number": "0123456789",
    "bank_code": "058",
    "bank_name": "GTBank"
  }'

Create a few more employees with different salaries to see how it’s handled.

Create a payroll:

curl -X POST http://localhost:3008/api/payrolls \
  -H "Content-Type: application/json" \
  -d '{
    "payroll_period": "2024-12"
  }'

This creates a payroll with all active employees.

Process the payroll:

curl -X POST http://localhost:3008/api/payrolls/1/process

This queues the payroll for background processing. The system will:

1. Create a bulk transfer request to Monnify

2. Update each payroll item with a transaction reference

3. Wait for webhooks to update final status

Authorize the bulk transfer (if OTP is required):

After processing, Monnify sends an OTP to your registered email. Use it to authorize:

curl -X POST http://localhost:3008/api/payrolls/batch/authorize \
  -H "Content-Type: application/json" \
  -d '{
    "reference": "BATCH_1702123456789",
    "authorizationCode": "123456",
    "payrollId": 1
  }'

Check the payroll status:

curl http://localhost:3008/api/payrolls/1/status

This returns detailed status including a summary of completed, failed, and pending items.

Now, reconcile if needed – if webhooks were missed or you need to sync status:

curl -X POST http://localhost:3008/api/payrolls/1/reconcile

Setting Up Webhooks for Production

For Monnify to send webhooks to your local development environment, you'll need to expose your local server. You can use ngrok:

ngrok http 3008

Then configure the webhook URL in your Monnify dashboard:

https://your-ngrok-url.ngrok.io/api/monnify/webhook

For production, use your actual server URL and ensure HTTPS is enabled.

Then when transactions are successful it will be revealed on the monnify dashboard as well as the transactions that failed.

Conclusion

You've built a complete payroll system that:

• Manages employees with their bank account details

• Creates payroll batches with automatic amount calculation

• Processes bulk payments using Monnify's disbursement API

• Uses background jobs to prevent request timeouts

• Handles webhooks for real-time status updates

• Supports reconciliation to ensure data consistency

Key Takeaways

1. Background jobs are essential: Processing payments synchronously would timeout for large payrolls. Bull and Redis provide reliable async processing.

2. Idempotency matters: Both the webhook handler and reconciliation process check current status before updating, preventing duplicate processing.

3. Bulk transfers save time: Monnify's batch API lets you process hundreds of payments with a single OTP authorization.

4. Status tracking is critical: The system tracks status at both the payroll and individual item level, making it easy to identify and handle failures.

5. Reconciliation is your safety net: When webhooks fail or get delayed, the reconciliation endpoint ensures your database stays in sync with actual payment status.

References:

• Monnify Docs

This article aims to show you how to:

• Properly assign a unique ID to API requests in your Express applications,

• Store and access the ID using the AsyncLocalStorage API in Node.js, and

• Use it in request logging.

Experience in creating API endpoints and using middleware in Express will be helpful as you follow along with this guide. You can also apply ideas from this article to frameworks like NestJS and Koa.

Table of Contents

• Getting Started with the Starter Repository

• Set Up Logger Utilities

• What is AsyncLocalStorage and Why is it Important?

• Store the Request ID in AsyncLocalStorage

• Use the Request ID in the Logger Utilities

• Set Header to have X-Request-Id (Optional Challenge)

• Conclusion

Getting Started with the Starter Repository

To make it easier to follow along, I’ve created a starter project and hosted it on GitHub. You can clone it from here. To get it up and running on your local computer, install its dependencies using your preferred JavaScript package manager (npm, yarn, pnpm, bun). Then start the application by running the npm start command in the terminal of the project.

If the application starts successfully, it should log the snippet below on the terminal:

Listening on port 3333

The application has only one API endpoint currently – a GET / . When you make an API request to the endpoint using curl or a browser by visiting http://localhost:3333, you will get an “OK” string as the payload response:

$ curl -i http://localhost:3333

OK%

If the snippet above is what you see, then congratulations! You have set the project up correctly.

Set Up Logger Utilities

The first step is to set up custom loggers for logging messages to the terminal. The loggers will log the events that occur during the process of handling an API request and log the summary of the request.

To achieve this, you’ll need to install two Express middlewares – morgan and winston – using your preferred package manager. If you use npm, you can run the command below in the folder terminal of the project:

$ npm install morgan winston

If the above command is successful, morgan and winston will be added to the dependencies object in package.json. Create a file named logger.js in the root folder of the project. logger.js will contain the code for the custom logger utilities.

The first logger utility you will create is logger, created from the winston package you installed earlier. logger is an object with two methods:

• info for logging non-error messages to the terminal

• error for logging error messages to the terminal

// logger.js

const winston = require("winston");

const { combine, errors, json, timestamp, colorize } = winston.format;

const logHandler = winston.createLogger({
  level: "debug",
  levels: winston.config.npm.levels,
  format: combine(
    timestamp({ format: "YYYY-MM-DD hh:mm:ss.SSS A" }), // set the format of logged timestamps
    errors({ stack: true }),
    json({ space: 2 }),
    colorize({
      all: true,
      colors: {
        info: "gray", // all info logs should be gray in colour
        error: "red", // all error logs should be red in colour     
        },
    })
  ),
  transports: [new winston.transports.Console()],
});

exports.logger = {
  info: function (message) {
    logHandler.child({}).info(message);
  },

  error: function (message) {
    logHandler.child({}).error(message);
  },
};

In the code snippet above, winston.createLogger is used to create logHandler. logger is exported out of the logger.js module and logger.info and logger.error are functions that use logHandler to log messages to the terminal.

The second logger utility will be a middleware that will log information about the request just before the request response is sent to the client. It will log information such as how long it took to run the request and the status code of the request. It will be called logRequestSummary and will use the morgan package and the http method of logHandler.

// logger.js

const winston = require("winston");
const morgan = require("morgan");

const { combine, errors, json, timestamp, colorize } = winston.format;

const logHandler = winston.createLogger({
  // ...
  format: combine(
    // ...
    colorize({
      all: true,
      colors: {
        // ...
        http: "blue", // 👈🏾 (new line) logs from logRequestSummary will be blue in colour
      },
    })
  ),
  // ...
});

exports.logger = {
    // ...
};

exports.logRequestSummary = morgan(
  function (tokens, req, res) {
    return JSON.stringify({
      url: tokens.url(req, res),
      method: tokens.method(req, res),
      status_code: Number(tokens.status(req, res) || "500"),
      content_length: tokens.res(req, res, "content-length") + " bytes",
      response_time: Number(tokens["response-time"](req, res) || "0") + " ms",
    });
  },
  {
    stream: {
      // use logHandler with the http severity
      write: (message) => {
        const httpLog = JSON.parse(message);
        logHandler.http(httpLog);
      },
    },
  }
);

The JSON string returned by the first function when the morgan function is executed is received by the write function of the stream object in the second argument passed to the organ function. It’s then parsed to JSON and passed to logHandler.http to be logged with the winston.npm http severity level.

At this point, two objects are exported from logger.js: logger and logRequestSummary.

In index.js, create a new controller to handle GET requests to the /hello path. Also import and use the exported objects from logger.js. Use logger to log information when events occur in controllers and include logRequestSummary as a middleware for the application.

// index.js
const express = require("express");
const { logRequestSummary, logger } = require("./logger");

const app = express();

app.use(
  express.json(),
  express.urlencoded({ extended: true }),
  logRequestSummary // logger middleware utility
);

app.get("/", function (req, res) {
  logger.info(`${req.method} request to ${req.url}`); // logger utility for events
  return res.json({ method: req.method, url: req.url });
});

app.get("/hello", function (req, res) {
  logger.info(`${req.method} request to ${req.url}`); // logger utility for events
  return res.json({ method: req.method, url: req.url });
});

// ...

Stop the application (with CTRL + C or OPTION + C), and start it again with npm start. Make API requests to both API endpoints, you’ll see output similar to the snippet below in the terminal – an event log first and a log of the summary of the request after.

{
  "level": "info",
  "message": "GET request to /",
  "timestamp": "2025-08-16 10:35:06.831 PM"
}
{
  "level": "http",
  "message": {
    "content_length": "26 bytes",
    "method": "GET",
    "response_time": "9.034 ms",
    "status_code": 200,
    "url": "/"
  },
  "timestamp": "2025-08-16 10:35:06.844 PM"
}

You can view the latest state of the code by switching to the 2-custom-logger-middleware branch using git checkout 2-custom-logger-middleware or by visiting branch 2-custom-logger-middleware of the repository.

Now that you’re able to log and view events that occur for each API request, how do you differentiate between two consecutive requests to the same endpoint? How do you figure out which API request logged a specific message? How do you specify the API request to trace when communicating with your teammates? By attaching a unique ID to each request, you’ll be able to answer all these questions.

What is AsyncLocalStorage and Why is it Important?

Before AsyncLocalStorage, users of Express stored request context information in the res.locals object. With AsyncLocalStorage, Node.js provides a native way to store information that’s necessary for executing asynchronous functions. According to its documentation, it’s a performant and memory-safe implementation that involves significant optimizations that would be difficult for you to implement by yourself.

When you use AsyncLocalStorage, you can store and access information in a similar manner to localStorage in the browser. You pass the store value (usually an object, but it can be a primitive value, too) as the first argument and the asynchronous function that should access the store value as the second argument when you execute the run method.

James Snell, one of the leading contributors of Node.js explains it further in this video Async Context Tracking in Node with Async Local Storage API.

Store the Request ID in AsyncLocalStorage

In the project, create a file with the name context-storage.js. In this file, you’ll create an instance of AsyncLocalStorage (if it hasn’t been created yet) and export it. This instance of AsyncLocalStorage will be used in storing and retrieving the request IDs for the logger and any other context that needs the request ID.

// context-storage.js
const { AsyncLocalStorage } = require("node:async_hooks");

let store;

module.exports.contextStorage = function () {
  if (!store) {
    store = new AsyncLocalStorage();
  }

  return store;
};

You’ll create another file called set-request-id.js which will create and export a middleware. The middleware will intercept API requests, generate a request ID, and store it in the instance of AsyncLocalStorage from context-storage.js if it doesn’t exist in it already.

You can use any ID-generating library you want, but here we’ll use randomUUID from the Node.js crypto package.

// set-request-id.js
const { randomUUID } = require("node:crypto");
const { contextStorage } = require("./context-storage");

/**
 * Preferably your first middleware.
 *
 * It generates a unique ID and stores it in the AsyncLocalStorage
 * instance for the request context.
 */
module.exports.setRequestId = function () {
  return function (_req, _res, next) {
    requestId = randomUUID();
    const store = contextStorage().getStore();

    if (!store) {
      return contextStorage().run({ requestId }, next);
    }

    if (store && !store.requestId) {
      store.requestId = requestId;
      return next();
    }

    return next();
  };
};

In the setRequestId function in the snippet above, the instance of AsyncLocalStorage created in context-storage.js is retrieved from the return value of executing contextStorage as store. If store is falsy, the run method executes the next Express callback, providing requestId in an object for access anywhere within next from contextStorage.

If store has a value but doesn’t have the requestId property, set the requestId property and its value on it and return the executed next function.

Lastly, place setRequestId as the first middleware of the Express application in index.js so that every request can have an ID before carrying out other operations.

// index.js
const express = require("express");
const { logRequestSummary, logger } = require("./logger");
const { setRequestId } = require("./set-request-id");

const app = express();

app.use(
  setRequestId(), // 👈🏾 set as first middleware
  express.json(),
  express.urlencoded({ extended: true }),
  logRequestSummary
);

// ...

You can check the current state of this project if you run the git checkout 3-async-local-storage-req-id command on your terminal or by visiting 3-async-local-storage-req-id of the GitHub repository.

Use the Request ID in the Logger Utilities

Now that the requestId property has been set in the store, you can access it from anywhere within next using contextStorage. You’ll access it within the functions in logger.js and attach it to the logs so that when messages are logged to the terminal for a request, the request ID will appear with the logged message.

// logger.js
const winston = require("winston");
const morgan = require("morgan");
const { contextStorage } = require("./context-storage");

const { combine, errors, json, timestamp, colorize } = winston.format;

const logHandler = winston.createLogger({
  level: "debug",
  levels: winston.config.npm.levels,
  format: combine(

    // 👇🏽 retrieve requestId from contextStorage and attach it to the logged message
    winston.format((info) => {
      info.request_id = contextStorage().getStore()?.requestId;
      return info;
    })(),
    // 👆🏽 retrieve requestId from contextStorage and attach it to the logged message

    timestamp({ format: "YYYY-MM-DD hh:mm:ss.SSS A" }),
    errors({ stack: true }),
    // ...
  ),
  transports: [new winston.transports.Console()],
});

// ...

In the combine function from winston, you’ll include a function argument that accepts the message to be logged – info – as an argument and attach the request_id property to it. Its value is the value of requestId retrieved from contextStorage. With this modification, any message logged for a request will have the request ID for that request attached to it.

With this complete, stop the project from running if it’s running already and run it again with the npm start command. Make API requests to the two endpoints and you’ll see output similar to the snippet below on the terminal:

{
  "level": "info",
  "message": "GET request to /hello",
  "request_id": "c80e92d0-eafe-42c7-b093-e5ffce014819",
  "timestamp": "2025-08-17 07:58:13.571 PM"
}
{
  "level": "http",
  "message": {
    "content_length": "31 bytes",
    "method": "GET",
    "response_time": "9.397 ms",
    "status_code": 200,
    "url": "/hello"
  },
  "request_id": "c80e92d0-eafe-42c7-b093-e5ffce014819",
  "timestamp": "2025-08-17 07:58:13.584 PM"
}

Unlike the previous log output, this one contains the request ID of each request. By using AsyncLocalStorage to efficiently store the value of the request ID and access it for use in the loggers, you can accurately trace logged messages to their API requests.

You can access the current state of the project if you run the git checkout 4-use-context-in-logger command on the terminal or by visiting 4-use-context-in-logger of the GitHub repository.

Set Header to have X-Request-Id (Optional Challenge)

You have been able to store, access, and attach a request’s ID to its logged message. Can you set the request ID as a header on the response? The challenge is to set a header, X-Request-Id, on the response so that every request response has the value of the request ID as the value of the X-Request-Id response header.

This is useful for communicating with the frontend when trying to debug requests.

Conclusion

When API requests can be monitored, you can track performance metrics to discover areas that need improvement and attention, identify issues like failed requests and server errors and why they happened, and study patterns in request volume metrics for planning and scalability.

When you attach a unique identifier to an API request, you can use it to trace the events that occurred within the lifetime of the request and differentiate it from other requests of the same type.

Aside from using AsyncLocalStorage to store request IDs, you can also use it to store other request context information such as the authenticated user details. Using AsyncLocalStorage to store request context information is considered a best practice.

REST works so well because it speaks the web’s native language. It uses familiar HTTP methods like GET, POST, PUT, and DELETE, treats data as resources, and follows clear conventions. All this makes it easy to understand, quick to implement, and widely supported, which is why most modern web APIs follow REST principles.

In this article, I will explore REST concepts and core principles while we build a simple Express application step by step. You will learn:

• What is REST architecture, and what are its advantages in web development?

• The core REST principles (statelessness, resources, HTTP methods, and so on)

• How to implement these principles in a real Express.js app

• Best practices for designing clean and consistent APIs

Here’s what we will cover:

• What is REST

• Why use REST?

• Core Principles of REST Architecture

• Build a Simple Express App

  • What is Express?

  • Set Up the Express App

  • Build RESTful Resources

  • Middlewares

  • Test your Express App

• Bad REST Practices to Avoid

• Conclusion

What is REST?

Representational State Transfer (REST) is a style of designing networked applications that emphasises a stateless client-server communication model centred around resources. Think of it like ordering at a restaurant: each time you ask for something, you have to tell the waiter exactly what you want, and they don't remember your previous orders.

RESTful APIs treat data as resources, each accessible through a unique web address (URI). Then, it leverages the standard actions defined by HTTP methods – POST to create new resources, GET to retrieve them, PUT to modify existing ones, and DELETE to remove them – providing a consistent and well-understood way to interact with data over the internet.

Why Use REST?

REST architectural principles offer a compelling set of advantages that contribute to building robust, scalable, and maintainable web services. Below are some of the key benefits that make REST a preferred choice for modern web development:

• Simplicity and familiarity: REST leverages standard HTTP, which is already well-understood by developers and infrastructure.

• Scalability: The stateless nature of REST allows for easy scaling of both client and server components independently.

• Flexibility and interoperability: RESTful APIs can be consumed by a wide variety of clients, regardless of the technology stack.

• Cacheability: REST's design supports caching mechanisms, leading to improved performance and reduced server load.

• Loose coupling: The client and server are independent, allowing for changes on either side without necessarily affecting the other.

• Visibility and monitoring: The straightforward nature of HTTP requests and responses makes it easier to monitor and debug interactions.

Core Principles of REST Architecture

REST (Representational State Transfer) is built on a few simple principles that make APIs easy to understand and use. Here’s a quick breakdown of the key ones:

Statelesness

Every request from the client to the server must contain all the information needed to process it. The server doesn’t store anything about the client between requests – no session, no memory of previous actions.

Example: If a user sends GET /movies/1, the server returns the movie data without needing to know whether the user is logged in or what they requested before.

This makes APIs easier to scale, since each request can be handled independently.

Resource and URIs

In REST, everything you work with is considered a resource – users, products, and so on. Each resource should be accessible via a unique, meaningful URL.

Example:

• /movies – a collection of movie resources

• /movies/42 – a specific movie with ID 42

Resources are treated like nouns. Actions are determined by the HTTP method used.

Standard HTTP Methods

REST takes full advantage of HTTP methods to describe what action you’re taking on a resource:

• GET – retrieve data

• POST – create a new resource

• PUT – update or replace a resource

• PATCH – partially update a resource

• DELETE – remove a resource

Example: To delete a movie, you’d send a DELETE request to /movies/42. That’s clear, consistent, and intuitive.

Uniform Interface

REST enforces a consistent structure for communication between client and server. This means all REST APIs should behave similarly, no matter who built them. It includes:

• Using URIs to identify resources

• Using standard HTTP methods

• Representing data in formats like JSON or XML

• Self-descriptive messages (for example, proper status codes and headers)

This consistency makes it easier for developers to understand and integrate with RESTful APIs.

Cacheability

Servers should label responses as cacheable (stored to be retrieved later) or not, so clients can reuse responses when appropriate. This reduces unnecessary server load and improves performance.

Example: A GET /movies response can be cached for 5 minutes if the data doesn’t change frequently. That means fewer repeated calls for the same info.

Client-Server Separation

The client (frontend) and server (backend) operate independently. The client just needs to know how to communicate with the API – it doesn’t care how the server handles data, and vice versa.

This separation allows teams to develop and scale frontend and backend systems separately.

The principles above help create APIs that are scalable, predictable, and easy to work with.

How to Build a Simple Express App

What is Express?

Express.js is a lightweight and flexible Node.js web application framework. Built on top of Node.js, it provides a robust set of features for building single-page, multi-page, and hybrid web applications and APIs. Think of it as a helpful toolkit that streamlines the process of setting up and managing web servers and routing requests.

In this exercise, you will be building an Express app that will:

1. Handle a simple in-memory collection of movies as a RESTful resource

2. Support basic CRUD operations using the appropriate HTTP methods (GET, POST, PUT, DELETE)

3. Parse incoming JSON requests using built-in middleware

4. Use a custom middleware function to validate movie input before creating or updating entries

5. Send clear and meaningful responses based on the outcome of each request

By the end, you’ll have a working API that follows REST principles and can be tested using a tool like Thunder Client or Postman.

Set Up the Express App

To get the most out of this exercise, there are a few tools and concepts you should already be familiar with. Since we’re focusing on REST principles and how to apply them using Express, we won’t dive deep into the basics of these prerequisites. Make sure you’re comfortable with the following:

• Node.js

• Npm

• Thunderclient extension

• Basic JavaScript

With that, let’s get started.

Open your command prompt and create a new directory (folder):

mkdir express-app

Navigate into your new directory:

cd express-app

Initialize an npm project:

npm init -y

Install the Express package in your project:

npm install express

Now, open your directory in your code editor with the following command:

code .

Create a new file, server.js, and set up your Express app in that file:

const express = require('express');
const app = express();
app.use(express.json());

app.listen(8000, () => console.log('Server running on port 8000'));

This snippet above sets up a basic Express server. It starts by importing the Express library you installed, enables JSON parsing for incoming requests (so we can work with request bodies), and listens on port 3000. It’s ready to handle RESTful routes like GET, POST, PUT, and DELETE as we build out our API.

To start your server, go back to your command prompt and type in this command:

node server.js

You should see this logged in your command prompt:

Build RESTful Resources

With our Express app set up, let’s build out the /movies resource using RESTful routes. We'll treat each movie as a resource and use HTTP methods to define what we want to do – retrieve, add, update, or delete movies. For simplicity, we'll store the movies in an in-memory array.

Here is the full set of routes. Add it in your server.js file just under your app.use(express.json()); line:

// In-memory database (for demonstration purposes)
// In a real application, you would use a database like MongoDB or PostgreSQL
const movies = [];

// Get all movies
app.get('/movies', (req, res) => {
  res.json(movies);
  console.log(movies);
});

// Get a particular movie by ID
app.get('/movies/:id', (req, res) => {
  const movie = movies.find(m => m.id === parseInt(req.params.id));
  if (!movie) return res.status(404).send('Movie not found');
  res.json(movie);
});

// Add a new movie
app.post('/movies', (req, res) => {
  const movie = {
    id: movies.length + 1,
    title: req.body.title,
    genre: req.body.genre,
    year: req.body.year
  };
  movies.push(movie);
  res.status(201).json(movie);
});

// Update a movie
app.put('/movies/:id', (req, res) => {
  const movie = movies.find(m => m.id === parseInt(req.params.id));
  if (!movie) return res.status(404).send('Movie not found');

  movie.title = req.body.title;
  movie.genre = req.body.genre;
  movie.year = req.body.year;

  res.json(movie);
});

// Delete a movie
app.delete('/movies/:id', (req, res) => {
  const movieIndex = movies.findIndex(m => m.id === parseInt(req.params.id));
  if (movieIndex === -1) return res.status(404).send('Movie not found');

  const deletedMovie = movies.splice(movieIndex, 1);
  res.json(deletedMovie);
});

The code above defines a complete set of RESTful routes for managing a movies resource using Express.

It begins with an in-memory array, movies, which acts as our temporary data store. The GET /movies route returns all books, while GET /movies/:id looks up a book by its ID using Array.find(), returning a 404 status if it's not found. The POST /movies route accepts JSON input, creates a new book with an auto-incremented ID, and adds it to the array, returning the new resource with a 201 Created status.

The PUT /movies/:id route handles full updates. It first finds the book, and if found, updates its title , genre, and year with the new values from the request body. The DELETE /movies/:id route removes a movie by finding its index in the array and using splice(). If the movie doesn't exist, both PUT and DELETE return a 404 error.

These routes demonstrate idempotency – that is, sending the same PUT or DELETE request multiple times will have the same effect as sending it once, a key REST principle. Each route also returns appropriate HTTP status codes and JSON responses, following REST conventions closely.

Each route follows a REST principle:

• Uses nouns for endpoints (/movies)

• Uses standard HTTP methods to express actions

• Ensure idempotency where appropriate (PUT, DELETE)

• Return appropriate status codes and messages

This structure keeps your API predictable and easy to use – exactly what REST is all about.

Middlewares

In RESTful APIs built with frameworks like Express, middleware plays a key role in maintaining clean, modular, and consistent request handling.

Middlewares are functions that sit in the middle of the request-response cycle in an Express app. When a client sends a request, middleware functions have access to the req (request), res (response), and next objects. They can inspect, modify, or act on the request before it reaches the route handler or even terminate the response early.

You have already seen a middleware here: app.use(express.json());. This is a global middleware that is used for parsing JSON. We will be creating a custom middleware for validating input for our POST and PUT requests.

Add the following code in your server.js file just before your routes:

// Middleware for simple validation
const validateMovie = (req, res, next) => {
  if (!req.body.title || !req.body.genre || !req.body.year) {
      return res.status(400).send('Title, genre, and year are required');
  }
  next();
};

This middleware function, validateMovie, performs basic validation on incoming requests before they reach the route handler. It checks if the title, genre, and year fields are present in the request body. If any of these fields are missing, it immediately responds with a 400 Bad Request status and an error message. If all required fields are provided, it calls next() to pass control to the next middleware or route. This keeps validation logic separate and reusable, helping maintain clean and RESTful route handlers.

To use this middleware, pass it as an argument in your POST and PUT routes, for example example:

app.post('/movies', validateMovie, (req, res) => {
  const movie = {
    id: movies.length + 1,
    title: req.body.title,
    genre: req.body.genre,
    year: req.body.year
  };
  movies.push(movie);
  res.status(201).json(movie);
});

Test Your Express App

To test the changes you have made, you need to restart your server. Go to your command prompt and press CTRL + C (for Windows) or Command + . (for MacOS). Input the start command like before to start the server again.

For this exercise, you will be testing the endpoints with the Thunder Client extension on VSCode. Thunder Client is a lightweight REST API extension for VSCode. With Thunder Client, you can your REST API routes and endpoints directly in VSCode.

To download Thunder Client, click on the Extension icon on the taskbar on your left and search “Thunder Client”. Then click Install:

After installation, you'll see the Thunder Client icon appear in your sidebar below the Extensions icon. Click it, then hit New Request to open a new tab. The request tab will open with a clean layout, ready for you to send and test API calls:

To start testing your API, set the request method (GET, POST, PUT, or DELETE) from the dropdown and enter the appropriate route, like http://localhost:3000/movies. For GET requests, just hit Send and you should see a response, in this case, an empty array []. To get a specific movie, you include an ID (for example, /movies/1).

For POST, PUT, and DELETE requests, switch to the Body tab and select the JSON format. Then provide the data you want to send:

• POST /movies: Add a new movie with {"title": "Movie Title", "genre": "Genre Name", "year": 0000}.

• PUT /movies/1: Update an existing movie with the same JSON structure.

• DELETE /movies/1: Remove a movie by its ID — no body is needed.

After sending each request, Thunder Client will display the response body, headers, and status code. Example:

Bad REST Practices to Avoid

When building REST APIs, aside from using the right HTTP methods, you also have to consider avoiding practices that break the core principles of REST. These bad practices can make your API harder to use, less predictable, or even misleading.

Here are some common REST bad practices and how to avoid them:

Using verbs in endpoints

Avoid routes like /getMovies or /createMovie. RESTful APIs rely on HTTP methods to express actions, so use nouns for endpoints and let methods do the talking — for example, use GET /movies to retrieve movies and POST /movie to create one.

Ignoring HTTP status codes

Returning 200 OK for every response, even errors, breaks REST conventions. Use the appropriate status codes: 201 Created for successful POSTs, 404 Not Found when a resource doesn’t exist, and 400 Bad Request for validation issues. This helps clients interpret responses correctly.

Overloading a single endpoint

Avoid writing one endpoint that changes behaviour based on the request body or headers. Each route should clearly map to a resource and method, like GET /movies/1 to fetch, and DELETE /movies/1 to remove, making the API predictable and easy to follow.

Not being idempotent where expected

PUT and DELETE should be idempotent, meaning repeated requests should have the same effect. If calling DELETE /movies/1 twice causes an error or unexpected behaviour, that’s a red flag. Design your handlers to handle these cases gracefully.

Exposing internal logic or database structure

Don’t leak internal details like database table names or query logic in your route naming (/api/movie_table_data). Keep your URIs clean, abstracted, and centred around the actual resource, like /movies.

Avoiding these practices not only keeps your API RESTful but also improves developer experience, consistency, and long-term maintainability.

Conclusion

You have explored the core ideas behind REST and put them into practice by building and testing a real Express API. This hands-on approach should help bridge the gap between theory and real-world application.

Along the way, you learned how REST treats resources, how to write clean and predictable routes, and why these principles matter when designing APIs that are easy to use and maintain.

Here’s a quick recap of what you’ve built and learned:

• Defined what REST is and why it’s widely used in web development

• Explored core REST principles like statelessness, resource-based routing, HTTP methods, status codes, and idempotency

• Set up a simple Express server to serve as the base for a RESTful API

• Built a complete set of routes (GET, POST, PUT, DELETE) for managing a movies resource

• Used middleware for tasks like JSON parsing and input validation

• Tested routes using Thunder Client and learned how to interact with each HTTP method

• Identified common REST anti-patterns and how to avoid them for cleaner design

To take your API further and follow more advanced RESTful practices, consider:

• Organising routes with Express Router to keep your code modular

• Adding robust error handling to return consistent and informative responses

• Using logging tools like morgan to monitor requests and debug more easily

• Securing endpoints with authentication methods like JWT or API keys

• Structuring responses consistently, possibly with pagination and filtering for larger datasets

• Validating user input thoroughly with libraries like Joi or express-validator

Explore the full project on GitHub, review the code, and try extending it with your features. Happy coding!

Table of Contents

• How to Get Your API Key

• Project Setup

• Front End Setup

• Server Setup

• Update the Front End

• Summary

• Conclusion

Prerequisites

To build this project, you should know the basics of React and Express.

What is the Gemini API?

Google's Gemini API is a powerful tool that lets you integrate advanced AI capabilities into your applications. Gemini is a multimodal model, which means you can use various types of input, like text, images, audio, and video.

It’s good at analyzing and processing large amounts of text as well as pulling information from videos – which makes it great for our use case of a subtitle generator.

How to Get Your API Key

An API key acts as a unique identifier and authenticates your requests to the service. It's essential for accessing and using Gemini AI’s capabilities. This key will allow our application to communicate with Gemini and help us build our project.

Go to Google AI Studio, then click “Get API Key”:

After you are redirected to the API KEY page, click “Create API Key“:

A new API KEY will be created. Then make sure you copy the key.

This is your API key. This key is used to authenticate your application's requests to the Gemini API. Each time your application sends a request to Gemini, this key must be included. Gemini uses this key to verify that the request is coming from an authorized source. Without this API key, your requests will be rejected, and you won't be able to access Gemini's services.

Project Setup

Start by creating a new folder for your project. Let's call it ai-subtitle-generator.

Inside the ai-subtitle-generator folder, create two subfolders: client and server. The client folder will contain the React frontend, and the server folder will contain the Express backend.

Front End Setup

First, we will focus on the front end and set up a basic React application.

Navigate to the client folder:

cd client

Then create a new React project using Vite. To do that, run the following command:

npm create vite@latest .

When prompted, choose “React“. Select “React + TS” or “React + JS”. In this tutorial, I will use React + TS. You can also follow along with JS.

Next, install the dependencies with this command:

npm install

Then start the development server:

npm run dev

How to Handle File Uploads in the Frontend

Now in client/src/App.tsx, add the following code:

//  client/src/App.tsx

const App = () => {
    const handleSubmit = async (e: React.FormEvent<HTMLFormElement>): Promise<void> => {
    e.preventDefault();
    try {
      const formData = new FormData(e.currentTarget);
      console.log(formData)
    } catch (error) {
      console.log(error);
    }
  };

  return (
    <div>
      <form onSubmit={handleSubmit}>
        <input type="file" accept="video/*,.mkv" name="video" />
        <input type="submit" />
      </form>
    </div>
  );
};

export default App;

In the above code, we have used an input tag that will accept the video and name it as video. This name will be appended to the FormData object.

While sending the video to the server, we need to send it as a key-value pair, where the key is a video and the value is the file data.

Why key-value pairs? Because when the server receives the request, it needs to parse the incoming chunks. After parsing, the video data will be available in req.files[key], where the key is the name we have assigned in the frontend (video in this case).

This is why we are using the FormData object. When we create a new FormData instance and pass e.target to it, all the form fields and their names will automatically be available as key-value pairs.

Server Setup

Now that we have our API key, let's set up the backend server. This server will handle video uploads from the frontend and communicate with the Gemini API for subtitle generation.

Navigate to server folder:

cd server

And initialize the project:

npm init -y

Then install the necessary packages:

npm install express dotenv cors @google/generative-ai express-fileupload nodemon

These are the back-end dependencies we’re using in this project:

• express: The web framework for creating the backend API.

• dotenv: Loads environment variables from a .env file.

• cors: Enables Cross-Origin Resource Sharing, allowing your frontend to communicate with your backend.

• @google/generative-ai: The Google AI library for interacting with the Gemini API.

• express-fileupload: Handles file uploads, making it easy to access uploaded files on the server.

• nodemon: Automatically restarts the server when you make changes to your code.

Set Up the Environment Variables

Now, create a file called .env. This is where you’ll manage your API keys.

//.env
API_KEY = YOUR_API_API
PORT = 3000

Update the package.json

For this project, we are using ES6 modules instead of CommonJS. To enable this, update your package.json file with the following code:

{
  "name": "server",
  "version": "1.0.0",
  "main": "index.js",
  "type": "module",       //Add "type": "module" to enable ES6 modules
  "scripts": {
    "start": "node server.js",
    "dev": "nodemon server.js"    //configure nodemon
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "@google/generative-ai": "^0.21.0",
    "cors": "^2.8.5",
    "dotenv": "^16.4.7",
    "express": "^4.21.1",
    "express-fileupload": "^1.5.1",
    "nodemon": "^3.1.7"
  }
}

Basic Setup of Express

Create a file server.js. Now, let’s set up a basic Express application.

//  server/server.js

import express from "express";
import { configDotenv } from "dotenv";
import fileUpload from "express-fileupload";
import cors from "cors"

const app = express();

configDotenv();           //configure the env
app.use(fileUpload());    //it will parse the mutipart data
app.use(express.json());  // Enable JSON parsing for request bodies
app.use(cors())           //configure cors

app.use("/api/subs",subRoutes);  // Use routes for the "/api/subs" endpoint

app.listen(process.env.PORT, () => {   //access the PORT from the .env
  console.log("server started");         
});

In this code, we create an Express app instance and then load our environment variables. This is where we keep sensitive data like API keys secure. Next, we apply middleware functions: fileUpload prepares the server to receive uploaded videos, express.json allows us to receive JSON data, and cors enables communication between our frontend and backend.

We define a route (/api/subs) that will handle all requests related to subtitle generation. The specific logic for these routes will be defined in subs.routes.js. Finally, we start the server, telling it to listen for requests on the port specified in our .env file.

Now we need to create some folders to manage the code. You can also manage the entire code in a single file, but structuring it into separate folders and managing them all that way will be easier.

This is the final folder structure for the server:

server/
├── server.js
├── controller/
│   └── subs.controller.js
├── gemini/
│   ├── gemini.config.js
├── routes/
│   └── subs.routes.js
├── uploads/
├── utils/
│   ├── fileUpload.js
│   └── genContent.js
└── .env

Note: Don’t worry about creating this folder structure now. This is just for reference. Follow along with me step by step, and we will build this structure together.

Create the Routes

Now create a routes folder and then create subs.routes.js:

// server/routes/sub.routes.js

import express from "express"
import { uploadFile } from "../controller/subs.controller.js"    // import the uploadFile function from the controller folder

const router = express.Router()

router.post("/",uploadFile)    // define a POST route that calls the uploadFile function

export default router     // export the router to use in the main server.js file

This code defines the routes for our server, specifically the route that handles video uploads and subtitle generation.

We create a new router instance using express.Router(). This allows us to define routes separate from our main server file, improving code organization. We define a POST route at the root path ("/") of our API endpoint. When a POST request is made to this route (which will happen when a user submits the video upload form on the frontend), the uploadFile function is called. This function will handle the actual upload and subtitle generation.

Finally, we export the router so that it can be used in our main server file (server.js) to connect this route to the main application.

Configure Gemini

Now, let's configure how our application will interact with Gemini.

Create a gemini folder and then create a new file called gemini.config.js:

//  server/gemini/gemini.config.js

import {
  GoogleGenerativeAI,
  HarmBlockThreshold,
  HarmCategory,
} from "@google/generative-ai";
import { configDotenv } from "dotenv";
configDotenv();

const genAI = new GoogleGenerativeAI(process.env.API_KEY);  // Initialize Google Generative AI with the API key

const safetySettings = [
  {
    category: HarmCategory.HARM_CATEGORY_HARASSMENT,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
  {
    category: HarmCategory.HARM_CATEGORY_HATE_SPEECH,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
  {
    category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
  {
    category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
];

const model = genAI.getGenerativeModel({
  model: "gemini-1.5-flash-001",    //choose the model
  safetySettings: safetySettings,   //optional safety settings
});

export default model;    //export the model

In the code above, the safetySettings are optional. These settings allow you to define thresholds for potentially harmful content (like hate speech, violence, or explicit material) in Gemini's output.

You can read more about Gemini’s safety settings here.

Create a Controller to Handle Endpoint Logic

Now, create a controller folder, and inside it create a file named subs.controller.js. In this file, you'll handle the endpoint logic for interacting with the Gemini model.

In server/controller/subs.controller.js, add this code:

// server/controller/subs.controller.js

import { fileURLToPath } from "url";
import path from "path";
import fs from "fs";

const __filename = fileURLToPath(import.meta.url);  //converts the module URL to a file path
const __dirname = path.dirname(__filename);   //get the current file directory

export const uploadFile = async (req, res) => {
  try {
    if (!req.files || !req.files.video) {   //if there is no file available, return error to the client
      return res.status(400).json({ error: "No video uploaded" });
    }

    const videoFile = req.files.video;   //access the video
    const uploadDir = path.join(__dirname, "..", "uploads");   //path to upload the video temporarily

    if (!fs.existsSync(uploadDir)) {   //check if the directory exists
      fs.mkdirSync(uploadDir);      //if not create a new one
    }

    const uploadPath = path.join(uploadDir, videoFile.name);  

    await videoFile.mv(uploadPath);  //it moves the video from the buffer to the "upload" folder

    return res.status(200).json({ message:"file uploaded sucessfully" });
  } catch (error) {
    return res
      .status(500)
      .json({ error: "Internal server error: " + error.message });
  }
};

Since we are using an ES6 module, the __dirname is not available by default. The file handling mechanism is different compared to CommonJS. Because of this, we’ll use fileURLToPath to handle file paths.

We moved the file from the default temporary location which is the buffer to the uploads folder.

But the file upload process is not yet complete. We still need to send the file to Google AI File Manager, and after uploading, it will return a URI. This URI will then be passed to the model for video analysis.

How to Upload a File to the Google AI File Manager

Create a folder utils and create a file fileUpload.js. You can refer to the folder structure provided above.

//  server/utils/fileUpload.js

import { GoogleAIFileManager, FileState } from "@google/generative-ai/server";
import { configDotenv } from "dotenv";
configDotenv();

export const fileManager = new GoogleAIFileManager(process.env.API_KEY);  //create a new GoogleAIFileManager instance

export async function fileUpload(path, videoData) {  
  try {
    const uploadResponse = await fileManager.uploadFile(path, {   //give the path as an argument
      mimeType: videoData.mimetype,  
      displayName: videoData.name,
    });
    const name = uploadResponse.file.name;
    let file = await fileManager.getFile(name);    
    while (file.state === FileState.PROCESSING) {     //check the state of the file
      process.stdout.write(".");
      await new Promise((res) => setTimeout(res, 10000));   //check every 10 second
      file = await fileManager.getFile(name);
    }
    if (file.state === FileState.FAILED) {   
      throw new Error("Video processing failed");
    }
    return file;   // return the file object, containing the upload file information and the uri
  } catch (error) {
    throw error;
  }
}

In the code above, we created a function called fileUpload that takes two arguments. These arguments will be passed from the controller function, which we'll set up later.

The fileUpload function uses the fileManager.uploadFile method to send the video to Google's servers. This method needs two arguments: the file path and an object containing metadata about the file (its MIME type and display name).

Because video processing on Google's servers takes time, we need to check the file's status. We do this using a loop that checks the file's state every 10 seconds using fileManager.getFile(). The loop continues as long as the file's state is PROCESSING. Once the state changes to either SUCCESS or FAILED, the loop stops.

The function then checks if the processing was successful. If so, it returns the file object, which contains information about the uploaded and processed video, including its URI. Otherwise, if the state is FAILED, the function throws an error.

Pass the URI to the Gemini Model

Now in the utils folder, create a file called genContent.js:

// server/utils/genContent.js

import model from "../gemini/gemini.config.js";
import { configDotenv } from "dotenv";
configDotenv();

export async function getContent(file) {
  try {
    const result = await model.generateContent([
      {
        fileData: {
          mimeType: file.mimeType,
          fileUri: file.uri,
        },
      },
      {
        text: "You need to write a subtitle for this full video, write the subtitle in the SRT format, don't write anything else other than a subtitle in the response, create accurate subtitle.",
      },
    ]);
    return result.response.text();
  } catch (error) {
    throw error;
  }
}

Import the model that we configured earlier. Create a function called getContent. The getContent function takes the file object (returned from the fileUpload function).

Pass the file URI and the mimi to the model. Then we’ll provide a prompt instructing the model to generate subtitles for the entire video in SRT format. You can also add your prompt if you want. Then return the response.

Update the subs.controller.js File

Finally, we need to update the controller file. We've created the fileUpload and getContent functions, and now we'll use them in the controller and provide the required arguments.

In the server/controller/subs.controller.js:

//  server/controller/subs.controller.js

import { fileURLToPath } from "url";
import path from "path";
import fs from "fs";
import { fileUpload } from "../utils/fileUpload.js";
import { getContent } from "../utils/genContent.js";

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

export const uploadFile = async (req, res) => {
  try {
    if (!req.files || !req.files.video) {
      return res.status(400).json({ error: "No video uploaded" });
    }

    const videoFile = req.files.video;
    const uploadDir = path.join(__dirname, "..", "uploads");

    if (!fs.existsSync(uploadDir)) {
      fs.mkdirSync(uploadDir);
    }

    const uploadPath = path.join(uploadDir, videoFile.name);

    await videoFile.mv(uploadPath);

    const response = await fileUpload(uploadPath, req.files.video);  //we pass 'uploadPath' and the video file data to 'fileUpload'
    const genContent = await getContent(response);   //the 'response' (containing the file URI) is passed to 'getContent'

    return res.status(200).json({ subs: genContent });   //// return the generated subtitles to the client
  } catch (error) {
    console.error("Error uploading video:", error);
    return res
      .status(500)
      .json({ error: "Internal server error: " + error.message });
  }
};

With this, the backend API is complete. Now, we'll move on to updating the front end.

Update the Front End

Our frontend currently only allows users to select a video. In this section, we'll update it to send the video data to our backend for processing. The frontend will then receive the generated subtitles from the backend and initiate a download of the .srt file.

Navigate to the client folder:

cd client

Install axios. We’ll use it to handle HTTP requests.

npm install axios

In the client/src/App.tsx:

//   client/src/App.tsx

import axios from "axios";

const App = () => {
  const handleSubmit = async (e: React.FormEvent<HTMLFormElement>): Promise<void> => {
    e.preventDefault();
    try {
      const formData = new FormData(e.currentTarget);
      // sending a POST request with form data
      const response = await axios.post(
        "http://localhost:3000/api/subs/",   
        formData
      );
// creating a Blob from the server response and triggering the file download
      const blob = new Blob([response.data.subs], { type: "text/plain" }); 
      const link = document.createElement("a");
      link.href = URL.createObjectURL(blob);
      link.download = "subtitle.srt";
      link.click();
      link.remove();
    } catch (error) {
      console.log(error);
    }
  };

  return (
    <div>
      <form onSubmit={handleSubmit}>
        <input type="file" accept="video/*,.mkv" name="video" />
        <input type="submit" />
      </form>
    </div>
  );
};

export default App;

axios makes the POST request to your backend API endpoint (/api/subs). The server will process the video, and this might take some time.

After the server sends the generated subtitles, the frontend receives them as a response. To handle this response and allow users to download the subtitles, we'll use a Blob. A Blob (Binary Large Object) is a web API object that represents raw binary data, essentially acting like a file. In our case, the subtitles returned from the server will be converted into a Blob, which will then allow us to trigger a download in the user's browser.

Summary

In this tutorial, you learned how to build an AI-powered subtitle generator using Google's Gemini API, React, and Express. You can upload videos, send them to the Gemini API for subtitle generation, and provide the generated subtitles for download.

Conclusion

That's it! You've successfully built an AI-powered subtitle generator using the Gemini API. For quicker testing, start with shorter video clips (3-5 minutes). Longer videos might take more time to process.

Want to create a customizable video prompting application? Just add an input field to let users enter their prompts, send that prompt to the server, and use it in place of the hardcoded prompt. That's all it takes.

For more information about the Gemini API, refer to the official Gemini API Docs

You can find the full code here: AI-Subtitle-Generator

If there are any mistakes or you have any questions, contact me on LinkedIn or Instagram.

Thank you for reading!

If you haven’t come across a kata yet, trust me—you will soon enough.

There’s a reason why developers love katas, whether they use them to sharpen their skills for personal projects or prepare for interviews.

A kata is all about deliberate practice. It comes from martial arts like Karate and Judo, and, according to Wikipedia, it’s defined as a pre-determined sequence of movements, techniques, and patterns that follow a specific order (source: wikipedia).

Kata Machines come from this idea: learning through drills and deliberate, conscious (choreographed) practice.

I realized just how perfect katas are when I was learning Haskell back in the day. If you know, you know. Haskell was a beast to learn for me back then!

So, I thought, why not do the same for the backend? Just pick one high-level concept, and drill down on it repeatedly and deliberately to its core and first principles.

In this article, I picked server-side frameworks. We're going to pick apart the idea of a "framework" using Express as an example.

We’re going to take high-level Express:

const express = require("express");

const PORT = 3000;
const app = express();

app.get("/", (req, res) => {
  res.send("Hello, world!");
});

app.listen(PORT, () => console.log("Server listening on port:", PORT));

And drill all the way down, repeatedly, until we touch Node.js native code:

void TCPWrap::New(const FunctionCallbackInfo<Value>& args) {
  CHECK(args.IsConstructCall());
  CHECK(args[0]->IsInt32());
  Environment* env = Environment::GetCurrent(args);
  int type_value = args[0].As<Int32>()->Value();
  TCPWrap::SocketType type = static_cast<TCPWrap::SocketType>(type_value);
  ProviderType provider;
  switch (type) {
    case SOCKET:
      provider = PROVIDER_TCPWRAP;
      break;
    case SERVER:
      provider = PROVIDER_TCPSERVERWRAP;
      break;
    default:
      UNREACHABLE();
  }
  new TCPWrap(env, args.This(), provider);
}

And having gained this new intuition, we’ll build back up with a custom "Express" implementation:

function serverMux() {
  function hook(req, res) {
    // To be implemented
  }
  return {
    hook
  };
}

const app = serverMux();

const server = http.createServer((req, res) => {
  app.hook(req, res);
});

It’s going to be quite the journey – and a rewarding one at that!

I'm assuming you have some backend knowledge and classify yourself as an advanced beginner who’s looking to level up.

If that sounds like you, we’re ready to proceed.

Here’s what we’ll cover:

• Form 1: Server-Side Frameworks

  • First Drill: Unpacking Express.js

  • The Server

  • The Socket in Node.js

  • The Socket in Node.js Source Code

• Form 2: Implementing a Custom Server Mux

  • Creating Our Custom Router

  • Basic Mux Skeleton

  • The Hook Function

  • Why Use a Queue?

    • Queue Operations

  • Request Wrapper

  • Testing the Queue

  • Processing Requests

  • Lookup Table and Handlers

  • Registering Handlers

• Wrapping Up

Form 1: Server-Side Frameworks

The term "server-side framework" is broad. Think about it: mysql2 could be considered a framework depending on how you classify frameworks and libraries. Even sharp.js for image editing could fit under the umbrella of server-side frameworks, right?

But the question is, what type of framework is Express.js?

Express is a multiplexer—specifically, a server multiplexer (server mux). I promise, the term isn’t as complex as it sounds. The implementation, though—that’s a whole different story.

In simple terms, a server mux is a router. Of course, Express and other server muxes handle more than just routing, but that’s the core idea.

Express takes in request and response objects from the server and routes them. Don’t worry, we’ll dive into routing soon.

Here’s an interesting point: if Express isn’t the server, then what exactly is the server?

To answer that, we need to look at the Express.js source code, which you can clone from GitHub:

git clone https://github.com/expressjs/express.git

Once you’re set, we can dive right in with our first deep dive.

First Drill: Unpacking Express.js

Open your Express source code in an editor. You’ll find the entry file express.js in the lib folder.

You can skim the file, but we’re going to focus on lines 42 and 43—the heart of it all:

mixin(app, EventEmitter.prototype, false);
mixin(app, proto, false);

What you’re looking at is object composition: a design pattern where an object is created by combining the properties and methods of other objects.

Our target object here is proto, which is imported from application.js, the core of Express.

Let’s open that file. There’s a lot of code, but remember, our goal is to figure out where the server is within Express.

If there’s one function in Express that everyone likely knows, it’s listen. The essence of a server is to "listen" over a network. So, do a quick Ctrl+F for "listen," and you’ll find the definition on line 633:

app.listen = function listen() {
  var server = http.createServer(this);
  return server.listen.apply(server, arguments);
};

There it is, the famous listen function. Did we just find the server?

var server = http.createServer(this);

We’ve already seen a version of this in the intro:

const server = http.createServer((req, res) => {
  app.hook(req, res);
});

This confirms that Express is indeed a server mux, and the actual server is returned by the Node.js createServer function from the http package.

That’s some solid progress!

We’ve peeled back a layer, but we can go deeper. What exactly does createServer do, and what is this server object?

The Server

A server is the basic unit of the backend. At its core, the concept is simple: how can two or more processes communicate over a network?

This is the fundamental idea behind network programming. We have devices equipped with IP addresses for identification and ports for data exchange over a network.

The communication itself is complex, which is where protocols come in to facilitate the process.

The most common protocols are UDP and TCP:

• UDP is a connectionless protocol and does not guarantee reliable communication, but allows for low-latency and efficient data transfer. This is ideal for time-sensitive applications such as video conferencing, online gaming, and voice over IP (VoIP) (source Wikipedia).

• TCP is a connection-oriented protocol with reliable, ordered, and error-checked data transmission between applications on networked devices. It’s a major part of internet applications (source Wikipedia).

TCP is the most widely used protocol due to its reliability, and most server-side applications you’ll work with, including Express, are TCP-based.

Although I love the quirks and power of UDP, we’ll focus on TCP, tracing its roots in Node.js.

We’ve already seen a glimpse of this:

void TCPWrap::New(const FunctionCallbackInfo<Value>& args) {
  // some code
  new TCPWrap(env, args.This(), provider);
}

Before we dig into that, we need to answer a key question: What does it really mean to be a server process?

Without getting too deep into file descriptors, sockets, or network layers, a server is an OS-level object responsible for handling communication between nodes. When you call:

const http = require("node:http");

const something = http.createServer({});

You’re creating an OS-level object, commonly known as a socket. This socket facilitates network communication between devices, along with handling data encoding and decoding.

In short, createServer abstracts and returns this socket object.

And, yes, we can implement this socket in Node.js. Remember, Node.js has native access to the OS, allowing JavaScript to function at the system level.

The Socket in Node.js

Here’s some code that creates a server socket:

// Using Node v20
const net = require('node:net');

const server = net.createServer((c) => {
  console.log('client connected', c.remoteAddress);
  c.write("Hello; world");

  c.on('end', () => {
    console.log('client disconnected');
  });
});

server.on('error', (err) => {
  throw err;
});

server.listen(3000, () => {
  console.log('server bound');
});

While net.createServer((c) is still a high-level abstraction like http.createServer, it returns the raw socket.

The c object represents the client that made the connection (dial). Beyond writing to it, we can do much more.

For instance, here’s a simple write operation:

c.write("Hello world");

Our socket is running on localhost:3000. If you make a request (or use curl):

curl localhost:3000

The OS-level network stack encodes not only your data but also information about who you are and where to find you—in the form of a response, among other things.

This is what the server receives, and it’s important to know where to send the response (like IP, and so on).

So, the c object represents all of that!

We’ve covered a lot of the surface-level concepts, but before we wrap up this part, here’s a bonus challenge:

Try writing a class on the server to manage multiple connections. You could store these connections in a data structure and periodically send data to them while the connection remains open.

We’re about three layers deep now, but the journey isn’t over. Remember the goal?

Now it’s time to clone the Node.js source code. Don’t worry, we’ll only focus on the relevant parts.

git clone https://github.com/nodejs/node.git

The Socket in Node.js Source Code

Let the tracing begin! Node.js is a massive codebase – it’s an entire engine that does way more than just handle sockets. But we only care about the networking part today.

First, navigate to the lib folder, and inside you’ll find a file called net.js. This is where most of the work happens for network applications. If you scroll down to line 210, you’ll see a familiar sight:

function createServer(options, connectionListener) {
  return new Server(options, connectionListener);
}

That’s it! Every time we create a server, it calls this function and returns a Server object. Anytime you see new in JavaScript, you should have a lightbulb moment—it means a new object or class (blueprint) is being created.

So we can trace and find the Server definition:

On line 1737

At first glance, it might seem like nothing special is happening. But JavaScript has a sneaky way of hiding complexity.

Here’s the thing: JavaScript is a prototype-based language. This means that objects can inherit features from other objects through prototypes. On line 1791, we see this in action:

ObjectSetPrototypeOf(Server.prototype, EventEmitter.prototype);

In plain English: our Server object is inheriting all the behavior from other objects like EventEmitter, for example. This is a common pattern in JavaScript libraries – remember the mixin in Express?

At this point, if you’ve never worked with prototypes or Object-Oriented JavaScript (OOJS), this might feel like advanced territory. But don’t worry – the good folks at MDN have an excellent guide on prototypes to get you up to speed.

Now, what’s one thing we know for sure about a Node.js server? It has a listen function. We use it all the time in server-side code (even in frameworks like Express). So, let’s check if our Server object has a listen function.

Scroll down a bit more, and there it is on line 2006:

Server.prototype.listen = function(...args) {}

This function handles a lot of stuff—like validating the port number—but the key part starts around line 2016, where the comment clearly tells us:

// start TCP server listening on host:port

We know what TCP is!

The important functions here are lookupAndListen and listenInCluster. They are responsible for starting the actual TCP server:

// start TCP server listening on host:port
if (options.host) {
  lookupAndListen(this, options.port | 0, options.host, backlog, options.exclusive, flags);
} else {
  listenInCluster(this, null, options.port | 0, 4, backlog, undefined, options.exclusive);
}

Digging into lookupAndListen (line 2156), we find that it calls listenInCluster, which leads us to another function: server._listen2 (yep, more tracing!):

server._listen2(address, port, addressType, backlog, fd, flags);

As the comments explain, this is all about backward compatibility:

// _listen2 sets up the listened handle, it is still named like this
// to avoid breaking code that wraps this method

I know this might feel like a wild goose chase, but trust me, tracing through a large codebase like Node.js requires patience. We’re getting close.

So, ._listen2 is defined in our Server object’s prototype and points to a function called setupListenHandle (line 1856). This function is the real hub where everything comes together.

Around line 1870 and 1883, you’ll find the function createServerHandle:

function createServerHandle(address, port, addressType, fd, flags) {
   handle = new TCP(TCPConstants.SERVER);
   isTCP = true;
   return handle;
}

Finally! We’ve hit the core: the TCP object. This is where the actual TCP server is created, the core. We could stop here, satisfied that we’ve found the TCP server, but why not dig deeper?

Remember that new TCP is creating an object, so we need to figure out what TCP actually represents.

Go back up to line 68, where you’ll see the following import:

const {
  TCP,
  TCPConnectWrap,
  constants: TCPConstants,
} = internalBinding('tcp_wrap');

This is where things get interesting. You might wonder: “What kind of import is that? It’s not your regular require or import statement.” That’s because JavaScript alone can’t handle TCP servers—it needs help from C++.

Node.js, which is built on the V8 engine, relies on C++ bindings to do the heavy lifting. These bindings are like a bridge, allowing JavaScript to communicate with low-level system functions (like creating a TCP server). internalBinding('tcp_wrap') is one of these bridges.

To truly trace things to their source, we need to dive into the Node.js C++ code. You’ll find tcp_wrap.cc in the src folder (among others like crypto, streams, async, fs). Open it, and you’ll find this function:

void TCPWrap::New(const FunctionCallbackInfo<Value>& args) {
  CHECK(args.IsConstructCall());
  CHECK(args[0]->IsInt32());
  Environment* env = Environment::GetCurrent(args);
  int type_value = args[0].As<Int32>()->Value();
  TCPWrap::SocketType type = static_cast<TCPWrap::SocketType>(type_value);
  ProviderType provider;
  switch (type) {
    case SOCKET:
      provider = PROVIDER_TCPWRAP;
      break;
    case SERVER:
      provider = PROVIDER_TCPSERVERWRAP;
      break;
    default:
      UNREACHABLE();
  }
  new TCPWrap(env, args.This(), provider);
}

This is where the TCP server is actually created. You can see more familiar functions like bind, and everything JavaScript does is just a mirror of these lower-level operations.

We’ve traced our way from high-level JavaScript all the way down to C++—the true beginning of a TCP server in Node.js.

We've completed the first part of the introduction: "And drill all the way down repeatedly until we touch Node.js native code" and now it's time to build up.

Form 2: Implementing a Custom Server Mux

Before diving into the code, the goal isn’t to focus on the complexity of mux (multiplexer) development (because that can get complicated). Instead, it’s to show how the server and mux fit together.

If anything, this is the key takeaway: the flow from the server to the router, and ultimately to the caller (the client that made the request).

Remember, we've already seen a similar concept in Express:

// Express app inherits from Node's EventEmitter
mixin(app, EventEmitter.prototype, false);
// Implements the server mux with functions like listen, handle, middleware
mixin(app, proto, false);

Behind the scenes, a lot of complex code is abstracted away. This helps simplify things and makes the code cleaner, but for teaching purposes, we’ll take a more verbose approach. This way, you can see how everything connects.

Creating Our Custom Router

Let's start simple and build a basic server. You probably already know how to create a native server in Node.js:

const server = http.createServer((req, res) => {
   app.hook(req, res);
});

Here, we’re introducing an object app with a hook function (which we’ll implement shortly). This is where the server redirects the req and res to our custom router. This hook is the meeting point—the interaction between the server and the router (mux).

Basic Mux Skeleton

Let's start by creating the structure of our mux:

function serverMux() {
  function hook(req, res) {
    // To be implemented
  }
  return {
    hook
  };
}

const app = serverMux();

const server = http.createServer((req, res) => {
  app.hook(req, res);
});

The Hook Function

The hook function is our middleman between the server and the mux. It receives the request (req) and response (res) objects from the server and passes them to our mux:

function hook(req, res) {
    requestsQueue.push(requestWrapper(req, res));
    console.log("new request!");
    processRequests();
}

Here, we introduced a few new things:

• requestWrapper: A function to wrap the req and res.

• processRequests: A function to handle the request processing.

• requestsQueue: A basic JavaScript array that will act as our queue for handling requests.

Let's update serverMux to reflect this:

function serverMux() {
    const requestsQueue = [];

    async function processRequests() {
      // To be implemented
    }

    function hook(req, res) {
      requestsQueue.push(requestWrapper(req, res));
      console.log("new request!");
      processRequests();
    }

    return {
      hook
    };
}

Why Use a Queue?

You might be wondering why we’re using a queue instead of handling requests immediately like Express does with app.handle. Well, storing requests in a queue helps simulate an event loop. This will give us better visibility into how requests are processed, one at a time.

Queue Operations

A queue is a first-in, first-out (FIFO) data structure. Just like a line at the store, the request that arrives first gets processed first.

In our case, the requestsQueue is an array. Here’s how we’ll handle enqueueing and dequeueing:

• Enqueue (push): We push requests into the queue with requestsQueue.push(requestWrapper(req, res));

• Dequeue (shift): We pull the next request out of the queue with const c = requestsQueue.shift();

Request Wrapper

The requestWrapper function is a simple utility that wraps the incoming req and res objects, and extracts some useful information:

function requestWrapper(req, res) {
    return {
      url: req.url,
      method: req.method,
      req,
      res
    };
}

In more advanced frameworks like Hono.js, the request wrapper might add additional functionality, such as helper methods for setting headers or parsing body content. For now, we’re keeping things simple and just returning the request and response with the URL and method.

Testing the Queue

Let’s test this out by logging the request queue on every new request. Update your hook function:

function hook(req, res) {
    requestsQueue.push(requestWrapper(req, res));
    console.log("New request queued!", requestsQueue);
    processRequests();
}

Start the server with:

node index.js

Now, open another terminal and make a request to the server:

curl http://localhost:3000

You should see the queue logged in the console. The terminal might look like it's hanging because we haven’t responded to the request yet. You can exit the process manually for now.

Processing Requests

Here’s the full processRequests function:

async function processRequests() {
    while (requestsQueue.length > 0) {
        const c = requestsQueue.shift();
        if (c) {
            const handler = lookupTable[c.url] || lookupTable["/notfound"];
            if (handler) {
                (async function() {
                    handler(c.req, c.res);
                })();
            } else {
                console.log("Missing not found handler!");
            }
        }
    }
}

Let’s break it down:

1. Queue processing: We loop through the queue, dequeueing each request one by one.

2. Handler lookup: For each request, we check if a handler exists in the lookupTable for the URL. If it doesn’t, we fall back to a /notfound handler.

3. Handler execution: We execute the handler, passing the request and response objects.

Lookup Table and Handlers

We need a way to map URLs to their respective handlers. This is where the lookupTable comes in:

const lookupTable = {
    "/": (req, res) => {
      res.writeHead(200, { 'Content-Type': 'text/plain' });
      res.end('Hello, World!\n');
    },
    "/notfound": (req, res) => {
      res.writeHead(404, { 'Content-Type': 'text/plain' });
      res.end('404 Not Found\n');
    }
};

When a request comes in, we check if the URL matches an entry in the table. If it does, we call the corresponding handler function.

For example, calling curl http://localhost:3000 will hit the / route and return "Hello, World!". If you hit a non-existent route like /random, it will trigger the 404 handler.

Registering Handlers

Finally, let’s add a method to register new handlers dynamically:

function serverMux() {
  const lookupTable = {};

  function registerHandler(path, handler) {
    if (typeof path !== 'string' || !path) {
      throw new Error("Path must be a non-empty string");
    }
    if (typeof handler !== 'function') {
      throw new Error("Handler must be a function");
    }
    lookupTable[path] = handler;
  }

  return {
    hook,
    registerHandler
  };
}

Now, we can dynamically register new routes with their handlers:

app.registerHandler("/", (req, res) => {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Home Page\n');
});

app.registerHandler("/about", (req, res) => {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('About Us\n');
});

Here's a full example of registerHandler in action:

const app =  serverMux()


app.registerHandler("/", (req, res)=> {

  res.writeHead(200, { 'Content-Type': 'text/plain' });

  res.end('Hello, World!');



})

app.registerHandler("/about", (req, res)=> {

    res.writeHead(200, { 'Content-Type': 'text/html' });

    res.end('<h1>About Us</h1>');

})


app.registerHandler("/contact", (req, res)=> {

  res.writeHead(200, { 'Content-Type': 'text/html' });

  res.end('<h1>Contact Us</h1>');

})


app.registerHandler("/api/data", (req, res)=> {

  res.writeHead(200, { 'Content-Type': 'application/json' });

  res.end(JSON.stringify({ message: 'Some data from the API' }));

})


app.registerHandler("/notfound", (req, res)=> {

   res.writeHead(404, { 'Content-Type': 'text/plain' });

    res.end('404 Not Found');

})



const server = http.createServer((req, res) => {

   console.log(req.url)

   app.hook(req, res)

});

Notice how similar it is to Express?

app.get("/", (req, res)=> {

})

Just a bit more verbose!

Now, run the server and put it to the test by pasting this into your terminal:

for /l %i in (1,1,100) do curl -X GET http://localhost:3000

This will send 100 requests. Try opening two or more terminals and running the same command concurrently to see how your server handles the load.

Congratulations! You’ve built a basic server multiplexer (mux). It may not revolutionize the world, but it's a solid starting point to understand how routing works in web frameworks.

Wrapping Up

In this article, we took a deep dive into the concept of server-side frameworks, using Express as our primary example. We traced it from its high-level abstractions all the way down to the native TCP server built in C++. Then, to cement these ideas, we built our own simple server mux.

It’s a powerful learning exercise, because we stripped away the magic and dug into the core of how things work. While this example is just the tip of the iceberg, it gives you the tools to explore even deeper. For a challenge, look into how Express handles pattern matching and registering routes—try improving our simple mux!

I left out more advanced topics like updating our queue with a linked list and simulating concurrent requests, so this is something you can explore.

Thanks for reading! I hope you enjoyed this exploration as much as I did writing it. If you have any thoughts, questions, or just want to connect I am on x, feel free to reach out.

And of course, enjoy your timezone!

We just published a course on the freeCodeCamp.org Spanish YouTube channel that is designed to teach you how to develop REST APIs step by step. You will also learn how to connect them to databases.

You will develop your API with TypeScript, Node.js, Express, MySQL, and TypeORM and you will test it using Postman, a platform for testing APIs.

You will start from the basics of Node.js and Express and gradually dive into more advanced concepts that will prepare you for connecting your API to a database. By the end of the course, you will be able to create your own APIs with Node.js and Express.

If you have Spanish-speaking friends, you are welcome to share the Spanish version of this article with them.

This course was created by Leonardo José Castillo. Leonardo is a software developer and content creator who loves teaching programming and sharing his knowledge.

Are you ready? Let's see a quick overview of APIs and what you will learn during the course.

What is an API?

If you ever need to make two applications communicate with each other, APIs are exactly what you are looking for. They are software that you can use to send data between two applications through requests and responses.

💡 Tip: API stands for Application Programming Interface.

The developers of the application that will send the data to the other application will implement an API and document its functionality and endpoints, so other developers can use it and access its resources and data.

💡 Tip: An endpoint is a location in the API that accepts requests and sends back responses.

The developers of the application that will receive data from the API will write code for making these requests, specifying the endpoints and handling the response received from the API appropriately.

Weather API Example

For example, a weather application may access an API to get the current weather data of a location entered by the user.

The developers of the weather app write code to make requests to the weather API, following its guidelines and documentation. The API will then access the data on a database and send it to the client who made the request.

This is the role of APIs. It's a very important role in the world of back-end web development.

You can implement them with many different technologies, including Node.js and Express.

Let's see what they are:

• Node.js is a JavaScript runtime environment that allows you to run JavaScript code outside of the browser.

• Express is a Node.js framework that makes developing servers and APIs much easier.

Learning how to design and implement APIs can open many career opportunities for you.

Career Opportunities

Talking about career opportunities – TypeScript, Node.js, Express, and MySQL, the technologies that you will be practicing in this project, are very popular and in high demand in the programming industry.

To show you how important they are, here we have the results of the Stack Overflow 2023 Developer Survey.

Node.js and Express were the first and fourth most popular web frameworks and technologies:

MySQL was also very high in the rankings. It was the second most popular database:

TypeScript was the fifth most popular language among all respondents:

These results show you how relevant these technologies are are will be for web development in 2024 and beyond.

💡 Tip: During the project, you will also use TypeORM, an Object-Relational Mapping tool that helps you to work with databases in JavaScript, TypeScript, and other programming languages.

APIs Course with Node.js and Express

Great. Now that you know why APIs are so important, let's check out the topics that you will learn about during the course:

• Introduction to Node.js and Express

• Application Architecture

• Dynamic routing

• Controllers

• Database structure

• Connecting the API to a database

• Implementing CRUD operations in TypeScript

• Modeling with TypeORM

• Implementing controllers with TypeORM

And more!

💡 Tip: To build this project, it's recommended to have a basic understanding of TypeScript and web development. If you need to review these topics, we have these courses on the channel:

• Aprende Node.js y Express - Curso desde cero

• Aprende TypeScript - Curso desde cero

If you are ready to start building this API, check out the course in Spanish on the freeCodeCamp.org Spanish YouTube channel:

✍️ Course created by Leonardo José Castillo.

• Youtube: @LeonardoCastillo79

• LinkedIn: Leonardo José Castillo Lacruz

• Twitter: @ljcl79

• GitHub: @ljcl79

In this handbook, I'm going to introduce you to the MERN stack, a widely-used technology stack embraced by many leading companies. I'll guide you through 7 essential steps to start learning these technologies on your own.

By the time you finish reading, you'll have a solid understanding of what the MERN stack entails, its component technologies, and various resources for learning. You'll also have 10 project ideas that you can develop and showcase in your portfolio.

Table of Contents

1. What is the MERN stack?
2. The MERN Stack Roadmap
   • STEP 1: Learn the right amount of HTML, JavaScript, and CSS
   • STEP 2: Get familiar with React
   • STEP 3: Understand REST API's and how a backend server works using Express/Node
   • STEP 4: Storing data with MongoDB and Mongoose
   • STEP 5: Writing tests
   • STEP 6: Using Git
   • STEP 7: Deployments
3. Top Resources to learn the MERN Stack
4. 10 project ideas you can try today
5. Wrapping up the MERN stack journey

What is the MERN stack?

The MERN stack, comprising MongoDB, Express.js, React, and Node.js, is a cohesive set of technologies used for building efficient and scalable web applications.

Its popularity stems from the seamless integration of each component: MongoDB's flexible data handling, Express.js's efficient server-side networking, React's dynamic user interfaces, and Node.js's powerful back-end runtime environment.

For beginners, the MERN stack is a smart choice because it uses JavaScript across all layers, simplifying the learning curve. This uniformity, coupled with a strong community and ample learning resources, makes it an accessible and practical toolkit for anyone looking to dive into full-stack development.

The MERN stack is also heavily utilized in the industry, favored by startups and large enterprises alike for its efficiency and the robust, modern web applications it can produce. This industry adoption not only validates its effectiveness but also opens up numerous career opportunities for those skilled in these technologies.

Let's look at a brief overview of what each part of the MERN Stack looks like:

Frontend (React)

The frontend of a website is like the dining area of a restaurant. It's everything you see and interact with directly on a website – the layout, design, buttons, and text.

Example: When you visit a website and see a beautiful homepage, interact with menus, or fill out forms, you're experiencing the frontend.

Backend (Node.js)

The backend is like the kitchen in a restaurant. It's where all the behind-the-scenes work happens. It includes the server, applications, and databases that work together to process the information you see on the frontend.

Example: When you order food (submit a form on the website), the kitchen (backend) processes the order (the data) and prepares your meal (the information or service you requested).

Database (MongoDB)

A database in web development is similar to a restaurant’s pantry or storage where all the ingredients (data) are kept. It stores and manages all the information needed by the website, like user profiles, content, and other data.

Example: In an online store, the database stores product information, prices, user reviews, and customer details.

REST APIs (Express)

REST APIs are like the waiters in a restaurant. They are the messengers or go-betweens for the frontend and backend. They take requests (like orders) from the frontend (customer), fetch or update data in the backend (kitchen), and then return responses (prepared orders).

The terms POST, PUT, DELETE, and GET are types of requests used in REST APIs:

• POST: Used to create new data. Like placing a new order in the restaurant.
• PUT: Used to update existing data. Similar to changing an order you've already placed.
• DELETE: Used to remove data. Like cancelling an order.
• GET: Used to retrieve data. Comparable to asking about the menu or checking the status of your order.

The MERN Stack Roadmap

STEP 1: Learn the right amount of HTML, JavaScript, and CSS

The MERN stack uses JavaScript heavily, so its a natural first step to learn. In this section you will look at the main things you will use day-to-day when creating full-stack MERN apps.

Understanding JavaScript is like knowing the right amount of ingredients needed for a recipe. You don't need to master everything at once, just the essential ingredients that make your particular dish (or your web project) come to life.

Variables

Variables in JavaScript are like labelled jars in your kitchen. You can store things in them (like numbers, text) and use these jars later in your cooking (or coding).

Example: A variable storing the user's name, so you can use it later to say "Hello, [Name]!"

Functions

Functions are like recipes in a cookbook. They are sets of instructions that perform a specific task. You can reuse these recipes whenever you need to perform that task again.

Example: A function that calculates the total price of items in a shopping cart.

Objects & Arrays

Objects are like information cards holding details about something (like a contact card), and arrays are like lists.

Example of an Object: A card holding a user's information (name, age, email).
Example of an Array: A list of all the user's favorite book titles.

If/else Statements, Switch Statements

These are like decision-making processes. If/else statements are like choosing what to wear based on the weather, and switch statements are like a more complex decision, like choosing what to cook based on multiple ingredients you have.

Example: If it's raining (if), take an umbrella (else), take sunglasses.

Callbacks/Promises/Async Await

These are ways to handle tasks that take some time, like ordering food and waiting for it. Callbacks are like calling a friend to do something when they’re free. Promises are like your friend promising to do it. Async-await is like making a plan to do tasks one after another in an organized way.

Example: Ordering a coffee (a task) and waiting to get it before leaving the cafe (ensuring order of actions).

ECMAScript (Template Strings, Destructuring Assignment, Spread Operator, Default Parameters, and so on)

These are advanced tools and shortcuts in JavaScript to make coding easier and cleaner. It's like having a food processor in your kitchen that makes chopping and mixing faster and more efficient.

Example: Automatically creating a welcome message like "Hello, [Name]!" without manually joining words and variables.

TypeScript

TypeScript is like JavaScript but with more rules for organizing your code (like a more detailed recipe book). It helps in managing larger projects by adding types to your code, making sure you don’t mix incompatible ingredients. This guide teaches you TypeScript basics.

Example: Specifying that a function should only take a number as an input, not text or anything else.

STEP 2: Get Familiar with React

Once you've got a feel for JavaScript, its time to venture into the wonderful world of frontend development.

React.js is a popular JavaScript library used for building user interfaces, particularly known for its efficiency in rendering dynamic, interactive web pages. It enables developers to create large web applications that can change data, without reloading the page, making for a smoother user experience.

Below is a list of common things you want to know when working with React.

Components

Think of components as individual LEGO blocks in a large LEGO model. Each block is a small, reusable piece that you can use to build different parts of your web application.

Example: A 'button' component in a website that can be used in many places, like for submitting a form or closing a pop-up.

JSX (JavaScript XML)

JSX lets you write your website's design code (like HTML) inside your JavaScript code. It's like writing the recipe and the cooking instructions in one place for easier reference.

Example: Writing a piece of JSX code that includes both JavaScript and HTML-like tags to create a user login form.

Props (Properties)

Props are like instructions or settings you pass to your LEGO blocks (components) to tell them what they should look like or do.

Example: Passing a 'color' prop to a 'button' component to make it red or blue depending on the situation.

State

State is like a personal notebook for each component, where it keeps track of its own information, like whether a button is clicked or not. Here's a course all about state management in React.

Example: A 'like' button keeping track of whether it has been clicked (liked) or not.

Hooks

Hooks are special tools in React that let you add features like state to your components without needing to use complex code structures.

Example: Using the useState hook to keep track of a counter in a component.

Event Handling

This is how you tell a component to do something when a user interacts with it, like clicking a button or entering text in a form.

Example: Setting up an event handler so that when a user clicks a 'submit' button, it sends their information to the server.

Conditional Rendering

This is like having a magic painting that can change its picture based on certain conditions. In React, you can show different components based on different conditions.

Example: Showing a 'login' button if a user is not logged in, and a 'logout' button if they are.

Lists and Keys

Keys are like name tags for items in a list. They help React keep track of which items are new, changed, or removed.

Example: Displaying a list of messages in a chat app, where each message has a unique key.

Context API

The Context API a way to share information (like a theme setting or user data) across many components without passing the information through each level manually.

Example: Using Context API to share the current logged-in user's information across different components in a web app.

Fragment

Fragments let you group several components or elements together without adding extra layers to the website. It's like putting multiple LEGO pieces on a baseplate without the baseplate being part of the final model.

Example: Grouping a list of items together in a menu without adding extra wrappers or divs around them.

STEP 3: Understand REST API's and how a backend server works using Express/Node

Every UI needs a way to store and retrieve the data it needs to make the frontend work. This is where our backend comes in.

In the MERN stack, the backend is composed of 3 main bits: Express, a Node.js server, and a database. We'll cover the database shortly, for now we'll focus on the Express/Node pieces of the MERN stack, as they're closely related.

Express.js is a lightweight, flexible framework for Node.js, designed to build web applications and REST APIs with ease and efficiency. Node.js is a powerful JavaScript runtime that allows for the development of scalable server-side applications, making the duo a popular choice for backend development.

Node.js: Foundation for Building Web Applications

Think of Node.js as the foundation for constructing a modern building. It's a platform that lets you build web applications, similar to how you would use a set of basic tools and materials to start building a house. You might hear this being to referred to as the "backend".

Example: Building a chat application where multiple users can send messages in real-time.

Express: Streamlining REST API Development

Express is a helper tool for Node.js. It's like having a pre-built kit that makes building certain parts of your house easier and faster, providing templates and shortcuts so you don't have to start from scratch.

Example: Using Express to quickly set up routes for a website, like a route to a contact page or a product catalog.

Modules and Packages: Ready-Made Components

In Node.js, modules are like pre-made components or sections of a house (like a bathroom unit or a kitchen set) that you can simply choose and add to your building project.

Example: Adding a 'date-time' module to display current times and dates on your web application.

Node Package Manager (NPM): The Tool and Material Warehouse

NPM acts like a vast warehouse where you can find all sorts of tools and materials (modules) you might need. It's a central place to get additional resources for building your web applications.

Example: Installing 'body-parser' from npm to handle JSON data in your web application.

Routing: Directing Web Traffic

Routing in Express is like setting up roads and paths in a housing complex. It's about directing the flow of traffic (data and user requests) to different parts of your web application.

Example: Creating routes in an online store, like /products for the product list and /products/:id for individual product details.

Middleware: Additional Functional Layers

Middleware in Express can be seen as extra layers or services in your building, like security, plumbing, or electrical systems. They add specific functionalities to your web application.

Example: Adding 'cookie-parser' middleware to handle cookies in your web application.

Request and Response: Communication Channels

Requests and responses in Express are like sending and receiving mail or messages. They are the way your web application communicates with users, sending them data or receiving their inputs.

Example: Your application receiving a user's login request (request) and then sending back a confirmation message (response).

Environment Variables: Secure Storage Spaces

Think of environment variables as secure storage spaces or safes in your building. They're used to store sensitive information like passwords or personal settings, keeping them secure and separate from the main construction.

Example: Storing the database connection string in an environment variable to keep it secure.

Security: Building Safeguards

In the context of web applications, security is about building safeguards into your project. It's like installing locks, security systems, and fire safety measures in a building to protect it from various threats.

Example: Implementing HTTPS to secure data transmission and using JWT (JSON Web Tokens) for user authentication to protect user data.

STEP 4: Storing data with MongoDB and Mongoose

MongoDB is a NoSQL database that offers high flexibility and scalability for storing and managing data, making it ideal for handling large volumes and diverse types of data.

Mongoose is an Object Data Modeling (ODM) library for MongoDB, providing a straightforward schema-based solution to model application data, enhancing database interaction with useful features and validation.

MongoDB: A NoSQL Database

MongoDB is a type of database that stores data in a flexible, JSON-like format. This makes it different from traditional databases, which use tables and rows. It's great for handling large volumes of data and is very scalable.

Example: Using MongoDB to store user profiles where each profile can have different fields.

Collections and Documents

In MongoDB, data is stored in 'collections', which are similar to tables in a relational database. Inside these collections, data is stored in 'documents'. Think of a document as a single record in your collection, like an entry in a diary or a contact in an address book.

Example: A 'users' collection with documents each representing a user with details like name and email.

Mongoose: A MongoDB Object Modeling Tool

Mongoose is a library for Node.js that makes it easier to interact with MongoDB. It provides a straightforward, schema-based solution to model your application data. It's like having a personal assistant to help manage the communication between your application and MongoDB.

Example: Using Mongoose to easily add, retrieve, and manage user data in a MongoDB database.

Schemas

In Mongoose, a schema is a structure that defines the format of the data to be stored in MongoDB (like defining fields and data types). Think of it as a blueprint for how your data should look.

Example: Creating a Mongoose schema for a blog post with fields like title, author, and content.

Models

A model in Mongoose acts as a constructor, compiled from the Schema definitions. It represents documents in a MongoDB database. Models are responsible for creating and reading documents from the underlying MongoDB database.

Example: Defining a 'User' model based on a user schema to interact with the 'users' collection in the database.

CRUD Operations

CRUD stands for Create, Read, Update, Delete. These are the basic operations you can perform on the database. Mongoose provides easy methods to execute these operations on your data.

Example: Using Mongoose methods to add new users, find users by name, update user information, or delete users.

Connecting to MongoDB

You can use Mongoose to connect your Node.js application to a MongoDB database. This is like setting up a phone line between your application and the database so they can talk to each other.

Example: Writing a Mongoose connect function to link your Node.js application to a MongoDB database hosted on Atlas.

Querying Data

Mongoose allows you to query your MongoDB database. This means you can search for specific data, filter your data based on certain criteria, and more.

Example: Using a Mongoose query to find all blog posts written by a specific author.

Data Validation

Mongoose provides built-in validation. This is a way to make sure the data being saved to your database is in the right format, like checking if an email address looks like an email address.

Example: Defining a schema in Mongoose where the email field must match the format of an email address.

Middleware (Pre and Post Hooks)

Mongoose middleware are functions which can be executed automatically before or after certain operations, like saving a document. They're useful for complex logic like hashing passwords before saving them to the database.

Example: Using a pre-save middleware in Mongoose to hash user passwords before saving them to the database.

Indexes

Indexes in MongoDB are used to improve the performance of searches. They are similar to indexes in a book, helping the database find data faster.

Example: Creating an index on the 'email' field in a user collection to speed up the search for users by email.

Aggregation

Aggregation in MongoDB is a powerful way to process data and get computed results. It's like having a sophisticated calculator to perform complex operations on your data, such as summing up values or averaging them.

Example: Using MongoDB's aggregation framework to calculate the average number of comments on blog posts.

STEP 5: Writing Tests

Testing in software development is like a safety check to ensure everything in your application works as expected. It's a crucial step in the development process where you look for bugs or issues before your users do. Think of it like proofreading an essay or checking a car before a road trip – it helps catch and fix problems early.

There are different types of testing, each serving a unique purpose:

Unit Testing

This is the most basic form of testing. Here, you test individual parts of your code (like functions or components) in isolation. It's like checking each light bulb in a string of Christmas lights.

In the MERN stack, tools like Jest or Mocha are commonly used for this. They let you write small tests to check if a specific part of your application behaves as expected.

Integration Testing

This type of testing checks how different parts of your application work together. It's like making sure all the light bulbs light up when connected and the string is plugged in.

For the MERN stack, you might still use Jest or Mocha, combined with other tools like Chai for assertions, to ensure that different components or services in your application interact correctly.

End-to-End (E2E) Testing

Here, you test your application's workflow from start to finish. It's like checking if the entire Christmas tree lights up and twinkles as expected.

For a MERN stack application, Cypress or Selenium are popular choices. They simulate real user scenarios, ensuring the entire application, from the front end in React to the back end with Express and Node.js, functions smoothly together.

Performance Testing

This checks if your application can handle stress, like heavy traffic or data processing. It's akin to ensuring the Christmas tree lights don't blow a fuse when they're all on. Tools like Loader.io or Apache JMeter can be used here.

Each type of testing serves to ensure a different aspect of your application is working correctly, and together, they contribute to building a robust, reliable, and user-friendly application.

By employing these tests in the MERN stack, you not only catch bugs early but also maintain a high standard of quality for your application.

STEP 6: Using Git

Git is a version control system, a tool that tracks changes in your code over time. Think of it like a detailed diary for your coding project. Every time you make changes to your code, Git keeps a record of what was changed, when, and by whom. This becomes incredibly useful when you’re working on complex projects, like those involving the MERN stack.

Why is Git so important when building with the MERN stack, or any software for that matter?

Collaboration

Git is like a team playbook. It allows multiple developers to work on the same project without stepping on each other's toes.

Everyone can work on different features or parts of the application simultaneously (like MongoDB database schemas, React components, or Express.js routes). Git helps manage these contributions, ensuring changes can be merged smoothly into the main project.

Tracking Changes

Imagine you've made changes to your React component, and suddenly things aren't working as they used to. Git allows you to go back in time and see what changes were made and by whom.

This historical data is invaluable for understanding how your project evolved and for fixing issues without having to start from scratch.

Branching and Merging

Git’s branching feature lets you diverge from the main line of development and experiment with new features or ideas in a controlled way.

You can create a branch, make your changes, and then merge those changes back into the main project when they're ready. This ensures the main project (often referred to as the 'master' or 'main' branch) remains stable.

Backup and Restore

With Git, your project's entire history is stored in the repository. If anything goes wrong, you can roll back to a previous state. It’s like having a fail-safe backup system.

Documentation

Commit messages in Git provide a narrative for your project. They allow you to document what changes were made and why, which is extremely helpful for both your future self and other developers who might work on the project.

When building applications with the MERN stack, Git offers a safety net and a collaborative workspace. It keeps your project organized, tracks every change, and allows multiple developers to work together more efficiently.

Using Git is essential for managing complex development projects in today's software development world.

STEP 7: Deployments

Code deployment is the process of taking code written by developers and making it operational on a live environment where users can interact with it.

In the context of the MERN stack, which involves technologies like MongoDB, Express.js, React, and Node.js, deployment is the final step in the journey of bringing a full-stack application to life.

Imagine you've built a house (your web application). Deployment is like moving it from the construction site (development environment) to its actual location where people can live in it (production environment). This process involves several key steps to ensure that everything works as expected when users access your app.

Preparing for Deployment

Before deploying, you need to prepare your application. This involves ensuring your code is complete and tested, dependencies are properly managed, and your app is configured for the production environment.

For MERN stack applications, this might mean setting up environment variables, configuring your database (MongoDB) connection for production, and optimizing your React front-end for performance.

Hosting and Servers

Choosing where to host your application is crucial. For MERN stack apps, you can use cloud-based hosting services like AWS, Heroku, or DigitalOcean. These platforms offer services to host both your Node.js backend and MongoDB database, and they often provide additional features like scaling, monitoring, and security.

Continuous Integration/Continuous Deployment (CI/CD)

CI/CD is a methodology that automates the deployment process. Whenever you make changes to your codebase (like fixing a bug or adding a new feature), CI/CD tools automatically test your code and deploy it if the tests pass. This ensures that your application is always up-to-date with the latest changes.

Tools like Jenkins, Travis CI, or GitHub Actions are commonly used for this purpose.

Monitoring and Maintenance

After deployment, it’s important to monitor your application for any issues and perform regular maintenance. This could involve checking server logs, updating dependencies, or rolling out fixes for any bugs that might arise.

Rollbacks

A good deployment strategy also includes plans for rollbacks. If something goes wrong after deployment, you need to be able to revert to the previous version quickly to minimize downtime.

In the MERN stack, each component (MongoDB, Express.js, React, and Node.js) might require specific considerations for deployment. For instance, you might use Docker containers to package your Node.js and Express.js backend, ensuring it runs consistently across different environments. React apps might be minified and bundled for optimal performance.

In essence, deployment in the MERN stack is about getting your application from your local machine to a server where it can serve users reliably and efficiently. It involves careful planning, choosing the right hosting solutions, automating the process as much as possible, and being prepared to address issues post-deployment.

Top Resources to Learn the MERN Stack

Now that you have a pretty good idea of what you need to learn to master the MERN stack, lets look at a list of free resources you can start using today to begin your journey!

• The freeCodeCamp curriculum is one of the best places to learn how to code for free. The curriculum is super in depth and covers alot of the topics mentioned in this post.
• The freeCodeCamp YouTube channel has a ton of free courses you can try as well. This MERN stack book project is an excellent one for beginners.
• Full Stack open is another good resource. It doesn't teach MongoDB but you learn about SQL instead which is equally as useful.
• My MERN stack hotel booking app project on YouTube is a free 15 hour course where you will learn everything talked about in this post.

10 Project Ideas You Can Try Today

One of my favourite ways to learn new technologies is to build projects. If you're struggling with coming up with ideas, here's 10 projects you can start building today. I'll be building some of these on my YouTube channel so feel free to subscribe to stay up to date!

Personal Blog Website

• Functionality: Users can read posts, and the admin can create, edit, or delete posts.
• Learning Focus: Basic CRUD operations, user authentication, integrating React with Express API, and using MongoDB for data storage.

Task Manager (To-Do List)

• Functionality: Users can add, view, edit, and delete tasks. Tasks can have deadlines and priority levels.
• Learning Focus: React state management, Express route handling, MongoDB operations, and basic UI development.

Simple E-commerce Site

• Functionality: Display products, add to cart, and checkout functionality. Admin features for adding or removing products.
• Learning Focus: React components for product listing, cart management, Express.js for product APIs, MongoDB for product data, and handling user inputs.

Recipe Sharing Application

• Functionality: Users can post, view, edit, and delete recipes. Implement a rating or comment system for interaction.
• Learning Focus: File upload for images, user-generated content management, and MongoDB schema design.

Budget Tracker

• Functionality: Users can input their expenses and income, categorize them, and view a summary of their finances.
• Learning Focus: Handling forms in React, creating RESTful services with Express, and effective data structuring in MongoDB.

Event Planning Application:

• Functionality: Users can create and manage events, send invitations, and track RSVPs.
• Learning Focus: Date handling in JavaScript, complex MongoDB documents, and Express middleware for authentication.

Fitness Tracker:

• Functionality: Log workouts, track progress over time, set fitness goals.
• Learning Focus: Data visualization with React, creating API endpoints for varied data types, and MongoDB for storing time-series data.

Chat Application:

• Functionality: Real-time chat rooms, private messaging, and user profiles.
• Learning Focus: WebSockets with Node.js for real-time communication, MongoDB for message storage, and React for dynamic UI updates.

Book Review Platform:

• Functionality: Users can post book reviews, rate books, and browse reviews by other users.
• Learning Focus: Integrating external APIs for book data, user-generated content moderation, and complex querying in MongoDB.

Local Business Directory:

• Functionality: Listing of local businesses with categories, user reviews, and contact information.
• Learning Focus: Geolocation, advanced querying and indexing in MongoDB, and creating a responsive layout with React.

Wrapping Up the MERN Stack Journey

As we reach the end of our exploration of the MERN stack, I hope this guide has illuminated the path for your journey into the world of full-stack development.

We've covered the fundamental components – MongoDB, Express.js, React, and Node.js – and delved into the practical aspects of building, testing, and deploying applications using this versatile stack.

Remember, the road to mastering the MERN stack is both challenging and rewarding. Each project you undertake will add to your skills and confidence. Use the resources and project ideas provided as stepping stones to build your portfolio and deepen your understanding.

The beauty of the MERN stack lies in its community-driven nature, so don't hesitate to seek help, collaborate, and share your experiences.

If you want to get in touch, you can reach me over at LinkedIn, or drop a comment on my YouTube channel. Good luck and happy coding!

As you continue to develop your software, you must also continue to integrate it with previous code and deploy it to servers.

Manually doing this is a time-consuming process that can occasionally result in errors. So we need to do this in a continuous and automated manner – which is what you will learn in this article.

We'll go over how you can improve your MERN (MongoDB, Express, React, and NodeJs) app development process by setting up a CI/CD pipeline with Jenkins. You'll see how to automate deployment for faster, more efficient releases.

Let's Get Started

Prerequisites

• Basic understanding of MERN stack technologies.
• Basic understanding of Docker.
• Get source code from GitHub

The Problem

Consider this productivity app – it's a MERN project that we are going to use in this article. There are numerous steps we must complete, from building the application to pushing it to the Docker hub.

First, we must run tests with a command to determine whether all tests pass or not. If all tests pass, we build the Docker images and then push those images to Docker Hub. If your application is extremely complex, you may need to take additional steps.

Now, imagine that we're doing everything manually, which takes time and can lead to mistakes.

Waiting for deployment without devops meme

The Solution

To address this problem, we can create a CI/CD Pipeline. So, whenever you add a feature or fix a bug, this pipeline gets triggered. This automatically performs all of the steps from testing to deploying.

What is CI/CD and Why is it Important?

Continuous Integration and Continuous Deployment is a series of steps performed to automate software integration and deployment. CI/CD is the heart of DevOps.

CI/CD steps

From development to deployment, our MERN app goes through four major stages: testing, building Docker images, pushing to a registry, and deploying to a cloud provider. All of this is done manually by running various commands. And we need to do this every time a new feature is added or a bug is fixed.

But this will significantly reduce developer productivity, which is why CI/CD can be so helpful in automating this process. In this article, we will cover the steps up until pushing to the registry.

CI/CD meme

The Project

The project we are going to use in this tutorial is a very simple full-stack MERN application.

Project demo

It contains two microservices.

1. Frontend
2. Backend

You can learn more about the project here.

Both of these applications contains a Dockerfile. You can learn how to dockerize a MERN application here.

What is Jenkins?

To run a CI/CD pipeline, we need a CI/CD server. This is where all of the steps written in a pipeline run.

There are numerous services available on the market, including GitHub Actions, Travis CI, Circle CI, GitLab CI/CD, AWS CodePipeline, Azure DevOps, and Google Cloud Build. Jenkins is one of the popular CI/CD tools, and it's what we'll use here.

How to Set Up Jenkins Server on Azure

Because Jenkins is open source and it doesn't provide a cloud solution, we must either run it locally or self-host on a cloud provider. Now, running locally can be difficult, particularly for Windows users. As a result, I've chosen to self-host it on Azure for this demo.

If you want to run locally or self-host somewhere other than Azure (follow these guides by Jenkins), skip this section and proceed to the How to Configure Jenkins section.

First, you'll need to sign in to your Azure account (Create one if you don't have one already). Open Azure Cloud Shell.

Opening Azure Cloud Shell

Then create a directory called jenkins to store all the Jenkins config, and switch to that directory:

mkdir jenkins
cd jenkins

Create a file called cloud-init-jenkins.txt. Open with nano or vim,

touch cloud-init-jenkins.txt
nano cloud-init-jenkins.txt

and paste this code into it:

#cloud-config
package_upgrade: true
runcmd:
  - sudo apt install openjdk-11-jre -y
  - wget -qO - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
  - sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
  - sudo apt-get update && sudo apt-get install jenkins -y
  - sudo service jenkins restart

Here, we'll use this file to install Jenkins after creating a virtual machine. First, we install openjdk, which is required for Jenkins to function. The Jenkins service is then restarted after we install it.

Next, create a resource group. (A resource group in Azure is like a container that holds all the related resources of a project in one group. Learn more about resource groups here.)

az group create --name jenkins-rg --location centralindia

Note: make sure to change the location to the one closest to you.

Now, create a virtual machine.

az vm create \
--resource-group jenkins-rg \
--name jenkins-vm \
--image UbuntuLTS \
--admin-username "azureuser" \
--generate-ssh-keys \
--public-ip-sku Standard \
--custom-data cloud-init-jenkins.txt

You can verify the VM installation with this command:

az vm list -d -o table --query "[?name=='jenkins-vm']"

Don't be confused. This command simply displays JSON data in a tabular format for easy verification.

Jenkins server runs on port 8080, so we need to expose this port on our VM. You can do that like this:

az vm open-port \
--resource-group jenkins-rg \
--name jenkins-vm  \
--port 8080 --priority 1010

Now we can access the Jenkins dashboard in the browser with the URL http://<your-vm-ip>:8080. Use this command to get the VM IP address:

az vm show \
--resource-group jenkins-rg \
--name jenkins-vm -d \
--query [publicIps] \
--output tsv

You can now see the Jenkins application in your browser.

Jenkins dashboard

As you'll notice, Jenkins is asking us to provide an admin password which is automatically generated during its installation.

But first let's SSH into our virtual machine where Jenkins is installed.

ssh azureuser@<ip_address>

Now, type in the below command to get the password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy and paste it. Then click Continue.

How to Configure Jenkins

First, you'll need to click Install suggested plugins. It will take some time to install all the plugins.

Installing suggested plugins

An admin user is needed to restrict access to Jenkins. So go ahead and create one. After finishing, click Save and continue.

Create an admin user

Now you will be presented with the Jenkins dashboard.

The first step is to install the "Blue Ocean" plugin. Jenkins has a very old interface, which may make it difficult for some people to use. This blue ocean plugin provides a modern interface for some Jenkins components (like creating a pipeline).

To install plugins, go to Manage Jenkins -> click Manage Plugins under "System Configuration" -> Available plugins. Search for "Blue Ocean" -> check the box and click Download now and install after restart.

Blue ocean

Great, we're all set. Now let's create a pipeline.

How to Write a Jenkinsfile

To create a pipeline, we need a Jenkinsfile. This file contains all the pipeline configurations – stages, steps, and so on. Jenkinsfile is to Jenkins as a Dockerfile is to Docker.

Jenkinsfile uses the Groovy syntax. The syntax is very simple. You can understand everything by just looking at it.

Let's start by writing:

pipeline {

}

The word 'agent' should be the first thing you mention in the pipeline. An agent is similar to a container or environment in which jobs run. You can use multiple agents to run jobs in parallel. You can find more information about Jenkins agents can here.

pipeline {
    agent any
}

Here we are telling Jenkins to use any available agent.

We have a total of 5 stages in our pipeline:

CI/CD pipeline stages

Stage 1: Checkout code

Different CI/CD tools use different naming conventions. In Jenkins, these are referred to as stages. In each stage we write various steps.

Our first stage is checking out code from a source code management system (in our case, GitHub).

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
    }
}

Commit the changes and push to your GitHub repo.

Since we haven't created any pipelines yet, let's do that now.

Before we begin, we must ensure that Git is installed on our system. If you followed my previous steps to install Jenkins on an Azure VM, Git is already installed.

You can test it by running the following command (make you are still SSHed into the VM):

git --version

If it isn't already installed, you can do so with:

sudo apt install git

Open blue ocean. Click Create new pipeline.

Creating new pipeline

Then select your source code management system. If you chose GitHub, you must provide an access token for Jenkins to access your repository. I recommend clicking on Create an access token here because it is a template with all of the necessary permissions. Then click Connect.

Selecting scm

After that, a pipeline will be created. Since our repository already contains a Jenkinsfile, Jenkins automatically detects it and runs the stages and steps we mentioned in the pipeline.

If everything went well, the entire page will turn green. (Other colors: blue indicates that the pipeline is running, red indicates that something went wrong in the pipeline, and gray indicates that we stopped the pipeline.)

Stage one successful

Stage 2: Run frontend tests

In general, all the CI/CD pipelines contains some tests that needs to be run before deploying. So I added simple tests to both the frontend and backend. Let's start with the frontend tests.

stage('Client Tests') {
    steps {
        dir('client') {
            sh 'npm install'
            sh 'npm test'
        }
    }
}

We're changing the directory to client/ because that's where the frontend code is. And then install the dependencies with npm install and run the tests with npm test in a shell.

Again, before we restart the pipeline, we have to make sure node and npm are installed or not. Install node and npm with these commands in the virtual machine:

curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -

After that, run the following:

sudo apt-get install -y nodejs

Now, commit the code and restart the pipeline.

Run client tests

Stage 3: Run backend tests

Now do the same thing for the backend tests.

But there is one thing we need to do before we proceed. If you take a look at the codebase and activity.test.js, we are using a few environment variables. So let's add these environment varibales in Jenkins.

How to add environment variables in Jenkins

To add environment variables, go to Manage Jenkins -> click Manage Credentials under "Security" -> System -> Global credentials (unrestricted) -> click + Add Credentials.

For Kind select "Secret text", leave Scope default, and for Secret write the secret value and ID. This is what we use when using these environment variables in the Jenkinsfile.

Add the following env variables:

Environment variables

Then in the Jenkinsfile, use these env variables:

environment {
    MONGODB_URI = credentials('mongodb-uri')
    TOKEN_KEY = credentials('token-key')
    EMAIL = credentials('email')
    PASSWORD = credentials('password')
}

Add a stage to install dependencies, set these variables in the Jenkins environment, and run the tests:

stage('Server Tests') {
    steps {
        dir('server') {
            sh 'npm install'
            sh 'export MONGODB_URI=$MONGODB_URI'
            sh 'export TOKEN_KEY=$TOKEN_KEY'
            sh 'export EMAIL=$EMAIL'
            sh 'export PASSWORD=$PASSWORD'
            sh 'npm test'
        }
    }
}

Again, commit the code and restart the pipeline.

Run server tests

Stage 4: Build Docker images

Now, we have to specify a step to build the Docker images from the Dockerfiles.

Before we proceed, install Docker in the VM (if you don't already have it installed).

To install Docker:

sudo apt install docker.io

Add the user jenkins to the docker group so that Jenkins can access the Docker daemon – otherwise you'll get a permission denied error.

sudo usermod -a -G docker jenkins

Then restart the jenkins service.

sudo systemctl restart jenkins

Add a stage in the Jenkinsfile.

stage('Build Images') {
    steps {
        sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
        sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
    }
}

Commit the code and restart the pipeline.

Build docker images

Stage 5: Push images to the registry

As a final stage, we will push the images to Docker hub.

Before that, add your docker hub username and password to the Jenkins credentials manager, but for Kind choose "Username with password".

Username with password type credential

Add the final stage where we login and push images to Docker hub.

stage('Push Images to DockerHub') {
    steps {
        withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
            sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
            sh 'docker push rakeshpotnuru/productivity-app:client-latest'
            sh 'docker push rakeshpotnuru/productivity-app:server-latest'
        }
    }
}

Push images to dockerhub

Here is the complete Jenkinsfile:

// This is a Jenkinsfile. It is a script that Jenkins will run when a build is triggered.
pipeline {
    // Telling Jenkins to run the pipeline on any available agent.
    agent any

    // Setting environment variables for the build.
    environment {
        MONGODB_URI = credentials('mongodb-uri')
        TOKEN_KEY = credentials('token-key')
        EMAIL = credentials('email')
        PASSWORD = credentials('password')
    }

    // This is the pipeline. It is a series of stages that Jenkins will run.
    stages {
        // This state is telling Jenkins to checkout the source code from the source control management system.
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        // This stage is telling Jenkins to run the tests in the client directory.
        stage('Client Tests') {
            steps {
                dir('client') {
                    sh 'npm install'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to run the tests in the server directory.
        stage('Server Tests') {
            steps {
                dir('server') {
                    sh 'npm install'
                    sh 'export MONGODB_URI=$MONGODB_URI'
                    sh 'export TOKEN_KEY=$TOKEN_KEY'
                    sh 'export EMAIL=$EMAIL'
                    sh 'export PASSWORD=$PASSWORD'
                    sh 'npm test'
                }
            }
        }

        // This stage is telling Jenkins to build the images for the client and server.
        stage('Build Images') {
            steps {
                sh 'docker build -t rakeshpotnuru/productivity-app:client-latest client'
                sh 'docker build -t rakeshpotnuru/productivity-app:server-latest server'
            }
        }

        // This stage is telling Jenkins to push the images to DockerHub.
        stage('Push Images to DockerHub') {
            steps {
                withCredentials([usernamePassword(credentialsId: 'dockerhub', passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
                    sh 'docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD'
                    sh 'docker push rakeshpotnuru/productivity-app:client-latest'
                    sh 'docker push rakeshpotnuru/productivity-app:server-latest'
                }
            }
        }
    }
}

Pipeline ran successfully

Conclusion

In summary, let's review what we've covered:

• We explored the significance of implementing Continuous Integration and Continuous Deployment (CI/CD) in software development.
• We delved into the fundamentals of Jenkins and acquired knowledge on how to deploy a Jenkins server on the Azure cloud platform.
• We customized Jenkins to meet our specific requirements.
• Lastly, we wrote a Jenkinsfile and built a pipeline utilizing the user-friendly interface of Jenkins Blue Ocean.

That's all for now! Thanks for reading 🙂.

Connect with me on twitter.

In this tutorial, you will learn what node modules are. You will also learn about the three types of node modules. And we'll go over the right way to use them in your applications.

What is a Module in JavaScript?

In simple terms, a module is a piece of reusable JavaScript code. It could be a .js file or a directory containing .js files. You can export the content of these files and use them in other files.

Modules help developers adhere to the DRY (Don't Repeat Yourself) principle in programming. They also help to break down complex logic into small, simple, and manageable chunks.

Types of Node Modules

There are three main types of Node modules that you will work with as a Node.js developer. They include the following.

• Built-in modules

• Local modules

• Third-party modules

Built-in Modules

Node.js comes with some modules out of the box. These modules are available for use when you install Node.js. Some common examples of built-in Node modules are the following:

• http

• url

• path

• fs

• os

You can use the built-in modules with the syntax below.

const someVariable = require('nameOfModule')

You load the module with the require function. You need to pass the name of the module you're loading as an argument to the require function.

Note: The name of the module must be in quotation marks. Also, using const to declare the variable ensures that you do not overwrite the value when calling it.

You also need to save the returned value from the require function in someVariable. You can name that variable anything you want. But often, you will see programmers give the same to the variable as the name of the module (see example below).

const http = require('http') 

server = http.createServer((req, res) => { 
    res.writeHead(200, {'Content-Type': 'text/plain'}) 
    res.end('Hello World!')
})

server.listen(3000)

You use the require function to load the built-in http module. Then, you save the returned value in a variable named http.

The returned value from the http module is an object. Since you've loaded it using the require function, you can now use it in your code. For example, call the .createServer property to create a server.

Local Modules

When you work with Node.js, you create local modules that you load and use in your program. Let's see how to do that.

Create a simple sayHello module. It takes a userName as a parameter and prints "hello" and the user's name.

function sayHello(userName) {
    console.log(`Hello ${userName}!`)
}

module.exports = sayHello

First, you need to create the function. Then you export it using the syntax module.exports. It doesn't have to be a function, though. Your module can export an object, array, or any data type.

How to load your local modules

You can load your local modules and use them in other files. To do so, you use the require function as you did for the built-in modules.

But with your custom functions, you need to provide the path of the file as an argument. In this case, the path is './sayHello' (which is referencing the sayHello.js file).

const sayHello = require('./sayHello')
sayHello("Maria") // Hello Maria!

Once you've loaded your module, you can make a reference to it in your code.

Third-Party Modules

A cool thing about using modules in Node.js is that you can share them with others. The Node Package Manager (NPM) makes that possible. When you install Node.js, NPM comes along with it.

With NPM, you can share your modules as packages via the NPM registry. And you can also use packages others have shared.

How to use third-party packages

To use a third-party package in your application, you first need to install it. You can run the command below to install a package.

npm install <name-of-package>

For example, there's a package called capitalize. It performs functions like capitalizing the first letter of a word.

Running the command below will install the capitalize package:

npm install capitalize

To use the installed package, you need to load it with the require function.

const capitalize = require('capitalize)

And then you can use it in your code, like this for example:

const capitalize = require('capitalize')
console.log(capitalize("hello")) // Hello

This is a simple example. But there are packages that perform more complex tasks and can save you loads of time.

For example, you can use the Express.js package which is a Node.js framework. It makes building apps faster and simple. To learn more about NPM, read this freeCodeCamp article on the Node Package Manager.

Conclusion

In this article, you learned about what Node modules are and the three types of node modules. You also learned about how to use the different types in your application.

Thanks for reading. And happy coding!

In this article, you will learn about Node.js. You will learn the following:

• What is Node.js?

• How the Node.js environment differs from the browser.

• Why you should learn Node.js.

• How to get started with Node.js.

• Resources to help you learn Node.js.

What is Node.js?

"Node.js is an open-source and cross-platform JavaScript runtime environment." - Nodejs.dev Docs

This sounds like a cool, straightforward answer. But for a beginner, this definition might raise further questions. So let's break it down and understand what it means.

Node.js is open-source: This means that the source code for Node.js is publicly available. And it's maintained by contributors from all over the world. The Node.js contribution guide shows you how to contribute.

Node.js is cross-platform: Node.js is not dependent on any operating system software. It can work on Linux, macOS, or Windows.

Node.js is a JavaScript runtime environment: When you write JavaScript code in your text editor, that code cannot perform any task unless you execute (or run) it. And to run your code, you need a runtime environment.

Browsers like Chrome and Firefox have runtime environments. That is why they can run JavaScript code. Before Node.js was created, JavaScript could only run in a browser. And it was used to build only front-end applications.

Node.js provides a runtime environment outside of the browser. It's also built on the Chrome V8 JavaScript engine. This makes it possible to build back-end applications using the same JavaScript programming language you may be familiar with.

Differences Between the Browser and Node.js Runtime Environments

Both the browser and Node.js are capable of executing JavaScript programs. But there are some key differences that you need to know. They include the following.

Access to the DOM APIs

With the browser runtime, you can access the Document Object Model (DOM). And you can perform all the DOM operations. But Node.js does not have access to the DOM.

Node.js exposes almost all the system resources to your programs. This means you can interact with the operating system, access the file systems, and read and write to the files. But, you do not have access to operating systems and file systems from the browser.

Window vs Global object

JavaScript has a built-in global object. The JavaScript global object for the browser is called the window object. In Node.js, the global object goes by the name global.

The window object contains methods and properties available only in the browser environment.

Control over runtime versions

With Node.js, you can choose which version to run your server-side application on. As a result, you can use modern JavaScript features without worrying about any version-specific inconsistencies.

Contrast this to the browser runtime environment. As a developer, you have no control over the version of browsers your clients use to access your app.

Loading modules (import vs require keywords)

Node.js offers out-of-the-box support for CommonJS and ES modules. You can load modules using the require keyword (CommonJS syntax) and the import keyword (ES syntax).

Some modern browsers support ES modules. This means you can use import ES modules. But you will still need to create bundles to cater to older browsers that do not support ES modules.

How Much JavaScript Do You Need to Get Started with Node?

If you are an absolute beginner to JavaScript, I recommend that you start with the basics.

Become familiar with basic JavaScript concepts first. Then, you can move on to learning to build server-side applications with Node.js.

There's no way you'll ever exhaust all there is to learn about JavaScript. So, how to determine when you know enough JavaScript to get started with Node.js?

The Node.js documentation provides a list of JavaScript topics to learn before diving deep with Node.js.

Once you have a grasp of JavaScript basics, then you can get started with Node.js

How to Get Started with Node.js

Let's see how you can create your first Node.js application. This section will show you how to run Node.js scripts from the command line.

How to download and install Node.js

First, you need to download and install Node.js. There are different ways you can do that. If you are a beginner, I would suggest that you download Node.js from the official website.

Screenshot of the official Node.js website

Official packages are available on the website for all major platforms (Windows, macOS, and Linux). Download and install the appropriate package for your system.

How to check the Node.js version

To check the Node.js version, run the command node --version in your terminal.
If the installation was successful, you will see the version of Node.js you installed. You should get a response like the screenshot below.

How to run Node.js from the command line

Let's build a simple Hello World app.

Create a new project folder. You can call it my-project. Open the project in your code editor. Inside the folder, create an app.js file.

Add the following code to app.js

As you can see, this is JavaScript code.

You can run the script in the command line by running the command node <fileName>. In this case, the file name is app.js.

Run the following command in your terminal to execute the Hello world. program:

node app.js

You should see the string "Hello world." logged in your terminal like so.

Congratulations! You just ran your first Node.js application.

Should You Learn Node.js?

Here are some reasons why you should consider learning Node.js

Node.js allows you to write JavaScript on both client and server.

One of the advantages of Node.js is that it allows you to work on both the front-end and back-end of your application. And you use one programming language – JavaScript – to do so.

This is good news for front-end developers who work with JavaScript. If you want to start working on the server side, it's easier compared to learning a new back-end language from scratch.

Node has a vibrant community.

As I mentioned earlier in the article, Node.js is open-sourced. It is actively maintained by developers from all over the world.

There is a vibrant community surrounding Node.js. You can find excellent tutorials and solutions to problems when you get stuck.

Node is built on top of Google Chrome's V8 engine.

Node.js is built on top of the Chrome V8 JavaScript engine. This is significant because the V8 engine powers some of Google's in-browser applications like Gmail. As such, Google invests heavily to ensure it offers high performance.

Demand in the market

Many big names like Netflix, Uber, Paypal, and LinkedIn, and others use Node.js. Apart from the big names, many startups also use Node.js in developing their applications.

Learning to work with Node.js will make you a desirable candidate in the job market.

The NPM library

The NPM library is one of the excellent resources that comes with Node.js.
The library contains a registry of over a million packages. A package is a reusable piece of code.

You can create a package for a recurring task or problem and share the code with others via the registry.

You can also download packages that others have shared. For many tasks that developers perform regularly, there are packages available for that.

Resources to Learn Node

If you are curious about learning how to build Node.js applications, I recommend the following resources.

• 8-Hour Node.js and Express.js Course on freeCodeCamp YouTube Channel.

• The freeCodeCamp Backend Development and APIs curriculum

• Nodejs.dev Documentation

Also, below is a link to a video of Ryan Dahl when he first presented Node.js.

Ryan Dahl: Original Node.js presentation at JSConf 2009

Conclusion

A single blog post like this is not enough to learn all there is to know about Node.js. The purpose of this article was to give you an overview of what Node.js is.

If you were not sure what Node.js is, I hope this article addressed your concerns and cleared your confusion.

Thanks for reading. And happy coding!

Express is a Web Framework built upon Node.js.

Node.js is an amazing tool for building networking services and applications.

Express builds on top of its features to provide easy to use functionality that satisfies the needs of the Web Server use-case. It's Open Source, free, easy to extend, and very performant.

There are also lots and lots of pre-built packages you can just drop in and use to do all kinds of things.

You can get a PDF and ePub version of this Express Handbook

Table of Contents

• How to Install Express

• The first "Hello, World" example

• Request Parameters

• How to Send a Response to the Client

• How to Send a JSON Response

• How to Manage Cookies

• How to Work with HTTP Headers

• How to Handle Redirects

• Routing in Express

• Templates in Express

• Express Middleware

• How to Serve Static Assets with Express

• How to Send Files to the Client

• Sessions in Express

• How to Validate Input in Express

• How to Sanitize Input in Express

• How to Handle Forms in Express

• How to Handle File Uploads in Forms in Express

How to Install Express

You can install Express into any project with npm.

If you're in an empty folder, first create a new Node.js project with this command:

npm init -y

then run this:

npm install express

to install Express into the project.

The First "Hello, World" Example

The first example we're going to create is a simple Express Web Server.

Copy this code:

const express = require('express')
const app = express()

app.get('/', (req, res) => res.send('Hello World!'))
app.listen(3000, () => console.log('Server ready'))

Save this to an index.js file in your project root folder, and start the server using this command:

node index.js

You can open the browser to port 3000 on localhost and you should see the Hello World! message.

Those 4 lines of code do a lot behind the scenes.

First, we import the express package to the express value.

We instantiate an application by calling the express() method.

Once we have the application object, we tell it to listen for GET requests on the / path, using the get() method.

There is a method for every HTTP verb: get(), post(), put(), delete(), and patch():

app.get('/', (req, res) => { /* */ })
app.post('/', (req, res) => { /* */ })
app.put('/', (req, res) => { /* */ })
app.delete('/', (req, res) => { /* */ })
app.patch('/', (req, res) => { /* */ })

Those methods accept a callback function – which is called when a request is started – and we need to handle it.

We pass in an arrow function:

(req, res) => res.send('Hello World!')

Express sends us two objects in this callback, which we called req and res. They represent the Request and the Response objects.

Both are standards and you can read more on them here and here.

Request is the HTTP request. It gives us all the request information, including the request parameters, the headers, the body of the request, and more.

Response is the HTTP response object that we'll send to the client.

What we do in this callback is send the 'Hello World!' string to the client, using the Response.send() method.

This method sets that string as the body, and it closes the connection.

The last line of the example actually starts the server, and tells it to listen on port 3000. We pass in a callback that is called when the server is ready to accept new requests.

Request Parameters

I mentioned how the Request object holds all the HTTP request information.

These are the main properties you'll likely use:

How to Send a Response to the Client

In the Hello World example, we used the send() method of the Response object to send a simple string as a response, and to close the connection:

(req, res) => res.send('Hello World!')

If you pass in a string, it sets the Content-Type header to text/html.

If you pass in an object or an array, it sets the application/json Content-Type header, and parses that parameter into JSON.

After this, send() closes the connection.

send() automatically sets the Content-Length HTTP response header, unlike end() which requires you to do that.

How to use end() to send an empty response

An alternative way to send the response, without any body, is by using the Response.end() method:

res.end()

How to set the HTTP response status

Use the status() method on the response object:

res.status(404).end()

or

res.status(404).send('File not found')

sendStatus() is a shortcut:

res.sendStatus(200)
// === res.status(200).send('OK')

res.sendStatus(403)
// === res.status(403).send('Forbidden')

res.sendStatus(404)
// === res.status(404).send('Not Found')

res.sendStatus(500)
// === res.status(500).send('Internal Server Error')

How to Send a JSON Response

When you listen for connections on a route in Express, the callback function will be invoked on every network call with a Request object instance and a Response object instance.

Example:

app.get('/', (req, res) => res.send('Hello World!'))

Here we used the Response.send() method, which accepts any string.

You can send JSON to the client by using Response.json(), a useful method.

It accepts an object or array, and converts it to JSON before sending it:

res.json({ username: 'Flavio' })

How to Manage Cookies

Use the Response.cookie() method to manipulate your cookies.

Examples:

res.cookie('username', 'Flavio')

This method accepts a third parameter, which contains various options:

res.cookie('username', 'Flavio', { domain: '.flaviocopes.com', path: '/administrator', secure: true })

res.cookie('username', 'Flavio', { expires: new Date(Date.now() + 900000), httpOnly: true })

The most useful parameters you can set are:

A cookie can be cleared with:

res.clearCookie('username')

How to Work with HTTP Headers

How to access HTTP headers values from a request

You can access all the HTTP headers using the Request.headers property:

app.get('/', (req, res) => {
  console.log(req.headers)
})

Use the Request.header() method to access one individual request header's value:

app.get('/', (req, res) => {
  req.header('User-Agent')
})

How to change any HTTP header value for a response

You can change any HTTP header value using Response.set():

res.set('Content-Type', 'text/html')

There is a shortcut for the Content-Type header, however:

res.type('.html')
// => 'text/html'

res.type('html')
// => 'text/html'

res.type('json')
// => 'application/json'

res.type('application/json')
// => 'application/json'

res.type('png')
// => image/png:

How to Handle Redirects

Redirects are common in Web Development. You can create a redirect using the Response.redirect() method:

res.redirect('/go-there')

This creates a 302 redirect.

A 301 redirect is made in this way:

res.redirect(301, '/go-there')

You can specify an absolute path (/go-there), an absolute URL (https://anothersite.com), a relative path (go-there) or use the .. to go back one level:

res.redirect('../go-there')
res.redirect('..')

You can also redirect back to the Referrer HTTP header value (defaulting to / if not set) using

res.redirect('back')

Routing in Express

Routing is the process of determining what should happen when a URL is called, or also which parts of the application should handle a specific incoming request.

In the Hello World example we used this code:

app.get('/', (req, res) => { /* */ })

This creates a route that maps accessing the root domain URL / using the HTTP GET method to the response we want to provide.