backend - freeCodeCamp.org

From Flutter to Backend: How to Build Production-Grade REST APIs with Dart and Serverpod

Oluwaseyi Fatunmole — Tue, 02 Jun 2026 16:25:50 +0000

Serverpod is one of the most performant backend frameworks built on Dart. It's a fully opinionated backend framework that comes with its own ORM, its own code generation system, migration tooling, authentication module, and deployment platform.

If you use a tool like Shelf to build your API, you assemble everything yourself. You choose your packages, write your own middleware, manage your own database connection, and wire every piece together manually. That's the Shelf way, and it teaches you exactly how server-side Dart works under the hood.

Serverpod is a different philosophy entirely.

Where Shelf gives you primitives, Serverpod gives you a complete system. You define your models in YAML, run a generator, and get fully typed database classes, serialization, and client-side code produced automatically.

For Flutter engineers, this feels immediately familiar. It's the same kind of productivity you get from the Flutter toolchain itself, applied to the backend.

In this article, we're going to build a User and Profile Management REST API from scratch using Serverpod. You'll learn how Serverpod's code generation, built-in ORM, and endpoint system work, and you'll have a fully deployed backend by the end.

Prerequisites
How Serverpod Differs from Shelf
Installing Serverpod
Creating the Project
Understanding the Project Structure
Serverpod Core Concepts
Starting the Development Server
Defining the Models
Building the API
Authentication
- Password Hashing and JWT
- Protecting Endpoints
Error Handling in Serverpod
Testing the API
Deployment
- Deploying with Docker and Fly.io
- Deploying with Serverpod Cloud
Conclusion

Prerequisites

Before starting, you should have:

Familiarity with Dart and Flutter development
Understanding of REST API concepts, endpoints, HTTP methods, status codes
Docker Desktop installed and running
Flutter SDK installed (Serverpod requires it even for server-only projects)
A Fly.io account or a Serverpod Cloud account for deployment

How Serverpod Differs from Shelf

Before writing a single line of code, it's worth understanding the fundamental difference in philosophy between Shelf and Serverpod. This will make every design decision in the framework feel deliberate rather than arbitrary.

With Shelf, you write everything. Request parsing, response formatting, database queries, migrations, auth, and logging. Every piece is explicit code that you understand because you wrote it.

With Serverpod, you define things and the framework writes code for you. You define a model in YAML, run serverpod generate, and get a full Dart class with database bindings, serialization, and client-side access automatically. You define an endpoint method, and the framework handles routing, parameter extraction, and response formatting.

This is the same trade-off Flutter makes compared to building with raw platform APIs. Flutter writes the layout engine, the rendering pipeline, and the gesture system for you. You focus on your product logic. Serverpod makes the same bet on the backend.

The cost of that productivity is flexibility. Serverpod has strong opinions about how things should be structured. If your use case fits those opinions, development is extremely fast. If it doesn't, you're working against the framework.

For the User and Profile Management API we're building here, Serverpod is a very good fit.

Installing Serverpod

Serverpod requires Flutter to be installed, even for server-only work. This is because its toolchain builds client packages alongside the server package during project creation.

Install the Serverpod CLI globally:

dart pub global activate serverpod_cli

Verify the installation:

serverpod
# Should print the Serverpod CLI help

Make sure Docker Desktop is running before proceeding. Serverpod uses Docker to manage PostgreSQL and Redis for local development.

Creating the Project

serverpod create user_profile_api
cd user_profile_api

This single command creates three Dart packages:

user_profile_api/
  user_profile_api_server/    ← your server code
  user_profile_api_client/    ← auto-generated client (do not edit)
  user_profile_api_flutter/   ← Flutter app pre-configured to connect

For this article, everything we write lives in user_profile_api_server. The client and Flutter packages are generated automatically and used when you want a Flutter frontend talking to your Serverpod backend.

Understanding the Project Structure

Inside user_profile_api_server:

user_profile_api_server/
  bin/
    main.dart                  ← entry point
  lib/
    src/
      endpoints/               ← your endpoint classes live here
      generated/               ← auto-generated code (never edit manually)
    user_profile_api_server.dart
  config/
    development.yaml           ← database and server config
    staging.yaml
    production.yaml
    passwords.yaml             ← database passwords
  migrations/                  ← auto-generated migration files
  web/                         ← optional web server files
  Dockerfile
  docker-compose.yaml
  pubspec.yaml

The most important thing to understand about this structure is the generated/ folder. Everything in there is produced by serverpod generate and should never be edited manually. When you change a model or endpoint, you run the generator and it rewrites that folder entirely.

The config/ folder holds environment-specific configuration. The development.yaml file is preconfigured to work with the Docker containers Serverpod spins up locally.

Serverpod Core Concepts

Endpoints and the Session Object

In Serverpod, an endpoint is a class that extends Endpoint. Every public method on that class becomes an API call that clients can make. There's no routing configuration, no handler registration, and no middleware mounting. The framework discovers and registers your endpoints automatically during code generation.

import 'package:serverpod/serverpod.dart';

class UserEndpoint extends Endpoint {
  Future greet(Session session, String name) async {
    return 'Hello, $name!';
  }
}

The Session object is the most important parameter in Serverpod. It's passed to every endpoint method and gives you access to:

session.db for database operations
session.auth for authentication information
session.log for structured logging
session.caches for caching
session.messages for real-time messaging

Think of Session as Serverpod's equivalent of Flutter's BuildContext. It's the gateway to everything the framework provides, and it's always the first parameter.

Model Files and Code Generation

This is where Serverpod's approach diverges most sharply from Shelf. Instead of writing Dart model classes manually, you define your data structures in .spy.yaml files and let Serverpod generate the Dart classes.

A model file for a Company looks like this:

class: Company
table: company
fields:
  name: String
  foundedDate: DateTime?

Running serverpod generate produces a full Dart class with:

Immutable fields with correct types
toJson and fromJson for serialization
Database bindings through the db static accessor
Constructor and copyWith method
The same class is generated in the client package so the Flutter app can use it directly

This is the core productivity gain. You define the shape once in YAML and get a consistent, typed model that works across the server, the database, and the client without duplication.

The Built-in ORM

Serverpod's ORM uses the generated model classes directly. All database operations go through the static db accessor on your model:

// Insert a row
var company = Company(name: 'Serverpod Corp', foundedDate: DateTime.now());
company = await Company.db.insertRow(session, company);

// Find by ID
var found = await Company.db.findById(session, company.id!);

// Find with condition
var result = await Company.db.findFirstRow(
  session,
  where: (t) => t.name.equals('Serverpod Corp'),
);

// Find all with ordering
var all = await Company.db.find(
  session,
  orderBy: (t) => t.name,
);

// Update
company = company.copyWith(name: 'New Name');
await Company.db.updateRow(session, company);

// Delete
await Company.db.deleteRow(session, company);

The where parameter uses a type-safe expression builder. The t parameter gives you typed access to the table's columns, so you get autocompletion and compile-time checks on your query conditions. No raw SQL, no string-based column names, no runtime surprises.

Migrations

When you change a model, Serverpod generates a migration automatically:

serverpod create-migration

This creates a SQL migration file in the migrations/ directory. Apply it when starting the server:

dart bin/main.dart --apply-migrations

Serverpod tracks which migrations have been applied and runs only the new ones. The migration system is fully integrated with the model system, so there's no drift between your Dart classes and your database schema.

Starting the Development Server

Before writing any code, get the development environment running.

Start the Docker containers (PostgreSQL and Redis):

cd user_profile_api_server
docker compose up --build --detach

Start the server with migrations applied:

dart bin/main.dart --apply-migrations

You should see:

SERVERPOD version: 2.x.x, mode: development
Insights listening on port 8081
Server default listening on port 8080
Webserver listening on port 8082

Three ports: Port 8080 is the main API server. Port 8081 is the Serverpod Insights tool for monitoring. Port 8082 is an optional web server. For this article, we'll work exclusively with port 8080.

Defining the Models

The User Model

Create lib/src/models/user.spy.yaml in the server package:

class: AppUser
table: app_users
fields:
  email: String
  passwordHash: String
  firstName: String
  lastName: String
  isActive: bool, default=true
indexes:
  app_users_email_idx:
    fields: email
    unique: true

A few things to note here. The class is named AppUser rather than User to avoid conflicts with Serverpod's internal User class from the auth module. The table key defines the PostgreSQL table name. The indexes block creates a unique index on the email column, enforcing uniqueness at the database level.

Serverpod automatically adds an id field of type int? to every model with a table key. You don't declare it yourself.

The Profile Model

Create lib/src/models/profile.spy.yaml:

class: Profile
table: profiles
fields:
  userId: int
  bio: String?
  avatarUrl: String?
  phone: String?
  location: String?
  website: String?
indexes:
  profiles_user_id_idx:
    fields: userId
    unique: true

userId is an int referencing the id of an AppUser. Serverpod's model system doesn't yet have a foreign key declaration syntax in the YAML, so referential integrity is handled at the application layer in the endpoint logic.

Running Code Generation

With both model files in place, run the generator:

serverpod generate

This produces Dart classes in lib/src/generated/. For AppUser, you get:

// This is auto-generated, never edit directly
class AppUser extends SerializableEntity {
  AppUser({
    this.id,
    required this.email,
    required this.passwordHash,
    required this.firstName,
    required this.lastName,
    this.isActive = true,
  });

  int? id;
  String email;
  String passwordHash;
  String firstName;
  String lastName;
  bool isActive;

  // db accessor for ORM operations
  static final db = AppUserRepository._();

  // Serialization methods
  factory AppUser.fromJson(Map jsonSerialization, ...) { ... }
  Map toJson() { ... }
}

The generated code is what your endpoints interact with. You never write this by hand.

Creating and Applying Migrations

With the models generated, create the migration:

serverpod create-migration

This creates timestamped SQL files in migrations/. Apply them:

dart bin/main.dart --apply-migrations

Your app_users and profiles tables now exist in PostgreSQL with the correct columns and indexes.

Building the API

The Auth Endpoint

Create lib/src/endpoints/auth_endpoint.dart:

import 'package:serverpod/serverpod.dart';
import 'package:bcrypt/bcrypt.dart';
import 'package:dart_jsonwebtoken/dart_jsonwebtoken.dart';
import '../generated/protocol.dart';

class AuthEndpoint extends Endpoint {
  Future> register(
    Session session,
    String email,
    String password,
    String firstName,
    String lastName,
  ) async {
    if (email.isEmpty || password.isEmpty || firstName.isEmpty || lastName.isEmpty) {
      throw Exception('All fields are required');
    }

    if (password.length < 8) {
      throw Exception('Password must be at least 8 characters');
    }

    // Check for existing user
    final existing = await AppUser.db.findFirstRow(
      session,
      where: (t) => t.email.equals(email),
    );

    if (existing != null) {
      throw Exception('An account with this email already exists');
    }

    final passwordHash = BCrypt.hashpw(password, BCrypt.gensalt());

    var user = AppUser(
      email: email,
      passwordHash: passwordHash,
      firstName: firstName,
      lastName: lastName,
    );

    user = await AppUser.db.insertRow(session, user);

    final token = _generateToken(user);

    return {
      'user': _sanitizeUser(user),
      'token': token,
    };
  }

  Future> login(
    Session session,
    String email,
    String password,
  ) async {
    if (email.isEmpty || password.isEmpty) {
      throw Exception('Email and password are required');
    }

    final user = await AppUser.db.findFirstRow(
      session,
      where: (t) => t.email.equals(email),
    );

    if (user == null || !BCrypt.checkpw(password, user.passwordHash)) {
      throw Exception('Invalid email or password');
    }

    if (!user.isActive) {
      throw Exception('This account has been deactivated');
    }

    final token = _generateToken(user);

    return {
      'user': _sanitizeUser(user),
      'token': token,
    };
  }

  String _generateToken(AppUser user) {
    final jwt = JWT({'sub': user.id, 'email': user.email});
    return jwt.sign(SecretKey(_jwtSecret), expiresIn: const Duration(hours: 24));
  }

  // Never return the password hash to the client
  Map _sanitizeUser(AppUser user) => {
        'id': user.id,
        'email': user.email,
        'firstName': user.firstName,
        'lastName': user.lastName,
        'isActive': user.isActive,
      };

  // Read from Serverpod's config system
  String get _jwtSecret =>
      Session.serverpod.getPassword('jwtSecret') ?? 'fallback_dev_secret';
}

Serverpod endpoints return typed values. When you return a Map, Serverpod serializes it automatically. When you throw an Exception, Serverpod catches it and returns a structured error response to the client. No manual response formatting, no status code management for common cases.

The User Endpoint

Create lib/src/endpoints/user_endpoint.dart:

import 'package:serverpod/serverpod.dart';
import '../generated/protocol.dart';

class UserEndpoint extends Endpoint {
  @override
  bool get requireLogin => true;

  Future>> getAll(Session session) async {
    final users = await AppUser.db.find(
      session,
      where: (t) => t.isActive.equals(true),
      orderBy: (t) => t.id,
    );

    return users.map(_sanitizeUser).toList();
  }

  Future> getById(Session session, int userId) async {
    final user = await AppUser.db.findById(session, userId);

    if (user == null || !user.isActive) {
      throw Exception('User not found');
    }

    return _sanitizeUser(user);
  }

  Future> update(
    Session session,
    int userId,
    String? firstName,
    String? lastName,
  ) async {
    final user = await AppUser.db.findById(session, userId);

    if (user == null || !user.isActive) {
      throw Exception('User not found');
    }

    final updated = user.copyWith(
      firstName: firstName ?? user.firstName,
      lastName: lastName ?? user.lastName,
    );

    await AppUser.db.updateRow(session, updated);
    return _sanitizeUser(updated);
  }

  Future delete(Session session, int userId) async {
    final user = await AppUser.db.findById(session, userId);

    if (user == null || !user.isActive) {
      throw Exception('User not found');
    }

    // Soft delete
    final deactivated = user.copyWith(isActive: false);
    await AppUser.db.updateRow(session, deactivated);
  }

  Map _sanitizeUser(AppUser user) => {
        'id': user.id,
        'email': user.email,
        'firstName': user.firstName,
        'lastName': user.lastName,
        'isActive': user.isActive,
      };
}

Notice @override bool get requireLogin => true. This is Serverpod's built-in mechanism for protecting endpoints. When this getter returns true, Serverpod validates the authentication token on every request to this endpoint before the method is called. Unauthenticated requests are rejected automatically by the framework.

The Profile Endpoint

Create lib/src/endpoints/profile_endpoint.dart:

import 'package:serverpod/serverpod.dart';
import '../generated/protocol.dart';

class ProfileEndpoint extends Endpoint {
  @override
  bool get requireLogin => true;

  Future> getByUserId(
    Session session,
    int userId,
  ) async {
    final user = await AppUser.db.findById(session, userId);
    if (user == null || !user.isActive) {
      throw Exception('User not found');
    }

    final profile = await Profile.db.findFirstRow(
      session,
      where: (t) => t.userId.equals(userId),
    );

    if (profile == null) {
      throw Exception('Profile not found');
    }

    return _profileToMap(profile);
  }

  Future> create(
    Session session,
    int userId,
    String? bio,
    String? avatarUrl,
    String? phone,
    String? location,
    String? website,
  ) async {
    final user = await AppUser.db.findById(session, userId);
    if (user == null || !user.isActive) {
      throw Exception('User not found');
    }

    final existing = await Profile.db.findFirstRow(
      session,
      where: (t) => t.userId.equals(userId),
    );

    if (existing != null) {
      throw Exception('Profile already exists for this user');
    }

    var profile = Profile(
      userId: userId,
      bio: bio,
      avatarUrl: avatarUrl,
      phone: phone,
      location: location,
      website: website,
    );

    profile = await Profile.db.insertRow(session, profile);
    return _profileToMap(profile);
  }

  Future> update(
    Session session,
    int userId,
    String? bio,
    String? avatarUrl,
    String? phone,
    String? location,
    String? website,
  ) async {
    final profile = await Profile.db.findFirstRow(
      session,
      where: (t) => t.userId.equals(userId),
    );

    if (profile == null) {
      throw Exception('Profile not found');
    }

    final updated = profile.copyWith(
      bio: bio ?? profile.bio,
      avatarUrl: avatarUrl ?? profile.avatarUrl,
      phone: phone ?? profile.phone,
      location: location ?? profile.location,
      website: website ?? profile.website,
    );

    await Profile.db.updateRow(session, updated);
    return _profileToMap(updated);
  }

  Map _profileToMap(Profile profile) => {
        'id': profile.id,
        'userId': profile.userId,
        'bio': profile.bio,
        'avatarUrl': profile.avatarUrl,
        'phone': profile.phone,
        'location': profile.location,
        'website': profile.website,
      };
}

After adding these endpoints, run the generator again so Serverpod registers them:

serverpod generate

Authentication

Password Hashing and JWT

Add the required packages to pubspec.yaml in the server package:

dependencies:
  serverpod: ^2.5.0
  bcrypt: ^1.1.3
  dart_jsonwebtoken: ^2.12.0

Then run dart pub get.

The _generateToken and _sanitizeUser helpers in the auth endpoint handle password hashing and JWT generation. The JWT secret is read from Serverpod's built-in password management system via Session.serverpod.getPassword('jwtSecret').

Add the secret to config/passwords.yaml:

development:
  database: 'dart_password'
  jwtSecret: 'your_development_jwt_secret_here'

This file is already in .gitignore by default in a Serverpod project. Production secrets are injected via environment variables or Serverpod Cloud's secret management.

Protecting Endpoints

Serverpod has two levels of endpoint protection:

requireLogin — rejects unauthenticated requests automatically:

@override
bool get requireLogin => true;

requiredScopes — requires specific permission scopes:

@override
Set get requiredScopes => {Scope.admin};

For the User and Profile endpoints in this article, requireLogin is sufficient. The token from the login response is passed in the Authorization header on every subsequent request, and Serverpod validates it before the endpoint method is called.

Verifying the token inside an endpoint to get the current user's ID:

Future someProtectedMethod(Session session) async {
  final authInfo = await session.authenticated;

  if (authInfo == null) {
    throw Exception('Not authenticated');
  }

  final userId = authInfo.userId;
  // proceed with userId
}

Error Handling in Serverpod

Serverpod handles exceptions thrown from endpoint methods and converts them into structured error responses automatically. When you throw:

throw Exception('User not found');

The client receives a structured error response. For more granular control, Serverpod provides typed exceptions:

throw ServerpodClientException('User not found', statusCode: 404);

For server-side logging without exposing details to the client:

session.log('Unexpected error during user creation', level: LogLevel.error);
throw Exception('An internal error occurred');

Serverpod's logging system stores logs in the database and makes them queryable through the Insights dashboard on port 8081. Every request is automatically logged with timing information, endpoint name, and outcome, no additional middleware required.

Testing the API

Serverpod exposes its endpoints over HTTP. You can test them directly with curl, though the request format follows Serverpod's RPC convention rather than a traditional REST structure.

The generated URL pattern for an endpoint method is:

POST /[endpoint]/[method]

With a JSON body containing the method parameters.

curl http://localhost:8080/auth/register \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "email": "seyi@example.com",
    "password": "securepassword",
    "firstName": "Seyi",
    "lastName": "Dev"
  }'

curl http://localhost:8080/auth/login \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"email": "seyi@example.com", "password": "securepassword"}'

Get all users (authenticated):

curl http://localhost:8080/user/getAll \
  -X POST \
  -H "Authorization: Bearer eyJhbGci..."

Get user by ID:

curl http://localhost:8080/user/getById \
  -X POST \
  -H "Authorization: Bearer eyJhbGci..." \
  -H "Content-Type: application/json" \
  -d '{"userId": 1}'

Create a profile:

curl http://localhost:8080/profile/create \
  -X POST \
  -H "Authorization: Bearer eyJhbGci..." \
  -H "Content-Type: application/json" \
  -d '{
    "userId": 1,
    "bio": "Flutter engineer turned backend developer",
    "location": "Lagos, Nigeria",
    "website": "https://example.com"
  }'

Update a user:

curl http://localhost:8080/user/update \
  -X POST \
  -H "Authorization: Bearer eyJhbGci..." \
  -H "Content-Type: application/json" \
  -d '{"userId": 1, "firstName": "Oluwaseyi"}'

Delete a user:

curl http://localhost:8080/user/delete \
  -X POST \
  -H "Authorization: Bearer eyJhbGci..." \
  -H "Content-Type: application/json" \
  -d '{"userId": 1}'

Deployment

Deploying with Docker and Fly.io

Serverpod generates a Dockerfile as part of the project creation. It's located in user_profile_api_server/Dockerfile and is ready to use.

The included docker-compose.yaml in the server package manages PostgreSQL and Redis for local development. For production deployment on Fly.io, the process follows the same Docker-based pattern covered in the deployment section below.

Step 1 — Authenticate with Fly:

fly auth login

Step 2 — Launch the app from the server directory:

cd user_profile_api_server
fly launch

Step 3 — Set production secrets:

fly secrets set JWT_SECRET="your_production_jwt_secret"

Step 4 — Update the production config:

Edit config/production.yaml with your Fly-provisioned database connection details. Fly injects the DATABASE_URL environment variable which you map to the Serverpod config format.

Step 5 — Deploy:

fly deploy

Step 6 — Apply migrations on first deploy:

fly ssh console
dart bin/main.dart --apply-migrations --mode production

Deploying with Serverpod Cloud

Serverpod Cloud is the native deployment platform built specifically for Serverpod applications. It handles database provisioning, scaling, monitoring, and deployments with minimal configuration.

Install the Serverpod Cloud CLI:

dart pub global activate serverpod_cloud_cli

Authenticate:

scloud login

Create a project in the Serverpod Cloud dashboard at cloud.serverpod.dev, then link your local project:

scloud link --project-id your-project-id

Deploy:

scloud deploy

Serverpod Cloud provisions a managed PostgreSQL database, applies your migrations, and deploys your server automatically. It also provides the Insights dashboard for monitoring requests, logs, and performance in production.

For teams already committed to the Serverpod ecosystem, Serverpod Cloud is the fastest path to a production deployment.

Conclusion

Serverpod takes a fundamentally different approach from Shelf. Where Shelf gives you control, Serverpod gives you speed. You define your models in YAML, run a generator, and get database classes, serialization, and client code produced automatically. You write an endpoint method and the framework handles routing, parameter extraction, authentication, and error formatting.

The ORM is the strongest part of the experience. Type-safe query expressions, automatic migration generation, and no SQL drift between your code and your schema make database work noticeably faster and safer than raw SQL.

The cost is rigidity. Serverpod's URL structure, serialization format, and architectural conventions aren't optional. If your API needs to conform to a specific REST structure that differs from Serverpod's RPC style, you'll be working against the framework.

For greenfield Flutter backends where the Dart client will consume the API, Serverpod is hard to beat. The code sharing between server and client, the automatic client generation, and the tight toolchain integration make it the most productive Dart backend option available.

For APIs that need to serve multiple clients, conform to external REST conventions, or integrate with existing infrastructure that doesn't expect Serverpod's format, a lower-level tool like Shelf gives you more control. If you want to see how the same User and Profile Management API is built with Shelf and compare the two approaches directly, you can find that article here.

Knowing which tool fits which job is what separates a developer who knows a framework from one who understands backend development.

Happy coding!

How to Use PostgreSQL as a Cache, Queue, and Search Engine

Aaron Yong — Tue, 21 Apr 2026 16:58:55 +0000

"Just use Postgres" has been circulating as advice for years, but most articles arguing for it are opinion pieces. I wanted hard numbers.

So I built a benchmark suite that pits vanilla PostgreSQL against a feature-optimized PostgreSQL instance — measuring caching, message queues, full-text search, and pub/sub under controlled conditions.

In this article, you'll learn how to use PostgreSQL's built-in features for caching, job queues, full-text search, and pub/sub. You'll see actual benchmark results (latency percentiles, throughput, and error rates) comparing naive PostgreSQL patterns against optimized ones, and understand where PostgreSQL's limits are so you can decide whether you really need that extra service in your stack.

Prerequisites
The Setup
Benchmark 1: Caching with UNLOGGED Tables
Benchmark 2: Job Queues with SKIP LOCKED
Benchmark 3: Full-Text Search with tsvector
Benchmark 4: Pub/Sub with LISTEN/NOTIFY
The Combined Workload: The Honest Test
What I Learned

Prerequisites

To follow along or reproduce the benchmarks, you'll need:

Docker and Docker Compose
Node.js 20+ (for the Express TypeScript API layer)
k6 for load testing
Basic familiarity with SQL and PostgreSQL

The full benchmark project is open source on GitHub — you can clone it and run every test yourself.

The Setup

The benchmark uses two identical PostgreSQL 17 instances running in Docker containers, each with fixed resource constraints (2 CPUs, 2 GB RAM). Both share the same Express TypeScript API layer — the only difference is which PostgreSQL features are enabled.

┌─────────┐     ┌──────────────────┐     ┌─────────────────┐
│   k6    │────>│  Express API     │────>│  PG Baseline    │
│  (load  │     │  (TypeScript)    │     │  (vanilla PG17) │
│  test)  │────>│  Port 3001/3002  │────>│  PG Modded      │
└─────────┘     └──────────────────┘     │  (features on)  │
                                         └─────────────────┘

The baseline instance uses naïve approaches (regular tables, ILIKE search, polling). The modded instance uses PostgreSQL's built-in features (UNLOGGED tables, tsvector with GIN indexes, LISTEN/NOTIFY, partial indexes). Same hardware, same API code, same data. Only the database features differ.

Both instances share this tuned postgresql.conf:

# Memory allocation
shared_buffers = 512MB           # 25% of available RAM
effective_cache_size = 1536MB    # 75% of RAM — helps the query planner
work_mem = 16MB                  # per-sort/hash operation memory

# SSD-optimized planner settings
random_page_cost = 1.1           # default 4.0 assumes spinning disks
effective_io_concurrency = 200   # allow parallel I/O on SSDs

These settings matter. The defaults assume spinning disks from the early 2000s. Setting random_page_cost = 1.1 tells the query planner that random reads are nearly as fast as sequential reads on SSDs, which encourages index usage over sequential scans.

Benchmark 1: Caching with UNLOGGED Tables

The idea: Use an UNLOGGED table as an in-database cache. UNLOGGED tables skip PostgreSQL's Write-Ahead Log (WAL) — the mechanism that guarantees durability. Since cache data is ephemeral by nature, losing it on a crash is acceptable, and skipping WAL removes the biggest write bottleneck.

-- Modded: UNLOGGED table for cache entries
CREATE UNLOGGED TABLE cache_entries (
    key TEXT PRIMARY KEY,
    value JSONB NOT NULL,
    expires_at TIMESTAMPTZ
);

-- Baseline: same schema, but a regular (logged) table
CREATE TABLE cache_entries (
    key TEXT PRIMARY KEY,
    value JSONB NOT NULL,
    expires_at TIMESTAMPTZ
);

Results (200 Virtual Users)

Mode	p50	p95	avg	req/s
Baseline (regular table)	1.87ms	6.00ms	2.50ms	1,754/s
Modded (UNLOGGED table)	1.71ms	5.24ms	2.17ms	1,760/s

A consistent 13% improvement across all percentiles. Not dramatic, but free — you change one keyword in your CREATE TABLE statement.

Under Stress (1,000 Virtual Users, No Sleep)

Mode	p50	p95	req/s	Total Requests
Baseline	83.38ms	143.23ms	7,663/s	728,021
Modded	77.69ms	126.39ms	8,062/s	765,934

The relative improvement stays locked at 12-13% regardless of load level. The UNLOGGED advantage is a per-write optimization — it saves the same amount of I/O whether you are doing 100 or 10,000 writes per second. The modded instance served 37,000 more requests in the same time window.

The Verdict

UNLOGGED tables won't match Redis for sub-millisecond hot-path caching (real-time bidding, gaming leaderboards). But for web applications where the difference between 2ms and 5ms is invisible to users, they eliminate an entire infrastructure dependency for zero additional complexity.

You do give up Redis data structures (sorted sets, HyperLogLog, streams). If you need those, a dedicated cache is still the right call.

Benchmark 2: Job Queues with SKIP LOCKED

The idea: Use PostgreSQL as a job queue with SELECT ... FOR UPDATE SKIP LOCKED. Multiple workers poll the same table, and SKIP LOCKED ensures each worker gets a different row — no duplicates, no contention.

-- Queue table with a partial index on pending jobs only
CREATE TABLE job_queue (
    id SERIAL PRIMARY KEY,
    payload JSONB NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Partial index: only indexes pending jobs
-- As jobs complete, they leave the index — it stays small forever
CREATE INDEX idx_pending_jobs ON job_queue (created_at)
    WHERE status = 'pending';

The dequeue pattern:

-- Atomic dequeue: select + update in one statement
UPDATE job_queue SET status = 'processing'
WHERE id = (
    SELECT id FROM job_queue
    WHERE status = 'pending'
    ORDER BY created_at
    LIMIT 1
    FOR UPDATE SKIP LOCKED  -- skip rows locked by other workers
) RETURNING *;

How SKIP LOCKED works: Worker A locks row 1. Worker B tries row 1, sees the lock, skips it, and takes row 2 instead. No blocking, no duplicates. If a worker crashes, the transaction rolls back and the row becomes available again.

Results (100 Producers + 50 Consumers)

Mode	p50	p95	avg	req/s
Baseline (full index)	1.90ms	5.01ms	2.30ms	1,053/s
Modded (partial index)	1.81ms	5.28ms	2.29ms	1,052/s

They're virtually identical. The partial index doesn't show its value in a 60-second benchmark because the table doesn't accumulate enough completed rows for the index size difference to matter. In a production system with millions of completed jobs, the partial index keeps the index at kilobytes while a full index grows to gigabytes.

The Verdict

SKIP LOCKED is production-ready for job queues. Libraries like pg-boss (Node.js) and river (Go) build on this exact pattern.

You do give up exchange/routing patterns (fan-out, topic-based routing) and consumer groups with message replay. If you need those, a dedicated message broker is still the right tool. For simple "process this job once" workloads, PostgreSQL handles it.

Benchmark 3: Full-Text Search with tsvector

The idea: Use PostgreSQL's built-in full-text search instead of a separate search service. A tsvector column stores pre-processed search tokens, and a GIN (Generalized Inverted Index) enables fast lookups using the same inverted index concept that powers Elasticsearch.

-- Search-optimized article table
CREATE TABLE articles (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    body TEXT NOT NULL,
    search_vector tsvector  -- pre-computed search tokens
);

-- GIN index for full-text search
CREATE INDEX idx_search ON articles USING GIN (search_vector);

-- Auto-update search_vector on insert/update
CREATE OR REPLACE FUNCTION update_search_vector() RETURNS trigger AS $$
BEGIN
    NEW.search_vector := to_tsvector('english',
        COALESCE(NEW.title, '') || ' ' || COALESCE(NEW.body, ''));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_search
    BEFORE INSERT OR UPDATE ON articles
    FOR EACH ROW EXECUTE FUNCTION update_search_vector();

The baseline uses ILIKE with a leading wildcard — the approach most developers reach for first:

-- Baseline: sequential scan on every query
SELECT * FROM articles
WHERE title ILIKE '%postgresql%' OR body ILIKE '%postgresql%';

-- Modded: GIN index lookup with relevance ranking
SELECT id, title,
    ts_rank(search_vector, plainto_tsquery('english', 'postgresql')) AS rank
FROM articles
WHERE search_vector @@ plainto_tsquery('english', 'postgresql')
ORDER BY rank DESC LIMIT 20;

Results (500 Virtual Users)

Mode	p50	p95	avg	req/s
Baseline (ILIKE)	1.96ms	101.83ms	25.22ms	561/s
Modded (tsvector + GIN)	2.76ms	10.39ms	3.76ms	675/s

This is the standout result. The baseline's p95 of 101ms versus the modded's 10ms is a 10x improvement.

Why the baseline's p50 (1.96ms) is slightly better than the modded's (2.76ms): simple ILIKE queries on small result sets can be fast when the data fits in shared_buffers. But as load increases and the buffer cache is contested, sequential scans degrade dramatically. The GIN index stays stable.

Under Stress (500 Virtual Users, No Sleep)

Mode	p50	p95	req/s	Total Requests
Baseline (ILIKE)	599ms	1,000ms	558/s	50,212
Modded (tsvector)	209ms	396ms	1,441/s	129,679

ILIKE collapses to 1-second p95 latencies. Each query forces a sequential scan of all 10,000 articles, blocking shared buffers and starving concurrent queries. The tsvector approach serves 2.6x more requests in the same time window because the GIN index lookup is O(log n) regardless of concurrency.

The Verdict

This is the strongest argument in the entire benchmark. The fix requires zero extensions — to_tsvector(), plainto_tsquery(), and CREATE INDEX USING GIN are all built into core PostgreSQL. If you're doing WHERE column ILIKE '%term%' on any table with more than a few thousand rows, you're leaving massive performance on the table.

You do give up distributed search across shards, complex analyzers for CJK languages, and aggregation/faceted search pipelines. For a product search bar, blog search, or internal tool — PostgreSQL is enough.

Benchmark 4: Pub/Sub with LISTEN/NOTIFY

The idea: Use PostgreSQL's native LISTEN/NOTIFY for pub/sub messaging, triggered automatically on INSERT via a database trigger.

-- Trigger that fires pg_notify on every new message
CREATE OR REPLACE FUNCTION notify_message() RETURNS trigger AS $$
BEGIN
    PERFORM pg_notify(NEW.channel, NEW.payload::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER trg_notify
    AFTER INSERT ON messages
    FOR EACH ROW EXECUTE FUNCTION notify_message();

Results (200 Virtual Users)

Mode	p50	p95	avg	req/s
Baseline (poll-based)	1.99ms	6.04ms	2.84ms	1,116/s
Modded (LISTEN/NOTIFY)	1.65ms	4.80ms	2.13ms	1,131/s

Here we have a 20% improvement at p95. The trigger-based approach does more work per INSERT (INSERT + NOTIFY), but the reduced round trips and better connection reuse patterns offset the overhead.

The Verdict

LISTEN/NOTIFY works for real-time features where you would otherwise reach for Redis pub/sub. The main limitation is payload size (8,000 bytes maximum) and the requirement for dedicated connections (incompatible with PgBouncer in transaction mode).

The Combined Workload: The Honest Test

Individual benchmarks are flattering. The real question: can one PostgreSQL instance handle caching, queues, search, and pub/sub simultaneously without degrading?

Results (All Four Workloads Running Together)

Mode	p50	p95	avg	req/s
Baseline	1.65ms	5.24ms	2.17ms	1,424/s
Modded	1.86ms	6.05ms	2.47ms	1,417/s

Under combined load, the baseline marginally outperforms the modded setup. The modded PostgreSQL does more work per operation — maintaining GIN indexes, firing triggers, running pg_cron in the background. When all these features are active simultaneously, the overhead is measurable: about 15% higher p95 latency.

But both setups stay comfortably under 10ms at p95. For most web applications, that's more than good enough.

What I Learned

After running all these benchmarks, here's what I would tell a team evaluating whether to "just use Postgres":

Do it for full-text search: Switching from ILIKE to tsvector with a GIN index is a 10x improvement that requires zero extensions. This is the single highest-ROI change in the entire PostgreSQL ecosystem, and most developers don't know it exists.
Do it for job queues: SKIP LOCKED is production-ready and eliminates RabbitMQ for simple "process this job" workloads. Use a library like pg-boss or river rather than rolling your own.
Consider it for caching: UNLOGGED tables give a steady 13% improvement over regular tables. If sub-millisecond latency is not a hard requirement (and for most web apps, it is not), you can drop Redis entirely.
Be honest about the overhead: Running all four roles simultaneously adds about 15% latency compared to running any single role. Whether that matters depends on your latency budget.
Know where to stop: PostgreSQL won't match Redis for sub-millisecond caching, Kafka for millions of messages per second, or Elasticsearch for distributed multi-node search with complex analyzers. The line is at extreme throughput or extreme specialization.

The honest conclusion is not "PostgreSQL does everything." It is: for most applications, a single well-configured PostgreSQL instance handles 80% of what you would otherwise need three to five additional services for. That is less infrastructure to deploy, monitor, and maintain — and fewer things to break at 3 AM.

Enterprise-scale applications processing millions of messages per second, serving sub-millisecond cache hits to millions of concurrent users, or running distributed search across terabytes of documents will still need specialized tools. Those tools exist for a reason, and at that scale the operational cost of running them is justified by the performance you get back.

But most of us aren't building at that scale — and may never need to. Starting with PostgreSQL for these roles means you ship faster with fewer moving parts. If and when you outgrow what PostgreSQL can handle, your benchmarks will tell you exactly which role needs to be extracted into a dedicated service. That is a much better position than starting with five services on day one because you assumed you would need them.

The benchmark project is open source if you want to reproduce these results or adapt the tests for your own workload.

You can find more of my writing at site.aaronhsyong.com.

How to Build Real-Time Update Systems with MQTT and Express.js

David Aniebo — Tue, 10 Mar 2026 16:05:25 +0000

Real-time updates are everywhere – like live sports scores, stock tickers, chat applications, and IoT dashboards. If you want to build systems that push data to users the moment it changes, you need the right tools.

Message Queuing Telemetry Transport (MQTT) is a lightweight messaging protocol that excels at this. Combined with a broker like Mosquitto and a web framework like Express, you can build a production-ready real-time system in a single afternoon.

In this tutorial, you'll build a complete real-time football (soccer) sports update system from scratch. You'll create an admin interface for uploading scores and match details, a viewer interface for watching live updates, and a backend that uses MQTT to broadcast changes instantly to every connected client.

By the end of this guide, you'll understand how to integrate MQTT with Express, set up the Mosquitto broker, and deliver real-time data to web browsers using Server-Sent Events. You will have a working system that you can extend for production use.

What You Will Learn
Prerequisites
Understanding the Architecture
What is MQTT and Why Use It?
- MQTT Topic Design
- Why Server-Sent Events Instead of WebSockets?
Project Setup
How to Set Up the MQTT Broker
How to Build the Express Server
How to Implement the Match Routes
How to Bridge MQTT to the Browser with Server-Sent Events
How to Build the Admin Upload Interface
- Admin HTML Structure and Styles
- Admin JavaScript Logic
How to Build the Live Viewer Interface
- Viewer JavaScript Logic
How to Build the Home Page
How to Run and Test the System
How to Extend the System for Production
API Reference
Troubleshooting
Conclusion

What You Will Learn

During this tutorial, you'll learn how to:

Connect an Express server to an MQTT broker using the MQTT.js library
Publish and subscribe to MQTT topics for real-time messaging
Use Server-Sent Events to push MQTT messages to web browsers
Build a REST application programming interface (API) for match and score management
Create a simple admin interface for uploading match data
Create a viewer interface that updates in real time without page refreshes

Prerequisites

Before you start, you should have:

Node.js version 18 or later installed on your machine
Basic familiarity with JavaScript, Express, and HTML
A terminal or command line for running commands
Docker installed (optional, for running Mosquitto in a container)

If you don't have Node.js installed, you can download it from the official Node.js website.

Understanding the Architecture

The system has three main parts:

Admin interface – A web page where you create matches, update scores, and add events such as goals and cards.
Express server – Receives HyperText Transfer Protocol (HTTP) requests from the admin, publishes data to MQTT, subscribes to MQTT topics, and streams updates to viewers via Server-Sent Events.
Viewer interface – A web page that connects to the server and displays live scores and events as they arrive.

How the flow works
When you submit a score update in the admin panel, the Express server publishes a message to the MQTT broker. The server also subscribes to those same topics. When a message arrives, the server forwards it to all connected viewers through Server-Sent Events. The viewers update their display without refreshing the page.

What is MQTT and Why Use It?

MQTT stands for Message Queuing Telemetry Transport. It's a lightweight, publish-subscribe messaging protocol designed for low bandwidth and unreliable networks. It's widely used in Internet of Things (IoT) applications, but it works well for any real-time system where you need to broadcast updates to many subscribers.

Here are some reasons to use MQTT for a sports update system:

Low overhead: Messages are small and efficient, which helps when you have many clients.
Built-in Quality of Service (QoS): You can choose how many times a message is delivered (at most once, at least once, or exactly once).
Topic-based routing: You organize messages by topic (for example, sports/football/match/123) so subscribers receive only what they need.
Broker-based: A central broker (Mosquitto) handles all message distribution, so your application logic stays simple.

Mosquitto is a popular, open-source MQTT broker that's easy to install and configure.

MQTT Topic Design

MQTT uses a hierarchical topic structure. For this project, the topics are:

sports/football/match/{id}: One topic per match. When you publish the full match object here, any subscriber receives the complete state. This makes it easy to add new match fields later without changing the topic structure.
sports/football/scores: A single topic for score-related notifications. Messages include a type field (match_created or score_update) so subscribers can handle them differently.
sports/football/events: A topic for match events such as goals and cards. Subscribers receive { type: 'match_event', matchId, event }.

The # wildcard in a subscription means "match this level and all levels below." So sports/football/# subscribes to every topic under sports/football, including sports/football/match/abc123 and sports/football/scores. The + wildcard matches exactly one level. For example, sports/+/match/# would match any sport, not just football.

Why Server-Sent Events Instead of WebSockets?

You might wonder why this tutorial uses Server-Sent Events (SSE) instead of WebSockets. Both can push data to the browser. The main difference:

Server-Sent Events: One-way (server to client). Built on HTTP. Automatic reconnection in the browser. Simpler to implement. No extra libraries.
WebSockets: Two-way. Requires a different protocol. More flexible but more complex.

For a sports score viewer, you only need server-to-client updates. The viewer never sends messages back over the same channel. Server-Sent Events is a better fit. If you later need the client to send commands (for example, to filter by league), you can add a separate HTTP API or switch to WebSockets.

Project Setup

Start by creating a new folder for your project and initialize it with npm. The mkdir command creates the directory, cd moves into it, and npm init -y creates a package.json file with default values without prompting you for input.

mkdir mqtt-football-scores
cd mqtt-football-scores
npm init -y

Install the required dependencies. Each package serves a specific role in the application. Run this command in the project root directory.

npm install express cors mqtt uuid

express: Web framework for the HTTP server and API. It provides routing, middleware, and static file serving.
cors: Enables Cross-Origin Resource Sharing so your frontend can call the API from a different origin (for example, if you serve the HTML from a different port or domain).
mqtt: MQTT client for Node.js. It handles connection, publish, subscribe, reconnection, and Quality of Service (QoS) flows.
uuid: Generates unique identifiers (Universally Unique Identifiers) for matches and events. Each ID is practically unique across all systems.

Next, create the following folder structure. The server folder holds the Node.js backend code. The public folder holds the HTML, CSS, and client-side JavaScript that the browser loads. The routes subfolder keeps the match-related route handlers separate from the main server file.

mqtt-football-scores/
├── server/
│   ├── index.js
│   ├── sse.js
│   └── routes/
│       └── matches.js
├── public/
│   ├── index.html
│   ├── admin.html
│   └── viewer.html
├── mosquitto.conf
├── docker-compose.yml
└── package.json

Add "type": "module" to your package.json so you can use JavaScript modules (import and export). The type field tells Node.js to treat .js files as ES modules, which allows you to use import and export syntax instead of CommonJS require and module.exports.

{
  "name": "mqtt-football-scores",
  "version": "1.0.0",
  "type": "module",
  "main": "server/index.js"
}

How to Set Up the MQTT Broker

You need an MQTT broker running before the server can connect. You have three options.

Option 1: Docker (Recommended)

Create a docker-compose.yml file. This file defines a single service named mosquitto that runs the Eclipse Mosquitto 2 image. The ports directive maps port 1883 on your host to port 1883 in the container so your Express server can connect. The volumes directive mounts your local mosquitto.conf into the container so the broker uses your configuration. The restart: unless-stopped option ensures the container restarts automatically if it crashes or if you reboot your machine.

version: "3.8"

services:
  mosquitto:
    image: eclipse-mosquitto:2
    container_name: mqtt-football-mosquitto
    ports:
      - "1883:1883"
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
    restart: unless-stopped

Create a mosquitto.conf file. The listener 1883 directive tells Mosquitto to listen on port 1883 for MQTT connections. The protocol mqtt specifies the standard MQTT protocol (as opposed to WebSocket). The allow_anonymous true setting permits connections without a username and password, which is fine for local development but should be disabled in production. The log_dest stdout and log_type all directives send all log output to the console so you can debug connection issues.

listener 1883
protocol mqtt
allow_anonymous true
log_dest stdout
log_type all

Start the broker:

docker-compose up -d

Option 2: Local Install

On macOS with Homebrew:

brew install mosquitto
mosquitto -c mosquitto.conf

On Ubuntu or Debian:

sudo apt install mosquitto mosquitto-clients
sudo systemctl start mosquitto

Option 3: Public Test Broker

You can use the public test broker at test.mosquitto.org without installing anything. Set the environment variable when you start the server:

MQTT_BROKER=mqtt://test.mosquitto.org npm start

Note: The public broker is shared and not suitable for production. Use it only for development and testing.

How to Build the Express Server

In this step, you'll create the main server that powers the real-time application. The Express server handles HTTP requests, serves static files like HTML and JavaScript, and acts as the bridge between the browser and the MQTT broker. It also provides endpoints that allow data to be sent and received in real time. Essentially, this server is the backbone of the application, enabling communication between all components.

Begin by creating the main server file at server/index.js:

import express from 'express';
import cors from 'cors';
import mqtt from 'mqtt';
import { fileURLToPath } from 'url';
import { dirname, join } from 'path';
import { v4 as uuidv4 } from 'uuid';

import { matchRoutes } from './routes/matches.js';
import { setupSSE, addSSEClient } from './sse.js';

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

const MQTT_BROKER = process.env.MQTT_BROKER || 'mqtt://localhost:1883';
const PORT = process.env.PORT || 3000;

const app = express();
app.use(cors());
app.use(express.json());
app.use(express.static(join(__dirname, '../public')));

let mqttClient = null;

function connectMQTT() {
  mqttClient = mqtt.connect(MQTT_BROKER, {
    clientId: `football-scores-${uuidv4().slice(0, 8)}`,
    reconnectPeriod: 3000,
    connectTimeout: 10000,
  });

  mqttClient.on('connect', () => {
    console.log('Connected to MQTT broker at', MQTT_BROKER);
    mqttClient.subscribe('sports/football/#', { qos: 1 }, (err) => {
      if (err) console.error('Subscribe error:', err);
    });
  });

  mqttClient.on('error', (err) => {
    console.error('MQTT error:', err.message);
  });

  mqttClient.on('close', () => {
    console.log('MQTT connection closed');
  });

  mqttClient.on('reconnect', () => {
    console.log('MQTT reconnecting...');
  });

  return mqttClient;
}

const mqttClientInstance = connectMQTT();
const { publishMatch, publishScoreUpdate, publishEvent, getMatches } = matchRoutes(mqttClientInstance);
setupSSE(mqttClientInstance);

app.get('/api/events', (req, res) => addSSEClient(res));
app.post('/api/matches', publishMatch);
app.patch('/api/matches/:id/score', publishScoreUpdate);
app.post('/api/matches/:id/events', publishEvent);
app.get('/api/matches', getMatches);

app.listen(PORT, () => {
  console.log(`Football Scores Server running at http://localhost:${PORT}`);
  console.log(`Admin (upload):  http://localhost:${PORT}/admin.html`);
  console.log(`Viewer:          http://localhost:${PORT}/viewer.html`);
});

Here is what each part does:

Imports: The fileURLToPath and dirname utilities replicate the __dirname variable that CommonJS provides, since ES modules don't have it. You need __dirname to build the path to the public folder.
Environment variables: MQTT_BROKER defaults to mqtt://localhost:1883 so the server connects to a local Mosquitto instance. You can override it for Docker or a remote broker. PORT defaults to 3000.
Middleware: cors() allows requests from any origin, which is useful during development. express.json() parses JSON request bodies. express.static() serves files from public so /admin.html and /viewer.html are available.
connectMQTT: Creates an MQTT client with a unique client ID (required by the broker), a 3-second reconnect interval, and a 10-second connection timeout. On connect, it subscribes to sports/football/# with QoS 1. The # wildcard means "all topics under sports/football."
matchRoutes: Returns the route handlers that create matches, update scores, and add events. Each handler publishes to MQTT and responds with JSON.
setupSSE: Registers a listener on the MQTT client's message event. When a message arrives, it forwards the payload to all connected Server-Sent Events clients.
addSSEClient: Called when a viewer opens /api/events. It sets the response headers for Server-Sent Events, flushes the headers so the connection stays open, and adds the response object to a set of active clients.
Routes: The GET /api/events route establishes the Server-Sent Events stream. The POST, PATCH, and GET routes for matches delegate to the handlers from matchRoutes.

The server serves static files from the public folder, so your HTML pages are available at the root.

How to Implement the Match Routes

In this step, you'll create the route handlers that manage matches, scores, and events. These routes allow the server to receive requests from the admin interface, update match data, and publish real-time updates to MQTT. They also store match information in memory so it can be retrieved and updated during the session.

Begin by creating the file at server/routes/matches.js:

import { v4 as uuidv4 } from 'uuid';

const TOPIC_MATCH = 'sports/football/match';
const TOPIC_SCORES = 'sports/football/scores';
const TOPIC_EVENTS = 'sports/football/events';

const matches = new Map();

function publish(client, topic, payload, qos = 1) {
  if (!client?.connected) {
    console.warn('MQTT not connected, message not published');
    return false;
  }
  client.publish(topic, JSON.stringify(payload), { qos, retain: false });
  return true;
}

export function matchRoutes(mqttClient) {
  return {
    publishMatch: (req, res) => {
      const { homeTeam, awayTeam, league, venue, kickoff } = req.body;
      if (!homeTeam || !awayTeam) {
        return res.status(400).json({ error: 'homeTeam and awayTeam are required' });
      }

      const match = {
        id: uuidv4(),
        homeTeam,
        awayTeam,
        homeScore: 0,
        awayScore: 0,
        league: league || 'Premier League',
        venue: venue || 'TBD',
        kickoff: kickoff || new Date().toISOString(),
        status: 'scheduled',
        minute: 0,
        events: [],
        createdAt: new Date().toISOString(),
      };

      matches.set(match.id, match);

      const topic = `\({TOPIC_MATCH}/\){match.id}`;
      publish(mqttClient, topic, match);
      publish(mqttClient, TOPIC_SCORES, { type: 'match_created', match });

      res.status(201).json(match);
    },

    publishScoreUpdate: (req, res) => {
      const { id } = req.params;
      const { homeScore, awayScore, minute, status } = req.body;

      const match = matches.get(id);
      if (!match) {
        return res.status(404).json({ error: 'Match not found' });
      }

      if (homeScore !== undefined) match.homeScore = homeScore;
      if (awayScore !== undefined) match.awayScore = awayScore;
      if (minute !== undefined) match.minute = minute;
      if (status !== undefined) match.status = status;

      const topic = `\({TOPIC_MATCH}/\){id}`;
      publish(mqttClient, topic, match);
      publish(mqttClient, TOPIC_SCORES, {
        type: 'score_update',
        matchId: id,
        homeScore: match.homeScore,
        awayScore: match.awayScore,
        minute: match.minute,
        status: match.status,
      });

      res.json(match);
    },

    publishEvent: (req, res) => {
      const { id } = req.params;
      const { type, team, player, minute, description } = req.body;

      const match = matches.get(id);
      if (!match) {
        return res.status(404).json({ error: 'Match not found' });
      }

      const event = {
        id: uuidv4().slice(0, 8),
        type: type || 'goal',
        team,
        player: player || 'Unknown',
        minute: minute ?? match.minute,
        description: description || `\({type}: \){player}`,
        timestamp: new Date().toISOString(),
      };

      match.events.push(event);
      if (type === 'goal') {
        if (team === match.homeTeam) match.homeScore++;
        else if (team === match.awayTeam) match.awayScore++;
      }

      const topic = `\({TOPIC_MATCH}/\){id}`;
      publish(mqttClient, topic, match);
      publish(mqttClient, TOPIC_EVENTS, { type: 'match_event', matchId: id, event });

      res.status(201).json({ match, event });
    },

    getMatches: (req, res) => {
      const list = Array.from(matches.values()).sort(
        (a, b) => new Date(b.createdAt) - new Date(a.createdAt)
      );
      res.json(list);
    },
  };
}

The routes use an in-memory Map to store matches. In production, you would replace this with a database such as PostgreSQL or MongoDB.

Explanation of the key logic:

publish: A helper that checks if the MQTT client is connected before publishing. If the broker is down, it logs a warning instead of throwing. The retain: false option means the broker doesn't store the last message for new subscribers.
publishMatch: Validates that homeTeam and awayTeam are present. Creates a match object with default values for league, venue, kickoff, status, and events. Stores it in the Map, publishes to the match-specific topic and the scores topic, and returns the match with status 201 (Created).
publishScoreUpdate: Looks up the match by ID. If not found, returns 404. Updates only the fields that are provided (using !== undefined so you can set a score to 0). Publishes the full match and a score_update notification.
publishEvent: Creates an event object with a short unique ID. Pushes it to the match's events array. If the event type is goal, increments the home or away score based on the team. Publishes the updated match and an event notification.
getMatches: Converts the Map to an array, sorts by createdAt descending (newest first), and returns the list as JSON.

MQTT topics used:

sports/football/match/{id}: Full match state. Used when a match is created or updated.
sports/football/scores: Score change notifications. Used for match creation and score updates.
sports/football/events: Match events such as goals and cards.

The publish function sends JavaScript Object Notation (JSON) payloads with QoS 1 (at least once delivery). MQTT defines three QoS levels: 0 (at most once), 1 (at least once), and 2 (exactly once). Level 1 ensures the broker will retry until the subscriber acknowledges the message, which reduces the chance of losing score updates if the connection drops briefly.

How to Bridge MQTT to the Browser with Server-Sent Events

In this step, you'll create the Server-Sent Events (SSE) module that bridges MQTT messages to the browser. MQTT works over TCP and browsers cannot connect to it directly, so we use SSE as a lightweight HTTP-based streaming channel. SSE keeps an open connection and allows the server to push updates to connected browsers in real time. This is what enables viewers to see match updates instantly without refreshing.

Now create the file at server/sse.js:

const clients = new Set();

export function setupSSE(mqttClient) {
  if (!mqttClient) return;

  mqttClient.on('message', (topic, message) => {
    try {
      const payload = JSON.parse(message.toString());
      const data = JSON.stringify({ topic, ...payload });
      clients.forEach((res) => {
        try {
          res.write(`data: ${data}\n\n`);
        } catch (e) {
          clients.delete(res);
        }
      });
    } catch (e) {
      console.error('SSE parse error:', e.message);
    }
  });
}

export function addSSEClient(res) {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');
  res.setHeader('X-Accel-Buffering', 'no');
  res.flushHeaders();

  clients.add(res);

  res.on('close', () => {
    clients.delete(res);
  });
}

Explanation of each part:

clients Set: A Set holds all active Server-Sent Events response objects. Using a Set makes it easy to add and remove clients without duplicates.
setupSSE: Attaches a message listener to the MQTT client. When any message arrives on a subscribed topic, the callback runs. It parses the message payload (which is JSON), merges the topic into the payload with { topic, ...payload }, and sends the result to every client. The Server-Sent Events format requires each message to be data: {content}\n\n (two newlines). The forEach loop catches write errors (for example, if a client disconnected) and removes the client from the set.
addSSEClient: Sets the Content-Type header to text/event-stream so the browser treats the response as an event stream. The Cache-Control: no-cache and Connection: keep-alive headers prevent the browser or proxy from caching or closing the connection. The X-Accel-Buffering: no header disables buffering in Nginx, which can delay or block Server-Sent Events. The flushHeaders call sends the headers immediately so the connection is established. The close event handler removes the client when the client disconnects (closes the tab or navigates away).

Server-Sent Events is one-way (server to client). For this use case, that is enough because viewers only need to receive updates, not send messages back over the same channel.

How to Build the Admin Upload Interface

The admin interface is a single HTML page where match creators can create new matches, update scores, and add events such as goals or cards. It uses standard HTML forms so data can be submitted to the server, and JavaScript will later handle form submissions and dynamic updates. All the markup, styles, and script live in public/admin.html, which the Express server serves as a static page.

Admin HTML Structure and Styles

The document starts with the standard HTML5 boilerplate. The charset and viewport meta tags ensure proper character encoding and responsive layout on mobile devices. The page loads the Outfit font from Google Fonts for a clean, modern look.

The Cascading Style Sheets (CSS) uses custom properties (variables) in the :root block so you can change the color scheme in one place. The --bg variable holds the dark background color, --surface for card backgrounds, --accent for the green accent color, and --text-muted for secondary text. The .grid class creates a two-column layout for form fields that collapses to one column on screens under 600 pixels wide. The .toast class positions the notification at the bottom-right and uses transform and opacity for a slide-in animation when the .show class is added.

Now create the file at public/admin.html:




  
  
  Admin - Football Scores Upload
  
  
  
  


  
    
      ⚽ Football Scores
      Admin
      → Open Viewer
    

    
      Create New Match
      
        
          
            Home Team
            
          
          
            Away Team
            
          
          
            League
            
          
          
            Venue
            
          
          
            Kickoff (ISO)
            
          
        
        
          
        
      
    

    
      Update Score
      
        
          
            Select Match
            
          
          
            Home Score
            
          
          
            Away Score
            
          
          
            Minute
            
          
          
            Status
            
          
        
        
          
        
      
    

    
      Add Match Event (Goal, Card, etc.)
      
        
          
            Select Match
            
          
          
            Event Type
            
          
          
            Team
            
          
          
            Player
            
          
          
            Minute
            
          
          
            Description
            
          
        
        
          
        
      
    

    
      Active Matches

The three forms use id attributes (createMatch, updateScore, addEvent) so the JavaScript can attach submit handlers. The match dropdowns (scoreMatchId and eventMatchId) are populated dynamically when the page loads. The matchList div is the container for the list of active matches. The toast element sits outside the main container so it can be fixed to the viewport.

Admin JavaScript Logic

The script block handles all interaction between the admin page and the server. It defines helper functions for showing notifications, fetching matches, updating the UI, and submitting data. When the page loads or after any action (create, update, or event submission), the UI refreshes so the latest match data is always visible.

Explanation of each part:

showToast: updates the toast text and adds the show class to trigger the CSS animation. The isError parameter switches the toast to a red background for error messages. The setTimeout removes the show class after 3 seconds so the toast slides back out.
fetchMatches: calls the GET /api/matches endpoint and returns the parsed JSON so the UI can display the latest data.
populateSelects: builds option elements from the matches array and injects them into both dropdowns, so the same match list appears in the Update Score and Add Event forms.
renderMatchList: either shows a placeholder when there are no matches or renders each match as a card with team names, scores, league, status, and an Update button.
quickScore: pre-fills the Update Score form when you click the Update button on a match card, so you can adjust the score without re-selecting the match.
loadMatches: fetches matches, populates the dropdowns, and renders the list. It runs on page load and after every successful create, update, or event submission.
onsubmit: calls e.preventDefault() to stop the default form submission, builds a request body from the form values, sends a fetch request to the appropriate endpoint, and on success shows a toast and calls loadMatches to refresh the UI.

How to Build the Live Viewer Interface

The viewer interface displays real-time match updates and connects to the Server-Sent Events endpoint so it can receive data the moment the server pushes it.

Unlike the admin page, the viewer is read-only: it shows live scores, match events, and status updates without requiring user input. The page uses a dark theme and a connection indicator so users can see whether the real-time stream is active.

Now create the file at public/viewer.html:




  
  
  Live Football Scores
  
  
  
  


  
    
      ⚽ Live Football Scores
      
        
        Connecting...
      
    

    

    
      Live Feed

Explanation of each part:

header: displays the page title and a connection indicator so users know whether the real-time stream is active.
status dot: a small circle that pulses while disconnected and turns green with a glow when the SSE connection is established.
matches container: the div with id="matches" is where match cards will be rendered dynamically by JavaScript as data arrives.
feed section: shows a chronological list of live updates so users can see recent events at a glance.
CSS theme: uses dark colors and custom properties so the look can be adjusted in one place. The live badge and border styles highlight matches that are in progress.
Server-Sent Events integration: JavaScript (added in the next step) will connect to /api/events and update this page whenever new data arrives.

The header shows the title and a connection status indicator. The matches div is the container for match cards, populated by JavaScript. The feed div displays the live feed of recent updates.

Viewer JavaScript Logic

The script powers the live viewer interface by maintaining an in-memory collection of matches and a feed of recent updates. It connects to the Server-Sent Events endpoint so data flows from the server to the browser in real time. When messages arrive, the page updates automatically to show the latest scores, events, and match details.

Explanation of each part:

The matches Map: stores match objects keyed by ID so updates can be applied efficiently without searching arrays.
The feed array: keeps a small history of recent events (limited to MAX_FEED) so the live feed remains lightweight.
setStatus: toggles the connected class on the status dot and updates the status text to “Live” or “Reconnecting…” so users know connection status.
renderMatches: converts the Map to a sorted array (newest first). If there are no matches, it displays an empty state. Otherwise it renders cards showing league, venue, teams, scores, status badge, minute, and events.
getEventIcon: returns an emoji for each event type so events are visually identifiable (goal, card, substitution, and so on).
renderFeed: displays the live feed items or a placeholder message when no updates exist.
addFeedItem: appends a timestamped message to the feed, keeps only the latest items, and re-renders the feed.
handleMessage: processes incoming data. It updates the matches Map for full match objects, score updates, and match events. For score updates and goals it adjusts scores and adds feed items so the viewer reflects real-time changes.
handleSSEMessage: parses the Server-Sent Events payload and forwards it to handleMessage. If the message contains a match topic and full match data, it stores it in the Map.
connectSSE: creates an EventSource connection to /api/events. On open it marks the connection as live. On error it closes and retries after three seconds so transient network issues do not break the stream.
loadInitial: fetches existing matches on page load so the viewer displays data even before real-time updates arrive.

How to Build the Home Page

The home page (public/index.html) is a simple landing page that links to the viewer and admin interfaces. It uses a centered card layout with two buttons. The primary button (green) goes to the viewer, and the secondary button (outlined) goes to the admin. The page uses the same dark theme and Outfit font for consistency. There is no JavaScript – it's purely static HTML and CSS.




  
  
  Football Scores - MQTT Real-Time
  
  
  
  


  
    ⚽ Football Scores
    Real-time updates via MQTT & Mosquitto
    
      View Live Scores
      Admin - Upload Scores

The Express server serves index.html when you visit the root URL (http://localhost:3000/) because the express.static middleware serves files from the public folder, and Express automatically serves index.html when the request path is /.

How to Run and Test the System

Start the MQTT broker (Docker or local install).

Then start the Express server:

npm start

Next, open http://localhost:3000/admin.html in the admin panel. Create a new match (for example, Manchester United vs Liverpool).

Open http://localhost:3000/viewer.html in another tab or window. Then update the score or add an event in the admin panel. The viewer should update within a second without refreshing.

How to Extend the System for Production

The current implementation uses in-memory storage. For production, you should:

Add a database: Store matches in PostgreSQL, MongoDB, or another database. Load matches on startup and persist every create, update, and event.
Add authentication: Protect the admin routes with JSON Web Tokens (JWT) or session-based auth so only authorized users can upload scores.
Add validation: Validate request bodies with a library such as Joi or Zod to prevent invalid data.
Enable TLS: Use HTTPS for the Express server and secure WebSockets or MQTTS for the broker in production.
Scale horizontally: If you run multiple server instances, each will have its own MQTT connection and SSE clients. The MQTT broker will deliver messages to all subscribers, so each instance will receive updates and forward them to its connected viewers.

API Reference

For quick reference, here are the endpoints your server exposes:

Method	Endpoint	Description
GET	`/api/matches`	Returns all matches as a JSON array
POST	`/api/matches`	Creates a new match. Body: `{ homeTeam, awayTeam, league?, venue?, kickoff? }`
PATCH	`/api/matches/:id/score`	Updates a match score. Body: `{ homeScore?, awayScore?, minute?, status? }`
POST	`/api/matches/:id/events`	Adds an event. Body: `{ type?, team, player?, minute?, description? }`
GET	`/api/events`	Server-Sent Events stream for real-time updates

The status field can be scheduled, live, halftime, or finished. The type field for events can be goal, yellow_card, red_card, substitution, or penalty.

Troubleshooting

You might come across various issues as you build this. Here are some common ones:

The server cannot connect to the MQTT broker. Check that Mosquitto is running. If you use Docker, run docker ps to verify the container is up. If you use the public broker, ensure you have internet connectivity and that your firewall allows outbound connections on port 1883.

The viewer does not update when you change scores. Open the browser developer tools and check the Network tab. The /api/events request should show a pending state (it stays open). If it fails or closes, check the server logs for errors. Ensure you are not behind a proxy that buffers or closes long-lived connections.

Matches disappear when you restart the server. The current implementation stores matches in memory. Restarting the server clears the data. Add a database as described in the production section to persist matches across restarts.

Conclusion

In this tutorial, you built a real-time football sports update system using MQTT, Mosquitto, and Express. You've learned how to:

Connect an Express server to an MQTT broker
Publish match and score updates to MQTT topics
Subscribe to topics and forward messages to browsers via Server-Sent Events
Build an admin interface for creating matches and updating scores
Build a viewer interface that displays live updates without page refreshes

The same pattern applies to other real-time systems: IoT dashboards, live notifications, collaborative editing, and more. MQTT gives you a reliable, scalable messaging layer, and Server-Sent Events gives you a simple way to push updates to web clients.

The code in this tutorial gives you a solid foundation. Try adding features such as match filtering by league, historical event logs, or push notifications to make the system your own.

How to Optimize Django REST APIs for Performance: Profiling, Caching, and Scaling.

Mari — Tue, 17 Feb 2026 18:22:09 +0000

Performance problems in APIs rarely start as performance problems. They usually start as small design decisions that worked perfectly when the application had ten users, ten records, or a single developer testing locally. Over time, as traffic increases and data grows, those same decisions begin to slow everything down.

In this article, we’ll walk step by step through how performance issues arise in Django REST APIs, how to see them clearly using profiling tools, and how to fix them using query optimization, caching, pagination, and basic scaling strategies.

This article will be most useful for developers who already understand Django, the Django REST Framework, and REST concepts, but are new to performance optimization.

Why Django REST APIs Become Slow

Before optimizing anything, it’s important to understand why APIs become slow in the first place.

Most performance issues in Django REST APIs come from three main sources:

Too many database queries
Doing expensive work repeatedly
Returning more data than necessary

Django is fast by default, but it does exactly what you ask it to do. If your API endpoint triggers 300 database queries, Django will happily run all 300.

Now let’s look at some common causes of performance issues in Django REST APIs.

1. N+1 Query Problems in Serializers

This happens when you loop over objects and access related fields, causing a separate query for each object.

# models.py
class Author(models.Model):
    name = models.CharField(max_length=100)

class Post(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)

# views.py (naive approach)
posts = Post.objects.all()
for post in posts:
    # This triggers a query per post to fetch the author
    print(post.author.name)

If you have 100 posts, this runs 101 queries: 1 for posts and 100 for authors. Django lazily loads related objects by default, so without intervention, your API performs repetitive database work that slows response times.

# Naive queryset fetching all related objects separately
posts = Post.objects.all()
authors = [post.author for post in posts]  # triggers extra queries per post

Each access to post.author triggers a new query. Even though you already fetched all posts, Django lazily loads related objects by default. This creates many extra queries, slowing down your API.

3. Serializing Large Datasets Without Pagination

Returning large query sets all at once can slow down your API and increase memory usage.

# views.py
from rest_framework.response import Response
from rest_framework.decorators import api_view
from .models import Post
from .serializers import PostSerializer

@api_view(['GET'])
def all_posts(request):
    posts = Post.objects.all()  # retrieves all posts at once
    serializer = PostSerializer(posts, many=True)
    return Response(serializer.data)

If your database has thousands of posts, this endpoint fetches everything in memory, serializes it, and sends it over the network. It’s slow and can crash under load. Later, we’ll learn to paginate results efficiently.

4. Recomputing Expensive Work Repeatedly

Some endpoints calculate the same values on every request instead of caching or precomputing.

def expensive_view(request):
    # Simulate expensive computation
    result = sum([i**2 for i in range(1000000)])
    return JsonResponse({"result": result})

Even if the data doesn’t change often, this computation happens on every request, consuming CPU time unnecessarily.

Performance optimization is about reducing unnecessary work.

At this point, it might be tempting to jump straight into fixes like caching responses or optimizing database queries. But doing that without evidence often leads to wasted effort or even new problems.

Before changing anything, you need to understand where your API is actually spending time. Is it the database? Is it serialization? Is it Python code running repeatedly on every request? This is where profiling becomes essential.

Profiling: Finding the Real Bottlenecks

Optimizing without profiling is guessing. Profiling helps you answer one question:

Where is my API actually spending time?

In practice, profiling means observing an API while it runs and collecting data about what it’s doing. This includes how many database queries are executed, how long those queries take, and how much time is spent in Python code, such as serializers or business logic.

By profiling first, you avoid making assumptions and can focus on fixing the parts of your API that are truly slowing things down.

Measuring Query Count in a View

During development, Django keeps track of all executed queries. You can inspect them directly:

from django.db import connection
from rest_framework.decorators import api_view
from rest_framework.response import Response
from .models import Post
from .serializers import PostSerializer

@api_view(["GET"])
def post_list(request):
    posts = Post.objects.all()
    serializer = PostSerializer(posts, many=True)

    response = Response(serializer.data)

    print(f"Total queries executed: {len(connection.queries)}")

    return response

If this prints 101 queries for 100 posts, you likely have an N+1 problem. This simple check confirms whether the database layer is the bottleneck.

One of the easiest ways to profile Django applications during development is by using tools that expose this information directly while requests are being processed.

The Django Debug Toolbar is one of the simplest ways to understand performance during development. It acts as a lightweight profiling tool that shows what happens behind the scenes when a request is handled.

It shows you:

How many SQL queries were executed
How long each query took
whether queries are duplicated
Which parts of the request lifecycle are slow

First, install it:

pip install django-debug-toolbar

In settings.py:

INSTALLED_APPS = [
    ...
    "debug_toolbar",
]

MIDDLEWARE = [
    ...
    "debug_toolbar.middleware.DebugToolbarMiddleware",
]

INTERNAL_IPS = [
    "127.0.0.1",
]

In urls.py:

import debug_toolbar
from django.urls import path, include

urlpatterns = [
    ...
    path("__debug__/", include(debug_toolbar.urls)),
]

When you load an endpoint in the browser during development, the toolbar displays total SQL queries, execution time, and duplicate queries. This makes inefficiencies immediately visible.

When you load an API endpoint and see 150 SQL queries for a single request, that’s a strong signal that something is wrong, often an N+1 query problem or inefficient serializer behavior.

Logging SQL Queries

Django allows you to log all executed SQL queries. This is especially useful when debugging API views.

Seeing the raw SQL makes inefficiencies obvious, such as repeated SELECT statements for the same table.

How to Enable SQL Query Logging

You can configure Django to log all SQL queries in settings.py:

LOGGING = {
    "version": 1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
    },
    "loggers": {
        "django.db.backends": {
            "handlers": ["console"],
            "level": "DEBUG",
        },
    },
}

With this configuration, every SQL query will be printed to the console when your API runs. Repeated SELECT statements or unexpected queries become obvious.

Profiling API Response Time

Database queries are only one part of API performance. Beyond queries, it’s also important to measure the total response time of an endpoint.

Profiling response time helps you understand whether delays are caused by database access or by other parts of the request lifecycle. For example, if an endpoint takes 1.2 seconds to respond but only 50 milliseconds are spent on database queries, the bottleneck is likely in serialization, business logic, or repeated computations in Python.

By comparing query time and total response time, profiling helps you identify what to fix first instead of optimizing the wrong layer of the system.

How to Measure Total Response Time

import time
from rest_framework.decorators import api_view
from rest_framework.response import Response

@api_view(["GET"])
def example_view(request):
    start_time = time.time()

    # Simulate work
    data = {"message": "Hello world"}

    response = Response(data)

    end_time = time.time()
    print(f"Response time: {end_time - start_time:.4f} seconds")

    return response

If database queries are fast but the total response time is high, the bottleneck may be serialization or expensive Python logic.

Once you’ve identified that database access is a significant contributor to slow response times, the next step is to look more closely at how Django retrieves related data.

SQL Query Optimization in Django REST APIs

One of the most common reasons Django REST APIs become slow is inefficient access to related objects. This often manifests as the N+1 query problem, where fetching related objects triggers a separate database query for each item. Identifying and fixing this problem can significantly reduce the number of queries and improve API performance.

Understanding the N+1 Query Problem

Consider a simple example:

You fetch a list of posts
Each post has an author
For every post, Django fetches the author separately

If you have 100 posts, this results in 101 queries: 1 for the posts and 100 for the authors. This happens because Django lazily loads related objects by default. Without intervention, your API performs repetitive database work that slows down response times.

Solving the Problem with `select_related` and `prefetch_related`

Django provides built-in tools to control how related objects are loaded efficiently: select_related and prefetch_related.

1. Using select_related

select_related is designed for foreign key and one-to-one relationships. It performs an SQL join and retrieves related objects in a single query.

Use it when:

You know you will access related objects
The relationship is one-to-one or many-to-one

posts = Post.objects.select_related("author")

for post in posts:
    print(post.author.name)  # No additional queries

This performs a SQL JOIN and retrieves posts and authors in a single query, eliminating the N+1 problem.

It reduces multiple queries into just one, avoiding repeated database hits.

2. Using prefetch_related

prefetch_related is used for many-to-many and reverse foreign key relationships. It performs separate queries for each related table but combines the results in Python.

Use it when:

A SQL join would produce too much duplicated data
You are dealing with collections of related objects

Example: How to Optimize a Many-to-Many Relationship

Consider a blog application where posts can have multiple tags:

# models.py
class Tag(models.Model):
    name = models.CharField(max_length=50)

class Post(models.Model):
    title = models.CharField(max_length=200)
    tags = models.ManyToManyField(Tag)

Now imagine fetching posts and accessing their tags:

posts = Post.objects.all()

for post in posts:
    print(post.tags.all())  # Triggers additional queries

If you have 100 posts, Django may execute:

1 query to fetch posts
1 query per post to fetch related tags

This results in many unnecessary database hits.

You can optimize this using prefetch_related:

posts = Post.objects.prefetch_related("tags")

for post in posts:
    print(post.tags.all())  # Uses prefetched data

With this approach, Django performs one query for posts and one query for all related tags. It then matches them in Python, eliminating repeated database queries.

Together, these tools allow you to optimize your queries and eliminate the N+1 problem efficiently.

Common Beginner Mistakes

Even after applying these optimizations, it’s easy to make mistakes. Watch out for:

Forgetting that serializers can trigger additional queries
Using select_related on many-to-many relationships
Assuming Django automatically optimizes queries
Not checking the query count after adding serializers

Paying attention to these pitfalls ensures your API remains fast and scalable.

Caching in Django REST APIs

Even after optimizing database queries, API performance can still suffer if the same computations or database lookups are performed repeatedly. This is where caching comes in. Caching is a technique for storing the results of expensive operations so they can be retrieved more quickly the next time they are needed.

At its core, caching exists because computers have multiple layers of memory with different speeds:

CPU registers (fastest)
L1, L2, L3 caches
Main memory (RAM)
SSD storage
HDD storage (slowest)

Each layer trades speed for size: the closer the data is to the CPU, the faster it can be accessed. Software systems use the same principle; by storing frequently accessed data in a “closer” or faster location, applications can respond more quickly.

Cache Eviction

Caches are limited in size, so when a cache is full, some data must be removed to make room for new data. This process is called cache eviction.

Common eviction strategies include:

Least Recently Used (LRU): removes the data that hasn’t been accessed for the longest time
Random Replacement: removes a random item from the cache

The goal is to keep the data that is most likely to be requested again while freeing space for new data. Understanding this helps developers use caching effectively.

Caching in Application Architectures

Caching exists at several levels in modern software systems:

Client-side caching: Web browsers cache HTTP responses to reduce the need for repeated network requests. This is controlled with HTTP headers like Cache-Control.
CDN caching: Content Delivery Networks store static assets closer to users, reducing latency and server load.
Backend caching: Backend services cache results from database queries, computed values, or API responses. This is where Django caching is most commonly applied.

By applying caching strategically at the backend, APIs can serve data faster while reducing computation and database load.

Caching in Django

Django provides a flexible caching framework that supports multiple backends, including in-memory, file-based, database-backed, and third-party stores like Redis. The main types of caching in Django are:

Per-view caching: caches the entire output of a view. Ideal for endpoints where responses rarely change.
```
 from django.views.decorators.cache import cache_page

 @cache_page(60 * 15)  # cache for 15 minutes
 def my_view(request):
```
1. Template fragment caching: caches specific parts of a template to avoid repeated rendering.
2. Low-level caching: gives full control over what is cached and for how long, making it ideal for API responses.

By combining these approaches, you can reduce repeated work in your API, lower database load, and speed up response times.

When to Use Redis

While Django’s built-in caching backends are sufficient for many projects, high-traffic APIs often require a shared, in-memory cache. This is where Redis excels. Redis is designed for fast access, low latency, and can handle frequent reads across multiple servers.

You should consider using Redis when:

Data is read frequently but changes infrequently
Low latency is important for API responses
You need cache expiration and eviction policies
You want a shared cache across multiple servers or services

Redis is particularly effective for API endpoints that serve the same data to many users, such as frequently accessed lists or computed results.

Common Beginner Mistakes

Caching is powerful, but it’s easy to misuse. Some common pitfalls include:

Caching everything blindly: not all data benefits from caching
Forgetting cache invalidation: stale data can lead to incorrect responses
Using cache where query optimization would suffice: sometimes optimizing database queries is a better solution than caching.

Remember: caching should complement good database design, not replace it.

Pagination and Limiting Expensive Datasets

Even with caching, returning large datasets in a single request can slow down your API and increase memory usage. Pagination is a simple and effective way to limit the amount of data returned at once.

Pagination helps by reducing:

Database load
Memory usage
Serialization time
Network transfer size

Django REST Framework provides built-in pagination classes that make it easy to paginate endpoints. As a rule of thumb, always paginate list endpoints unless there is a strong reason not to.

Load Testing and Measuring Improvement

Optimizations are only meaningful if you can measure their impact. Load testing simulates multiple users accessing your API simultaneously, helping you answer key questions:

How many requests per second can my API handle?
Where does the API start to break under load?
Did caching, query optimization, and pagination actually improve performance?

By running load tests before and after optimization, you can validate that your changes have the desired effect and avoid optimizing the wrong parts of your system.

Summary and Next Steps

Optimizing Django REST APIs isn’t about chasing every tiny micro-optimization. It’s about reducing unnecessary work and focusing on the parts of your API that actually slow down performance.

Key Takeaways

Profile before optimizing: Identify the real bottlenecks before making changes.
Reduce database queries: Use techniques like select_related, prefetch_related, and avoid N+1 queries.
Cache frequently accessed data: Use Django caching and Redis to reduce repeated computations.
Paginate large datasets: Limit memory usage and network load by returning data in chunks.
Measure performance changes: Always verify that your optimizations have a real impact.

Next Steps for Your APIs

Add profiling to your existing APIs to understand where time is spent.
Identify one slow endpoint and focus on optimizing it first.
Optimize database queries using Django ORM best practices.
Introduce caching carefully; avoid caching everything blindly.
Measure the results with load testing and performance metrics.

Remember: Performance optimization is not a one-time task. It’s a habit built by continuously observing how your system works, testing improvements, and applying changes where they make the most impact.

How to Use the Django REST Framework - Build Backend APIs with DRF

Mari — Fri, 21 Nov 2025 21:13:28 +0000

When you click on most backend development tutorials, they often teach you what to do, not how to think.
That’s why many developers only realize their mistakes after they start building.

So, how does one actually think like a backend developer? Before answering that, let’s start with the basics: what exactly is backend development?

What is Backend Development?
Why Django REST Framework?
How to Think Like a Backend Developer
How to Install the Django REST Framework
The Backend Developer’s Mindset
Common Mistakes Beginners Make
Further Reading
Conclusion

What is Backend Development?

Backend development is the foundation of most web and mobile applications. It focuses on everything that happens behind the scenes, from processing logic and handling data to connecting with databases and APIs.

While it’s true that backend developers build APIs that communicate with the frontend, the job goes far beyond that. The backend is where data is validated, protected, stored, and retrieved.

In short: backend development is about building systems that ensure data integrity, performance, and scalability.

Backend developers are the ones responsible for designing and maintaining those systems. They ensure that every user request is processed efficiently and securely.

Now, how does the Django REST Framework (DRF) fit into all this?

Why Django REST Framework?

A beginner-friendly tutorial must use a tool that:

Teaches good structure
Encourages best practices
Hides unnecessary complexity
Helps you learn backend fundamentals correctly

That’s why this guide uses the Django REST Framework (DRF). Here’s how it compares to other popular Python frameworks.

Flask

Flask is a lightweight and flexible microframework. It is great for small projects, but:

You have to set up everything manually (routing, JSON handling, database handling).
You need extra libraries for authentication, validation, or serialization.
Beginners often create unstructured projects because Flask doesn’t enforce architecture.

Flask teaches freedom, not structure.

FastAPI

FastAPI is modern, fast, and async-first. However:

It assumes you already understand APIs.
It requires understanding Python type hints deeply.
The ecosystem is still growing.
Beginners may not understand its underlying concepts (dependency injection, async IO).

FastAPI teaches speed, not fundamentals.

Django REST Framework

DRF is ideal for beginners because:

It sits on top of Django, a very stable full-stack framework.
It encourages good architecture from day one.
It handles serialization, authentication, routing, validation, and permissions for you.
It gives you structure instead of chaos.

Bottom line: DRF can help you learn how backend systems work from scratch.

How to Think Like a Backend Developer

Thinking like a backend developer is not about memorizing code. It’s about learning to see the bigger picture, how data moves, how logic flows, and how to build systems that work reliably and can grow.

Backend thinking can be summarized into six main principles:

1. Think in Systems, Not Lines of Code

Many beginners focus on writing code that works for one feature. A backend developer thinks about the entire system.

Analogy: Imagine a factory. Each machine (function or endpoint) does one task, but the factory only works efficiently if every machine is arranged correctly and communicates properly.

Example: When a user submits a form to create a task:

The request reaches the server.
The backend validates the data.
The backend stores it in the database.
The backend sends a response to the user.

A backend developer doesn’t just write a function to save data. They ask:

Where should this logic live — view, serializer, or service layer?
How will the data be validated and cleaned?
How will the system scale if thousands of users submit tasks at the same time?

Seeing the system first makes code predictable, maintainable, and scalable.

2. Separate Concerns — Keep Things Organized

Backend thinking is about structure. Every piece of code should have a clear responsibility:

Models: Store and define your data
Serializers: Convert data to a format the client understands (like JSON)
Views: Apply the business logic and respond to requests

Why this matters: Without separation, code becomes messy and hard to debug. You might find yourself mixing database queries with validation or formatting, which leads to errors later.

Simple analogy: Think of a restaurant.

The chef prepares the food (model/data).
The waiter delivers the food to customers in a presentable way (serializer).
The manager decides who gets what and handles special requests (view/logic).

Each role is separate but connected. This is exactly how backend developers structure code.

3. Anticipate Problems Before They Happen

Backend developers don’t just code for today. They think ahead:

What if the user sends invalid data?
What if two users try to edit the same record at the same time?
How will the system handle millions of requests in the future?

Example: If a user tries to create a task without a title, a beginner might just let it crash. A backend developer writes validation rules to catch this and return a clear error message.

Rule of thumb: Always ask, “What could go wrong here?” and design your code to handle it gracefully.

4. Make Your Code Predictable and Readable

Backend development is about writing code for humans, not just computers.

Use clear variable names (task_title instead of x).
Keep functions short and focused.
Document your code.

This way, anyone can pick up your code and understand it, including your future self.

Tip: A backend system that is easy to read and predict is easier to debug, extend, and scale.

5. Think in the Request → Logic → Response Cycle

Every backend action fits into this pattern:

Request: The client sends data.
Logic: The server validates, processes, and decides what to do.
Response: The server sends data back in a structured way.

Example: User creates a task:

Request: { "title": "Learn DRF" }
Logic: Check title is not empty → save to database → mark completed as False
Response: { "id": 1, "title": "Learn DRF", "completed": false }

Thinking in this cycle makes debugging and designing systems intuitive.

6. Practice Thinking Like a Backend Developer

Ask questions before coding: “Where should this logic live? How will this affect other parts of the system?”
Break down problems into steps: Don’t just code the solution; code the process.
Visualize data flow: Draw diagrams if necessary, from user request to database and back.
Learn by doing: Build small projects and reflect on each component’s role.

Check out Andy Harris’s video on how to think like a programmer.

Now that you understand how backend developers think, let’s walk through setting up a real backend environment using Django REST Framework.

How to Install the Django REST Framework

Here’s how to get the Django REST framework running on your machine from scratch.

Step 1: Install Python

Make sure you have Python 3.8+ installed. You can check if Python is installed with this command:

python --version

If it’s not installed, download it from the official Python documentation.

Step 2: Create a Project Folder

Choose a location on your computer and create a folder for your project:

mkdir my_drf_project
cd my_drf_project

This keeps all your files organized in one place.

Step 3: Create a Virtual Environment

A virtual environment keeps your project dependencies separate from other projects.

Create a virtual environment:

python -m venv venv

Next, activate it. For Windows (PowerShell):

.\venv\Scripts\Activate.ps1

For Mac/Linux:

source venv/bin/activate

You’ll know it’s active when your terminal prompt starts with (venv).

Step 4: Install Django

Now install Django inside the virtual environment:

pip install django

Check that Django is installed:

python -m django --version

Step 5: Create a Django Project

Create a new Django project:

django-admin startproject core .

The . at the end means “create the project here.” Run the server to make sure it works:

python manage.py runserver

Visit http://127.0.0.1:8000/ in your browser. You should see the Django welcome page.

Step 6: Install Django REST Framework

Install DRF using pip:

pip install djangorestframework

Step 7: Add DRF to Installed Apps

Open core/settings.py and find the INSTALLED_APPS list. Add:

'rest_framework',

It should look like this:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
]

Step 8: Run Initial Migrations

Set up your database:

python manage.py migrate

Create a superuser for accessing the admin panel:

python manage.py createsuperuser

Follow the prompts for username, email, and password.

Step 9: Start the Server

Run your development server again:

python manage.py runserver

Visit:

http://127.0.0.1:8000/ → Django welcome page
http://127.0.0.1:8000/admin/ → Admin panel (login with superuser)

You now have Django + DRF installed and ready for API development.

Step 10: Verify DRF Installation

The easiest way to confirm that Django REST Framework is installed correctly is to build a very small test API. Each part of the setup helps you verify that DRF is working end-to-end.

Create a new app:

python manage.py startapp api

This creates an api folder where you’ll place your test model, serializer, and view. Adding it to INSTALLED_APPS tells Django to recognize the new app.

Add it to INSTALLED_APPS:

'api',

Create a simple models.py in the api app:

from django.db import models

class Task(models.Model):
    title = models.CharField(max_length=200)
    completed = models.BooleanField(default=False)

This model represents a basic task with a title and a completion status. Creating even a simple model lets you test whether DRF can serialize and expose database objects as API responses.

Run migrations:

python manage.py makemigrations
python manage.py migrate

These commands generate and apply database tables for the Task model. Without migrations, DRF won’t have anything to fetch and serialize.

Create a serializer (api/serializers.py):

from rest_framework import serializers
from .models import Task

class TaskSerializer(serializers.ModelSerializer):
    class Meta:
        model = Task
        fields = '__all__'

A serializer converts your Task model into JSON so it can be returned as an API response. This step confirms that DRF’s serializer tools are working.

Create a view (api/views.py):

from rest_framework import viewsets
from .models import Task
from .serializers import TaskSerializer

class TaskViewSet(viewsets.ModelViewSet):
    queryset = Task.objects.all()
    serializer_class = TaskSerializer

ModelViewSet automatically creates the CRUD API endpoints for your model. If this loads correctly, it means DRF’s generic views and viewsets are functioning.

Wire it to URLs (core/urls.py):

from django.urls import path, include
from rest_framework.routers import DefaultRouter
from api.views import TaskViewSet

router = DefaultRouter()
router.register('tasks', TaskViewSet)

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include(router.urls)),
]

The router generates routes /api/tasks/ for you. If routing works, DRF is properly integrated into your Django project.

Test the API by visiting:

http://127.0.0.1:8000/api/tasks/

If everything is set up correctly, you’ll see Django REST Framework’s browsable API. This confirms that DRF is installed, your project recognizes it, and it can serialize and return data successfully.

The Backend Developer’s Mindset

When writing backend code, your goal isn’t just to make something work; it’s to make it predictable, scalable, and maintainable.

Professional backend developers focus on:

Predictability over cleverness — Code should be clear to others.
Separation of concerns — Keep logic, data, and presentation layers distinct.
Validation — Never trust user input; always validate.
Consistency — Stick to naming conventions and reusable patterns.

This mindset is what separates backend coders from backend engineers.

Common Mistakes Beginners Make

Writing too much logic in views: Keep views light. Move business logic into services or serializers.
Ignoring validation: Always define validation rules in your serializers.
Not planning for scalability: Even small projects grow. Build like you expect more users.

Conclusion

Thinking like a backend developer isn’t about memorizing syntax; it’s about understanding how systems behave.

When you start reasoning through requests, logic, and responses, you begin to see the bigger picture, and that’s when you stop writing code and start building systems.

With Django REST Framework, that process becomes easier, cleaner, and more intuitive.

As you continue learning, build small APIs and gradually add features. The more you understand how data flows through a system, the more naturally backend thinking will come.

How to Get Started with PocketBase: Build a Lightweight Backend in Minutes

Manish Shivanandhan — Fri, 14 Nov 2025 18:11:55 +0000

If you’re a developer looking for a simple, fast, and self-hosted backend, PocketBase might be exactly what you need.

It’s an open-source backend written in Go that lets you set up a complete backend with database, authentication, file storage, and real-time updates, all in a single executable file.

In this guide, we’ll explore what makes PocketBase special, how to set it up, and how you can deploy it to the cloud.

What We’ll Cover:

What is PocketBase?
Why Developers Love PocketBase
How to Install Pocketbase
Extending PocketBase with JavaScript
Using SDKs to Interact with PocketBase
Self-Hosting PocketBase using Sevalla
Security and Open Source Nature
When to Use PocketBase
Conclusion

What is PocketBase?

PocketBase is an all-in-one backend that provides everything you need to power a modern web or mobile app, eliminating the need for large infrastructure.

It includes an embedded SQLite database, real-time subscriptions, file and user management, a clean admin dashboard, and a REST-style API.

Since it runs from a single file, you can deploy it almost anywhere, from a VPS to your local machine or even a Raspberry Pi.

It’s designed for developers who want control and simplicity at the same time. You don’t need to manage separate servers for authentication, storage, and API endpoints. PocketBase handles all of this out of the box. You can use it as a standalone backend or embed it in your Go application to create a custom solution.

Why Developers Love PocketBase

PocketBase focuses on speed and simplicity. You don’t need to install multiple packages or services.

Once downloaded, you can start it with a single command. Then it’ll launch a web-based admin dashboard.

The database is built using SQLite, which means data is stored locally by default, but you can use extensions to connect it with your existing workflows or cloud storage.

Another major advantage is its real-time capabilities. Every change in the database can be broadcast instantly to connected clients through WebSocket subscriptions. This makes it perfect for building apps like chat systems, dashboards, and collaboration tools that require instant updates.

How to Install PocketBase

Getting PocketBase running takes less than a minute. You can download a prebuilt executable from the official releases page.

It supports all major platforms, including Windows, macOS, and Linux.

Once downloaded, extract the archive and navigate to the folder in your terminal. Run the following command:

./pocketbase serve

This command starts a local server and launches the admin dashboard at http://127.0.0.1:8090/_/. From there, you can create collections, add users, upload files, and manage data. There’s no setup wizard or dependency installation – everything is self-contained inside that one binary.

If you’re a Go developer, you can also install PocketBase as a Go module and use it directly in your project to build custom logic or extend the existing API.

Using PocketBase as a Go Framework

PocketBase can act as a Go framework, letting you build your own backend logic while keeping everything in one file. Here’s a simple example that shows how you can extend it with a custom route.

package main

import (
    "log"
    "github.com/pocketbase/pocketbase"
    "github.com/pocketbase/pocketbase/core"
)
func main() {
    app := pocketbase.New()
    app.OnServe().BindFunc(func(se *core.ServeEvent) error {
        se.Router.GET("/hello", func(re *core.RequestEvent) error {
            return re.String(200, "Hello world!")
        })
        return se.Next()
    })
    if err := app.Start(); err != nil {
        log.Fatal(err)
    }
}

Once the file is ready, run these commands:

go mod init myapp && go mod tidy
go run main.go serve

You’ll now have a working backend with both the PocketBase dashboard and your custom /hello endpoint available at the same time.

This makes PocketBase flexible, you can use it as a ready-to-run backend or as part of a more complex Go project.

Extending PocketBase with JavaScript

PocketBase includes a built-in JavaScript engine that lets you extend its behavior without modifying or recompiling the Go code. This makes it easy to add custom logic, validation, automation, and event-driven workflows directly inside your backend.

You can create JavaScript files inside the pb_hooks folder, and PocketBase will automatically load and run them. These scripts can listen to events like record creation, updates, authentication, and more.

Here’s a simple example that sends a welcome email whenever a new user signs up.

Create a file named pb_hooks/user_email.js inside your PocketBase directory:

/// 

onRecordAfterCreate("users", async (e) => {
    const user = e.record;

    console.log("New user registered:", user.email);

    // Example: send a welcome email using a third-party API
    const emailResponse = await $http.send({
        url: "https://api.example.com/send-email",
        method: "POST",
        body: JSON.stringify({
            to: user.email,
            subject: "Welcome to our app!",
            message: `Hi ${user.username}, thanks for signing up!`
        }),
        headers: {
            "Content-Type": "application/json"
        }
    });

    console.log("Email status:", emailResponse.status);
});

This script runs automatically whenever a new record is created in the users collection. It picks up the user’s email, logs it, and uses PocketBase’s built-in HTTP client ($http) to call an external email service.

You can use the same pattern to validate data before saving, trigger webhooks, block actions, or update related records. Since everything runs inside PocketBase, you don’t need extra servers or functions to automate backend logic.

This makes it friendly for teams who may not be comfortable with Go but still want to add dynamic logic to their backend. You can find more details in the official documentation under “Extend with JavaScript.”

Using SDKs to Interact with PocketBase

To make it easier to communicate with your backend, PocketBase provides official SDKs for JavaScript and Dart.

The JavaScript SDK works well for browser-based or Node.js projects, while the Dart SDK is ideal for mobile apps built with Flutter. Both SDKs provide an easy way to connect, authenticate users, and perform CRUD operations without manually writing HTTP requests.

For example, in JavaScript you can connect and fetch data like this:

import PocketBase from 'pocketbase'

const pb = new PocketBase('http://127.0.0.1:8090')
const records = await pb.collection('posts').getList(1, 20)
console.log(records)

This simplicity allows you to focus on building your frontend while PocketBase handles authentication, database operations, and real-time updates.

Self-Hosting PocketBase using Sevalla

When you are ready to move beyond testing, PocketBase gives you two options. You can self-host it using your own infrastructure or use their managed cloud version at Pocketbase.io.

Self-hosting gives you full control and is usually preferred by technical teams who want to keep sensitive data in-house.

You can choose any cloud provider, like AWS, DigitalOcean, or others to set up Pocketbase. But I will be using Sevalla.

Sevalla is a PaaS provider designed for developers and dev teams shipping features and updates constantly in the most efficient way. It offers application hosting, database, object storage, and static site hosting for your projects.

I am using Sevalla for two reasons:

Every platform will charge you for creating a cloud resource. Sevalla comes with a $50 credit for us to use, so we won’t incur any costs for this example.
Sevalla has a template for PocketBase, so it simplifies the manual installation and setup for each resource you will need for installation.

Click on the “PocketBase” template. You will see the resources needed to provision the application. Click on “Deploy Template”:

You can see the resource being provisioned. Once the deployment is complete, go to the PocketBase application and click on “Visit app”

You will see a 404 message. Add _ to the url and you will see the login dashboard.

To login for the first time, you will need a superuser login. To create that, go back to the application and click “Logs”. You will see a url starting with https://0.0.0.0.

Replace the 0.0.0.0 with your new cloud URL and copy paste the full path into the browser. You will see the option to create a super user for your PocketBase deployment.

Once you have created the super user, you can again go to /_ and login using the username and password. You should now see the PocketBase dashboard.

You now have a production-grade Pocketbase server running on the cloud. You can use this to set up tables for your database and use the JavaScript or other SDKs to interact with Pocketbase.

Security and Open Source Nature

PocketBase is open source and licensed under MIT, which means you’re free to use it in personal or commercial projects. If you find a bug or security issue, you can report it to the maintainers, and they’ll address it promptly.

The project’s transparent development and active community make it a solid choice for startups, indie developers, and hobbyists who prefer to own their infrastructure.

Because it’s still in active development, backward compatibility isn’t guaranteed before version 1.0. But it’s already stable enough for small to medium-scale applications.

When to Use PocketBase

PocketBase is perfect for projects that need a simple backend with low maintenance. It’s ideal for prototypes, small SaaS products, indie apps, internal tools, and educational projects.

Instead of setting up a complex stack with PostgreSQL, Node.js, and nginx, you can get your backend running instantly and focus on your product.

Suppose your project later grows into something bigger. In that case, you can migrate to a more complex setup or continue using PocketBase as a lightweight service for specific features like authentication or real-time data sync.

Conclusion

PocketBase brings back the joy of fast development without complicated setups. With just one executable, you get a backend that supports authentication, real-time updates, file uploads, and an admin dashboard. It’s open source, fast, and customizable, making it a great choice for developers who want to move quickly without giving up control.

Whether you’re building a personal app, a startup MVP, or an internal dashboard, PocketBase gives you the power to set up a full backend in minutes. You can start small, extend it as needed, and deploy it anywhere – all while keeping your workflow simple and efficient.

Hope you enjoyed this article. Find me on Linkedin or visit my website.

How to Implement Dependency Injection in FastAPI

Nneoma Uche — Fri, 14 Nov 2025 14:46:01 +0000

Several languages and frameworks depend on dependency injection—no pun intended. Go, Angular, NestJS, and Python's FastAPI all use it as a core pattern.

If you've been working with FastAPI, you've likely encountered dependencies in action. Perhaps you saw Depends() in a tutorial or the docs and were confused for a minute. I certainly was. That confusion sparked weeks of experimenting with this system. The truth is, you can't avoid dependency injection when building backend services with FastAPI. It's baked into the framework's DNA, powering everything from authentication and database connections to request validation.

FastAPI's docs describe its dependency injection system as 'powerful but intuitive.' That’s accurate, once you understand how it works. This article breaks it down, covering function dependencies, class dependencies, dependency scopes, as well as practical examples.

Prerequisites
Dependencies and Dependency Injection in FastAPI
Getting Started: Environment Setup
Types of Dependencies in FastAPI
- How to Use Function Dependencies in FastAPI
- How to Use Class Dependencies in FastAPI
Dependency Scope
Common Use Cases for Dependency Injection
Conclusion

Prerequisites

To follow along with this article, you should have:

Working knowledge of Python.
Ability to create and activate virtual environments.
Basic understanding of FastAPI.
Familiarity with Object-Oriented Programming (OOP) concepts.

Dependencies and Dependency Injection in FastAPI

A dependency is a reusable piece of logic, like authentication, database connection, or validation, that your path operations require. Dependency injection (DI) is how FastAPI delivers these dependencies to specific parts of your application: you declare them using Depends() and FastAPI automatically executes them when the associated route receives a request.

Think of it as requesting the tools your application needs. You declare dependencies once and FastAPI provides them wherever needed, with no repetitive setup across routes.

This makes for modular, scalable applications. Without DI, you would have to repeat the same setup code on every endpoint, making updates tedious and bugs more likely.

Getting Started: Environment Setup

Let's set up your development environment to work through the examples in this guide.

Start by creating a project folder, then:

Create and activate a virtual environment:

python -m venv deps
source deps/bin/activate          #on Mac
deps\Scripts\activate             # On Windows

Install FastAPI with all dependencies:

pip install 'fastapi[all]'

Organize your project as follows:

fastapi-deps/
├── deps/                 # Virtual environment
├── function_deps.py
├── class_deps.py
├── router_deps.py
├── app.py
└── requirements.txt

Types of Dependencies in FastAPI

In FastAPI, a dependency is a callable object that retrieves or verifies information before a route executes. Dependencies can be implemented as either functions or classes.

Function dependencies are the most straightforward approach and work well for most use cases, including validation, authentication, and data retrieval. Class dependencies can handle the same tasks but are particularly useful when you need stateful logic, multiple instances with different configurations, or prefer object-oriented patterns.

How to Use Function Dependencies in FastAPI

A function dependency is a helper function (such as for authentication or data retrieval) that can be injected into path operations. To demonstrate, we'll create a simple user authentication dependency using an in-memory database—a list of dictionaries.

Recall the folder structure from earlier? We’ll write this code in fastapi-deps/function_deps.py.

Start by importing the required modules:

from fastapi import FastAPI, Depends, HTTPException
import uvicorn

You bring in FastAPI to create the app instance, Depends for dependency injection, and HTTPException to handle errors gracefully. uvicorn will be used to run the application later.

Next, instantiate the FastAPI application:

app = FastAPI()

app = FastAPI() creates your application instance: the object that will hold all your endpoints and dependencies.

Next, create an in-memory database. Define a list of dictionaries to act as your temporary database. Each dictionary represents a user entry containing a name and a password.

users = [
    {"name": "Ore", "password": "jkzvdgwya12"},
    {"name": "Uche", "password": "lga546"},
    {"name": "Seke", "password": "SK99!"},
    {"name": "Afi", "password": "Afi@144"},
    {"name": "Sam", "password": "goTiger72*"},
    {"name": "Ozi", "password": "xx%hI"},
    {"name": "Ella", "password": "Opecluv18"},
    {"name": "Claire", "password": "cBoss@14G"},
    {"name": "Sena", "password": "SenDaBoss5"},
    {"name": "Ify", "password": "184Norab"}  
]

💡

This type of database isn’t persistent; any data stored therein is lost when the application restarts.

Then, define a dependency function for user validation. The simple helper function below checks whether a username and password provided by the user match an existing user in the database.

#the dependency function
def user_dep(name: str, password: str):
    for u in users:
        if u["name"] == name and u["password"] == password:
            return {"name": name, "valid": True}

This function expects two string parameters, name and password, from the incoming request. If it finds a match in the users database, it returns a dictionary confirming the user’s validity. FastAPI automatically converts this dictionary into a JSON response.

Next, inject the dependency into a path function:

#the web endpoint
@app.get("/users/{user}")
def get_user(user = Depends(user_dep)) -> dict:
    if not user:
        raise HTTPException(status_code=401, detail="Invalid username or password")
    return user

The user_dep function is injected into the path operation using Depends(). When an HTTP request is made to this endpoint, FastAPI executes the dependency first, validates the input, and passes its return value to the user parameter.

The -> dict: annotation indicates that the function returns a dictionary, which FastAPI auto-converts to JSON. If no matching record is found, an HTTPException with a 401 status code is raised; otherwise, the verified user data is returned.

Now you’ll start the FastAPI server. To start the server, open your terminal in the project directory and run:

uvicorn function_deps:app --reload

function_deps is the name of your Python file (without the .py extension).
--reload automatically restarts the server whenever you save changes.

Once it starts, you’ll see an output similar to the image below:

Now you can test the endpoint. Open your browser or the Postman desktop app to validate the user “Seke”. Paste this URL into your browser: http://127.0.0.1:8000/users/{user}?name=Seke&password=SK99!

Alternatively, you can test the endpoint using FastAPI’s built-in docs at: http://127.0.0.1:8000/docs

In the Swagger UI:

Click on the Get User endpoint
Click Try it out
Enter “Seke” in the name field and “SK99!” in the password field
Click Execute

You should get a 200 status code, with the payload in this image:

You can also test the endpoint with usernames or passwords that don’t exist in the database. Each time, you should see a 401 error like this:

How to Use Class Dependencies in FastAPI

While functions are the most common way to define dependencies, FastAPI also supports class-based dependencies. Classes are useful when you need reusable instances with configurable state or prefer object-oriented patterns.

Class dependencies inject the same way: through the Depends function in your path operation.

Let's convert the user_dep function dependency to a class. It will authenticate users, grant access to valid credentials, and raise exceptions for unauthorized attempts. We'll apply it to a user dashboard endpoint to ensure only authenticated users access their resources.

#Dependency class for user authentication
class UserAuth():
    def __init__(self, name: str, password: str):
        self.name = name
        self.password = password

    def __call__(self):
        #check if name and password entered correspond to any row in the db
        for user in users:
            if user["name"] == self.name and user["password"] == self.password:
                pass
        #If no match found, raise an error
        raise HTTPException(status_code=401, detail="Invalid username or password")

The __init__ method receives the parameters from the request (name and password) and stores them as instance attributes. These can then be accessed in the __call__ method, which contains the dependency logic.

Note that __call__ doesn't return a value in this example. It simply raises an HTTPException if authentication fails. The __call__ method makes the class instance callable, allowing FastAPI to invoke it like a regular function.

Here’s how to inject UserAuth into a path function:

#Injecting the class dependency into a path operation
@app.get("/user/dashboard")
def get_dashboard(user: UserAuth = Depends(UserAuth)):
    return {"message": f"Access granted to {user.name}"}

What's happening here:

When a client requests the /user/dashboard endpoint, FastAPI executes the dependency first. Recognizing UserAuth as a class, FastAPI automatically creates an instance and populates it with values from the query parameters.

Here’s the execution flow to help you understand:

Depends(UserAuth) tells FastAPI: “Before running this route, create a UserAuth instance.”
FastAPI extracts name and password from the request URL (for example, /user/dashboard?name=Seke&password=SK99!).
It then calls UserAuth(name=”Seke”, password=”SK99!”) to create the instance.

The UserAuth instance, with its stored name and password attributes, is passed to the user parameter in get_dashboard.
The route function can access user.name and user.password directly.
If __call__ raises an exception, the route never executes.

Test the endpoint with valid credentials from the users list, and you should see output like this:

A closer look at FastAPI’s official documentation provides an alternative approach to classes as dependencies. However, using the __call__ method, in my opinion, is the most straightforward and self-contained approach. It keeps your authentication logic modular without adding extra code to the path operation.

The trade-off is that class dependencies are more verbose than helper functions, but cleaner for complex logic.

Dependency Scope

FastAPI offers two ways to inject dependencies into a path operation: as a function parameter or via the path decorator. When you include a dependency as a function parameter, the dependency's return value is available within the function. But when injected into the decorator, the dependency executes without passing a return value to the path function.

Beyond single endpoints, FastAPI lets you inject dependencies at the router or global level. Let’s examine these scopes in more detail.

Path Operation Level

While the first example injected dependencies into path function parameters, you can also inject them directly into the decorator using the dependencies parameter. This approach is useful for side-effects (for example, authentication guards, rate limiting or request logging) where the return data is not required in the path operation.

Replace the previous code in fastapi-deps/function_deps.py with this:

#dep function to pass in decorator
def user_dep(name: str, password: str):
    for u in users:
        if u["name"] == name and u["password"] == password:
            return
    raise HTTPException(status_code=401, detail="Invalid username or password")

#path function
@app.get("/users/{user}", dependencies=[Depends(user_dep)])
def get_user() -> dict:
    return {"message" : "Access granted!"}

This decorator-based dependency acts as a pre-check before the endpoint executes. It validates credentials without passing any values to the path function. On authentication failure, FastAPI raises an HTTPException and prevents the path operation from running.

If you test this using a valid name and password from the in-memory database, your output should look like this:

Router Level

Injecting dependencies at the router level allows multiple endpoints to share common logic without repeating the dependency in each route.

We'll use the same user_dep function but inject it at the router level. Add these imports to fastapi-deps/router_deps.py:

from fastapi import APIRouter, Depends

#import the dependency function
from function_deps import user_dep

Then, create an APIRouter instance, passing your dependency to the dependencies parameter. This makes the dependency run automatically for every route you define under this router.

In this example, user_dep executes before get_user() and any other endpoints you add to the router, eliminating the need to declare it on each route.

router = APIRouter(prefix="/users", dependencies=[Depends(user_dep)])

#define the routes with or without additional dependencies
@router.get("/{user}")
def get_user() -> dict:
    return {"message" : "Access granted!"}

In your main application file (app.py), import the router and register it with your FastAPI application using include_router(). This makes all routes defined in the router accessible through your application.

from fastapi import FastAPI
import uvicorn
from router_deps import router as user_router

app = FastAPI()
app.include_router(user_router)

if __name__ == "__main__":
    uvicorn.run("app:app", reload=True)

Start your server and test the route using a valid name–password pair from the users list, then try a mismatched one. You should get a 200 status for the correct credentials and 401 for invalid ones.

Application Level

Application-level dependencies (also called global dependencies) are defined when instantiating the FastAPI app and apply to every route in your application. Unlike router-level dependencies that target specific endpoint groups, app-level dependencies extend across the entire application. Any dependency injected into the FastAPI app object will automatically execute for all path functions.

Let's inject a simple logging dependency alongside the user authentication dependency we've used throughout this article.

Update fastapi-deps/app.py with this code:

from fastapi import FastAPI, Depends
import uvicorn
from function_deps import user_dep
from router_deps import router as user_router
from datetime import datetime

#Basic logging dependency
def log_request():
    print(f"[{datetime.now()}] Request received.")

app = FastAPI(dependencies=[Depends(log_request), Depends(user_dep)])
app.include_router(user_router)

@app.get("/home")
def get_main():
    return "Welcome back!!!"


if __name__ == "__main__":
    uvicorn.run("app:app", reload=True)

When you send a request to any endpoint within this application, log_request acknowledges it and outputs what time the request was made. Since we aren’t sending the logs to any database in particular, it will just print to the terminal (or console) like so:

Request the endpoint with valid credentials using your browser, cURL, Postman, or the Swagger UI. You should get this response:

💡

Although the same authentication and logging logic apply to all registered routers, the specific message users see depends on what you program into each router.

Common Use Cases for Dependency Injection

Dependency injection solves several common challenges in API development. Here are the most frequent use cases where you'll apply this pattern.

Database Connections: Reusing connection logic across multiple endpoints prevents connection leaks, and ensures each request has an isolated session.
Authentication & Authorization: Dependencies help validate tokens and verify user roles across protected routes.
Logging & Monitoring: A logging dependency can automatically record each request to your monitoring system or database. It is beneficial for debugging and tracking API usage.
Rate Limiting: You can control request frequency and prevent API abuse by injecting rate-limiting logic in path functions.
Configuration & Settings: FastAPI’s dependency injection system simplifies configuration management by letting you inject settings such as API keys or environment variables wherever needed, keeping your code consistent.
Pagination & Filtering: Injecting common parameters like page_size and limit standardize data retrieval patterns across endpoints.

Conclusion

FastAPI's dependency injection system helps you manage shared logic and resources efficiently while adhering to DRY principles. However, knowing when to inject a dependency versus when to skip it is a skill that comes with practice.

Dependency injection isn't needed for simple, standalone logic. But for resources requiring lifecycle management, shared logic, or modularity, FastAPI's dependency injection system simplifies checks and app operations—with or without return values.

How to Write Unit Tests and E2E Tests for NestJS Applications

Gordan Tan — Wed, 16 Apr 2025 14:40:41 +0000

Recently, I have been writing unit tests and E2E tests for a NestJS project. This was my first time writing tests for a backend project, and I found the process different from my experience with frontend testing, making it challenging to begin.

After looking at some examples, I have gained a clearer understanding of how to approach testing. So I wrote an article to record and share my learning to help others who may be facing similar confusion.

In addition, I have put together a demo project with the relevant unit and E2E tests completed, which may be of interest. I’ve uploaded the code to Github here.

Prerequisites
Difference Between Unit Testing and E2E Testing
Writing Unit Tests
Writing E2E Tests
Whether to Write Tests
When Not to Write Tests?
Conclusion
Reference Materials

Prerequisites

Before diving into this tutorial, you should have:

Basic knowledge of TypeScript and Node.js
Familiarity with NestJS fundamentals
Understanding of RESTful APIs
MongoDB installed (as the example uses MongoDB)
Node.js and npm/yarn installed on your system
Basic understanding of testing concepts

You can find the complete code examples in the demo repository. You can clone it to follow along with the examples.

Difference Between Unit Testing and E2E Testing

Unit tests and E2E tests are methods of software testing, but they have different goals and scopes.

Unit testing involves checking and verifying the smallest testable unit within the software. A function or a method, for example, can be considered a unit. In unit testing, you provide expected outputs for various inputs of a function and validate the correctness of its operation. The goal of unit testing is to quickly identify bugs within the function, and they are easy to write and execute rapidly.

On the other hand, E2E tests often simulate real-world user scenarios to test the entire application. For instance, the frontend typically uses a browser or headless browser for testing, while the backend does so by simulating API calls.

Within a NestJS project, unit tests might assess a specific service or a method of a controller, such as verifying if the update method in the Users module correctly updates a user. An E2E test, however, may examine a complete user journey, from creating a new user to updating their password and eventually deleting the user, which involves multiple services and controllers.

How to Write Unit Tests

Writing unit tests for a utility function or method that doesn't involve interfaces is relatively straightforward. You only need to consider the various inputs and write the corresponding test code. But the situation becomes more complex once interfaces come into play. Let's use code as an example:

async validateUser(
  username: string,
  password: string,
): Promise {
  const entity = await this.usersService.findOne({ username });
  if (!entity) {
    throw new UnauthorizedException('User not found');
  }
  if (entity.lockUntil && entity.lockUntil > Date.now()) {
    const diffInSeconds = Math.round((entity.lockUntil - Date.now()) / 1000);
    let message = `The account is locked. Please try again in ${diffInSeconds} seconds.`;
    if (diffInSeconds > 60) {
      const diffInMinutes = Math.round(diffInSeconds / 60);
      message = `The account is locked. Please try again in ${diffInMinutes} minutes.`;
    }
    throw new UnauthorizedException(message);
  }
  const passwordMatch = bcrypt.compareSync(password, entity.password);
  if (!passwordMatch) {
    // $inc update to increase failedLoginAttempts
    const update = {
      $inc: { failedLoginAttempts: 1 },
    };
    // lock account when the third try is failed
    if (entity.failedLoginAttempts + 1 >= 3) {
      // $set update to lock the account for 5 minutes
      update['$set'] = { lockUntil: Date.now() + 5 * 60 * 1000 };
    }
    await this.usersService.update(entity._id, update);
    throw new UnauthorizedException('Invalid password');
  }
  // if validation is sucessful, then reset failedLoginAttempts and lockUntil
  if (
    entity.failedLoginAttempts > 0 ||
    (entity.lockUntil && entity.lockUntil > Date.now())
  ) {
    await this.usersService.update(entity._id, {
      $set: { failedLoginAttempts: 0, lockUntil: null },
    });
  }
  return { userId: entity._id, username } as UserAccountDto;
}

The code above is a method validateUser in the auth.service.ts file, primarily used to verify whether the username and password entered by the user during login are correct. It contains the following logic:

Check if the user exists based on username. If not, throw a 401 exception (a 404 exception is also feasible).
See if the user is locked out. If so, throw a 401 exception with a relevant message.
Encrypt the password and compare it with the password in the database. If it's incorrect, throw a 401 exception (three consecutive failed login attempts will lock the account for 5 minutes).
If the login is successful, clear any previously failed login attempt counts (if applicable) and return the user id and username to the next stage.

As you can see, the validateUser method includes four processing logics. So we need to write corresponding unit test code for these four points to ensure that the entire validateUser function is operating correctly.

The First Test Case

When we start writing unit tests, we encounter a problem: the findOne method needs to interact with the database, and it looks for corresponding users in the database through username. But if every unit test has to interact with the database, the testing will become very cumbersome. So we can mock fake data to achieve this.

For example, assume we have registered a user named woai3c. Then, during login, the user data can be retrieved in the validateUser method through const entity = await this.usersService.findOne({ username });. As long as this line of code can return the desired data, there is no problem, even without database interaction. We can achieve this through mock data.

Now, let's look at the relevant test code for the validateUser method:

import { Test } from '@nestjs/testing';
import { AuthService } from '@/modules/auth/auth.service';
import { UsersService } from '@/modules/users/users.service';
import { UnauthorizedException } from '@nestjs/common';
import { TEST_USER_NAME, TEST_USER_PASSWORD } from '@tests/constants';
describe('AuthService', () => {
  let authService: AuthService; // Use the actual AuthService type
  let usersService: Partial>;
  beforeEach(async () => {
    usersService = {
      findOne: jest.fn(),
    };
    const module = await Test.createTestingModule({
      providers: [        AuthService,
        {
          provide: UsersService,
          useValue: usersService,
        },
      ],
    }).compile();
    authService = module.get(AuthService);
  });
  describe('validateUser', () => {
    it('should throw an UnauthorizedException if user is not found', async () => {
      await expect(
        authService.validateUser(TEST_USER_NAME, TEST_USER_PASSWORD),
      ).rejects.toThrow(UnauthorizedException);
    });
    // other tests...
  });
});

We get the user data by calling the findOne method of usersService, so we need to mock the findOne method of usersService in the test code:

beforeEach(async () => {
    usersService = {
      findOne: jest.fn(), // mock findOne method
    };
    const module = await Test.createTestingModule({
      providers: [        AuthService, // real AuthService, because we are testing its methods
        {
          provide: UsersService, // use mock usersService instead of real usersService
          useValue: usersService,
        },
      ],
    }).compile();
    authService = module.get(AuthService);
  });

We use jest.fn() to return a function to replace the real usersService.findOne(). If usersService.findOne() is called now, there will be no return value, so the first unit test case will pass:

it('should throw an UnauthorizedException if user is not found', async () => {
  await expect(
    authService.validateUser(TEST_USER_NAME, TEST_USER_PASSWORD),
  ).rejects.toThrow(UnauthorizedException);
});

Since findOne in const entity = await this.usersService.findOne({ username }); of the validateUser method is a mocked fake function with no return value, the 2nd to 4th lines of code in the validateUser method could execute:

if (!entity) {
  throw new UnauthorizedException('User not found');
}

It throws a 401 error, which is as expected.

The Second Test Case

The second logic in the validateUser method is to determine if the user is locked, with the corresponding code as follows:

if (entity.lockUntil && entity.lockUntil > Date.now()) {
  const diffInSeconds = Math.round((entity.lockUntil - Date.now()) / 1000);
  let message = `The account is locked. Please try again in ${diffInSeconds} seconds.`;
  if (diffInSeconds > 60) {
    const diffInMinutes = Math.round(diffInSeconds / 60);
    message = `The account is locked. Please try again in ${diffInMinutes} minutes.`;
  }
  throw new UnauthorizedException(message);
}

As you can see, we can determine that the current account is locked if there is a lock time lockUntil in the user data and the lock end time is greater than the current time. So we need to mock a user data with the lockUntil field:

it('should throw an UnauthorizedException if the account is locked', async () => {
  const lockedUser = {
    _id: TEST_USER_ID,
    username: TEST_USER_NAME,
    password: TEST_USER_PASSWORD,
    lockUntil: Date.now() + 1000 * 60 * 5, // The account is locked for 5 minutes
  };
  usersService.findOne.mockResolvedValueOnce(lockedUser);
  await expect(
    authService.validateUser(TEST_USER_NAME, TEST_USER_PASSWORD),
  ).rejects.toThrow(UnauthorizedException);
});

In the test code above, an object lockedUser is first defined, which contains the lockUntil field we need. Then, it is used as the return value for findOne, achieved by usersService.findOne.mockResolvedValueOnce(lockedUser);. Thus, when the validateUser method is executed, the user data within it is the mocked data, successfully allowing the second test case to pass.

Unit Test Coverage

Unit test coverage (Code Coverage) is a metric used to describe how much of the application code has been covered or tested by unit tests. It is typically expressed as a percentage, indicating how much of all possible code paths have been covered by test cases.

Unit test coverage usually includes the following types:

Line Coverage: How many lines of code are covered by the tests.
Function Coverage: How many functions or methods are covered by the tests.
Branch Coverage: How many code branches are covered by the tests (for example, if/else statements).
Statement Coverage: How many statements in the code are covered by the tests.

Unit test coverage is an important metric to measure the quality of unit tests, but it is not the only metric. A high coverage rate can help to detect errors in the code, but it does not guarantee the quality of the code. A low coverage rate may mean that there is untested code, potentially with undetected errors.

The image below shows the unit test coverage results for a demo project:

For files like services and controllers, it's generally better to have a higher unit test coverage, while for files like modules there's no need to write unit tests – nor is it possible to write them, as it's meaningless.

This is because NestJS modules are primarily configuration files that define the structure of your application by connecting controllers, services, and other components together. They don't contain actual business logic to test, but rather serve as wiring instructions for the dependency injection system. Testing modules would only verify that NestJS's core functionality works correctly, which is already tested by the NestJS team themselves.

The above image represents the overall metrics for the entire unit test coverage. If you want to view the test coverage for a specific function, you can open the coverage/lcov-report/index.html file in the project's root directory. For example, I want to see the specific test situation for the validateUser method:

As you can see, the original unit test coverage for the validateUser method is not 100%, and there are still two lines of code that were not executed. But it doesn't matter much, as it does not affect the four key processing nodes, and we shouldn’t pursue high test coverage unidimensionally.

How to Write E2E Tests

In the unit tests section, you learned how to write unit tests for each feature of the validateUser() function, using mocked data to ensure that each feature could be tested.

In e2E testing, we need to simulate real user scenarios, so connecting to a database for testing is necessary. So, the methods in the auth.service.ts module that we'll be testing all interact with the database.

The auth module primarily includes the following features:

Registration
Login
Token refresh
Reading user information
Changing password
Deleting users

E2E tests need to test these six features one by one, starting with registration and ending with deleting users. During testing, we can create a dedicated test user to conduct the tests and then delete this test user upon completion, so we don’t leave any unnecessary information in the test database.

beforeAll(async () => {
  const moduleFixture: TestingModule = await Test.createTestingModule({
    imports: [AppModule],
  }).compile()
  app = moduleFixture.createNestApplication()
  await app.init()
  // Perform a login to obtain a token
  const response = await request(app.getHttpServer())
    .post('/auth/register')
    .send({ username: TEST_USER_NAME, password: TEST_USER_PASSWORD })
    .expect(201)
  accessToken = response.body.access_token
  refreshToken = response.body.refresh_token
})
afterAll(async () => {
  await request(app.getHttpServer())
    .delete('/auth/delete-user')
    .set('Authorization', `Bearer ${accessToken}`)
    .expect(200)
  await app.close()
})

The beforeAll hook function runs before all tests begin, so we can register a test account TEST_USER_NAME here. The afterAll hook function runs after all tests end, so it's suitable to delete the test account TEST_USER_NAME here. It also conveniently tests the registration and deletion functions.

In the previous section's unit tests, we wrote relevant unit tests around the validateUser method. Actually, this method is executed during login to validate if the user's account and password are correct. So this E2E test will also use the login process to demonstrate how to compose the E2E test cases.

The entire login test process includes five small tests:

describe('login', () => {
    it('/auth/login (POST)', () => {
      // ...
    })
    it('/auth/login (POST) with user not found', () => {
      // ...
    })
    it('/auth/login (POST) without username or password', async () => {
      // ...
    })
    it('/auth/login (POST) with invalid password', () => {
      // ...
    })
    it('/auth/login (POST) account lock after multiple failed attempts', async () => {
      // ...
    })
  })

These five tests are as follows:

Successful login, return 200
If the user does not exist, throw a 401 exception
If password or username is not provided, throw a 400 exception
Login with the wrong password, throw a 401 exception
If the account is locked, throw a 401 exception

Now let's start writing the E2E tests:

// login success
it('/auth/login (POST)', () => {
  return request(app.getHttpServer())
    .post('/auth/login')
    .send({ username: TEST_USER_NAME, password: TEST_USER_PASSWORD })
    .expect(200)
})
// if user not found, should throw 401 exception
it('/auth/login (POST) with user not found', () => {
  return request(app.getHttpServer())
    .post('/auth/login')
    .send({ username: TEST_USER_NAME2, password: TEST_USER_PASSWORD })
    .expect(401) // Expect an unauthorized error
})

Writing E2E test code is relatively straightforward: you simply call the interface and then verify the result. For example, for the successful login test, we just need to verify that the returned result is 200.

The first four tests are quite simple. Now let's look at a slightly more complicated E2E test, which is to verify whether an account is locked.

it('/auth/login (POST) account lock after multiple failed attempts', async () => {
  const moduleFixture: TestingModule = await Test.createTestingModule({
    imports: [AppModule],
  }).compile()
  const app = moduleFixture.createNestApplication()
  await app.init()
  const registerResponse = await request(app.getHttpServer())
    .post('/auth/register')
    .send({ username: TEST_USER_NAME2, password: TEST_USER_PASSWORD })
  const accessToken = registerResponse.body.access_token
  const maxLoginAttempts = 3 // lock user when the third try is failed
  for (let i = 0; i < maxLoginAttempts; i++) {
    await request(app.getHttpServer())
      .post('/auth/login')
      .send({ username: TEST_USER_NAME2, password: 'InvalidPassword' })
  }
  // The account is locked after the third failed login attempt
  await request(app.getHttpServer())
    .post('/auth/login')
    .send({ username: TEST_USER_NAME2, password: TEST_USER_PASSWORD })
    .then((res) => {
      expect(res.body.message).toContain(
        'The account is locked. Please try again in 5 minutes.',
      )
    })
  await request(app.getHttpServer())
    .delete('/auth/delete-user')
    .set('Authorization', `Bearer ${accessToken}`)
  await app.close()
})

When a user fails to log in three times in a row, the account will be locked. So in this test, we cannot use the test account TEST_USER_NAME, because if the test is successful, this account will be locked and unable to continue the following tests. We need to register another new user TEST_USER_NAME2 specifically to test account locking, and delete this user after the test is successful.

So, as you can see, the code for this E2E test is quite substantial, requiring a lot of setup and teardown work, but the actual test code is just these few lines:

// login three times
for (let i = 0; i < maxLoginAttempts; i++) {
  await request(app.getHttpServer())
    .post('/auth/login')
    .send({ username: TEST_USER_NAME2, password: 'InvalidPassword' })
}
// test if the account is locked
await request(app.getHttpServer())
  .post('/auth/login')
  .send({ username: TEST_USER_NAME2, password: TEST_USER_PASSWORD })
  .then((res) => {
    expect(res.body.message).toContain(
      'The account is locked. Please try again in 5 minutes.',
    )
  })

Writing E2E test code is relatively simple. You don't need to consider mock data or test coverage. It's sufficient if the entire system process runs as expected.

When to Write Tests

If possible, I generally recommend writing tests. Doing so can enhance the robustness, maintainability, and development efficiency of the system. Here are some key reasons why writing tests is useful:

Enhancing System Robustness

When writing code, you usually focus on the program flow under normal inputs to ensure the core functionality works properly. But you might often overlook some edge cases, such as abnormal inputs.

Writing tests changes this, as it forces you to consider how to handle these cases and respond appropriately, thus preventing crashes. So we can say that writing tests indirectly improves system robustness.

Enhancing Maintainability

Taking over a new project that includes comprehensive tests can be very pleasant. They act as a guide, helping you quickly understand the various functionalities. Just by looking at the test code, you can easily grasp the expected behavior and boundary conditions of each function without having to go through each line of the function's code.

Enhancing Development Efficiency

Imagine a project that hasn't been updated for a while suddenly receives new requirements. After making changes, you might worry about introducing bugs. Without tests, you would need to manually test the entire project again — wasting time and being inefficient.

With complete tests, a single command can tell you whether the code changes have impacted existing functionalities. Even if there are errors, you can quickly locate them and address them.

When Not to Write Tests

For short-term projects and projects with very fast requirement iterations, it's not recommended to write tests.

For example, a project built for an event that will be useless after the event ends doesn't need tests. Also, for projects that undergo very fast requirement iterations, writing tests could enhance development efficiency, but that's based on the premise that function iterations are slow. If the function you just completed changes in a day or two, the related test code must be rewritten.

So, it's better not to write tests at all in these cases and rely on the testing team instead – because writing tests is very time-consuming and not worth the effort for these situations.

Conclusion

I’ve explained in detail how to write unit tests and E2E tests for NestJS projects. But I still want to reiterate the importance of testing. It can enhance the robustness, maintainability, and development efficiency of the system.

If you don't have the opportunity to write tests, I suggest you start a practice project yourself or participate in some open-source projects and contribute code to them. Open-source projects generally have stricter code requirements. Contributing code may require you to write new test cases or modify existing ones.

Reference Materials

NestJS: A framework for building efficient, scalable Node.js server-side applications.
MongoDB: A NoSQL database used for data storage.
Jest: A testing framework for JavaScript and TypeScript.
Supertest: A library for testing HTTP servers.

What is Backend as a Service (BaaS)? A Beginner's Guide

Ijeoma Igboagu — Mon, 17 Feb 2025 10:21:26 +0000

Building an authentication system can be complex, often requiring a server to store user data. Sometimes, you need a faster, easier solution.

For those new to development or without technical expertise, managing servers, databases, and user logins can be overwhelming. This is where Backend as a Service (BaaS) helps.

BaaS platforms provide ready-made backend solutions, making app development simpler. Whether you're a developer or someone with no coding experience, BaaS allows you to focus on your app’s features instead of handling backend complexities.

This article will explore BaaS, its features, pricing, and popular BaaS tools.

What is Backend as a Service (BaaS)?
Key Features of Baas
Why use Backend as a Service (BaaS)?
When to Use Backend as a Service (BaaS)
What are the Popular Backend as a Service (BaaS) Tools?
How to Get Started with BaaS (Quick Example)
Conclusion

What is Backend as a Service (BaaS)?

BaaS is a cloud platform that provides pre-built backend infrastructure and services. It eliminates the need for developers to manage servers, databases, and other backend tasks.

Source: https://www.cloudflare.com

Key Features of BaaS

Here are some features of BaaS:

BaaS makes it easy to create and manage user accounts and logins without much coding.
It lets you store and manage data, eliminating the need to set up a database from scratch.
BaaS comes with tools (APIs and SDKs) that help connect your application to the backend easily.
Many BaaS platforms let you see updates in real time, so your application can show live data to users.
BaaS offers space in the cloud to store files and images, making it easy to handle user uploads.
You don’t need to worry about managing servers—BaaS takes care of that for you, so you can focus on building your application.
Some BaaS platforms allow you to send notifications to users about updates or messages.
BaaS often provides tools to track user interactions, helping you understand what works and what doesn’t.
It also makes it easy to integrate with other services like payment systems and social media with minimal effort.
As your app grows, BaaS scales with it, handling more users and data seamlessly.

Why use Backend as a Service (BaaS)?

There are several key reasons why BaaS is an excellent choice for developers:

Pre-built features reduce development time, allowing you to focus on design and functionality instead of backend issues.
With BaaS, you don’t have to worry about servers, scaling, or security updates—the provider takes care of it all.
Most BaaS platforms offer essential features like user authentication, data storage, and real-time updates, helping you build your app without starting from scratch.
As your application gets more users, BaaS can handle it! These services adjust to support more users and data, so you can focus on growing your app.
BaaS handles the infrastructure so you don’t need to spend time or money on the backend. This allows you to focus on design and creating user experiences that add value to your users.

When to Use Backend as a Service (BaaS)

BaaS is perfect for building an app in a short amount of time without managing the backend. Here are the scenarios when BaaS makes sense:

BaaS handles your app’s backend, letting you focus on its features. For example, when building a to-do list app, BaaS makes it easy to manage user logins and task data without setting up servers from scratch.
For small teams or solo devs, BaaS handles the backend. You do not need extra resources.
If you're launching a startup, Baas lets you release a Minimum Viable Product (MVP) without delay. It helps you speed up development and cut costs.
If your app needs features like user authentication, data storage, or push notifications, BaaS provides them out of the box. For example, when building a social media app, BaaS simplifies user logins and file uploads, saving you from starting from scratch.
BaaS automatically scales to support more users, allowing you to focus on improving your app. For example, a small multiplayer game can start with a few players, and as it grows, BaaS will seamlessly handle thousands without extra backend effort.

What are the Popular Backend as a Service (BaaS) Tools?

If you're looking to explore BaaS, here are popular platforms you can use:

Clerk

Clerk software focuses on user management. It offers tools for authentication, user profiles, and permissions management. It’s great for developers who need simple user management in their apps.

Features of clerk

Clerk provides:

Multi-factor authentication (MFA)
Passwordless login (magic links, OTPs)
Social & OAuth login (Google, GitHub, and so on)
Enterprise SSO (SAML, OAuth)
Biometric login (Face ID, Touch ID)

It also handles:

User profiles & custom attributes
Roles & permissions
Teams & organizations
Session management

For security, it offers:

Token-based authentication (JWT)
Rate limiting
Audit logs
GDPR & SOC 2 compliance

For developers, it comes with:

Prebuilt UI components
SDKs for React, Next.js, Vue, and so on
Custom email & SMS templates

To learn more, click here: Clerk

Pricing

Clerk offers a Free Plan that includes up to 10,000 Monthly Active Users (MAUs) at no cost. For more advanced features, the Pro Plan is available at $25 per month, which also includes the first 10,000 MAUs.

For detailed and up-to-date information on Clerk's pricing plans, please visit their official pricing page:

Firebase

Firebase is a Google-backed BaaS platform. It is known for its real-time databases, authentication, and cloud storage. It also has easy-to-use tools for web and mobile apps.

Features of Firebase

Firebase provides:

Backend Services

Firestore & Realtime Database
Cloud Storage
Serverless Functions
Web Hosting

Authentication

Email & password login
Social logins (Google, Facebook, and so on)
Phone authentication
Anonymous sign-in

Analytics & Monitoring

Google Analytics
Crash tracking (Crashlytics)
Performance monitoring
A/B testing

Engagement Tools

Push notifications
Remote app updates
In-app messaging

Machine Learning

Text recognition
Image labelling

To learn more, click here: Firebase

Pricing plan

Firebase offers a Spark Plan (free tier) and a Blaze Plan (pay-as-you-go). The Spark Plan provides limited free usage, while the Blaze Plan charges based on your actual usage. For detailed and up-to-date information on Firebase's pricing plans, please visit their official pricing page.

Convex

Convex is a serverless BaaS platform. It provides real-time data sync and scalable backend services. The design simplifies serverless computing for developers.

Convex Features

Database – Real-time data storage
Serverless Functions – Run backend logic without managing servers
Authentication – Built-in user auth & access control
Caching – Faster data retrieval
Webhooks & Crons – Automate tasks & trigger events

To learn more, click here: Convex

Pricing

Free Plan – Limited resources for small projects
Pro Plan – Pay-as-you-go based on usage

Check out full details for convex pricing

8base

A low-code platform that allows developers to build serverless apps with minimal setup. It provides database management, authentication, and API development tools.

8base Features

Backend Builder – Manage your database easily.
Serverless Functions – Run custom backend logic.
GraphQL API – Auto-generated API for your data.
Authentication – Built-in user login & access control.
File Management – Store and manage files.

To learn more, click here: 8base

Pricing

Free Plan – $0/month (1 developer, basic features).
Developer Plan – $25/month per developer.
Professional Plan – $150/month (5 developers).
Custom Plan – Contact 8base for enterprise solutions.

Check out full pricing details here: 8base Pricing

Backendless

Backendless is a no-code platform that makes app development easy. It provides APIs, data storage, user management, and real-time updates in one place.

Features

UI Builder: Design your app's front end visually without coding.
Real-Time Database: Store and sync data in real-time across clients.
User Authentication: Manage user sign-ups, logins, and roles.
Cloud Code: Implement custom server-side logic without managing servers.
Push Notifications: Send real-time alerts to users on various devices.

To learn more, click here: Backendless

Pricing

Backendless offers several plans to suit different needs:

Free Plan: Ideal for small projects or learning purposes.
Scale Fixed Plan: Provides predictable monthly billing with set resource limits.
Scale Variable Plan: Offers flexibility with usage-based billing, scaling as your app grows.
Backendless Pro: A self-hosted solution for enterprises requiring unlimited scalability and control.

For more details on Backendless's pricing plans, please visit their official pricing plan page.

Appwrite

Appwrite is an open-source BaaS that provides databases, authentication, file storage, real-time updates, serverless functions, and API management. It supports multiple platforms and offers built-in security and scalability for modern apps.

Features

Authentication: Secure user login with over 30 methods, including email/password, OAuth, and magic URLs.
Database: Scalable storage with advanced permissions, custom data validation, and support for relationships.
Functions: Deploy serverless functions in over 13 languages, with automatic GitHub deployment and custom domain support.
Storage: Manage and serve files with built-in security and privacy features.
Real-Time: Subscribe to database events for instant updates.

To learn more, click here: Appwrite

Pricing

Free – $0/month (5GB bandwidth, 2GB storage, 750K function runs).
Pro – Starts at $15/month (more storage, bandwidth, & features).
Scale – Starts at $599/month (for large-scale projects).

For more details on the pricing plan check their official pricing page.

Nhost

Nhost is a full backend platform with a GraphQL API, database, authentication, and storage. It’s easy to set up and great for modern app development.

Nhost Features

Authentication – Secure login with email, OAuth, and so on.
Database – Scalable storage with permissions.
Serverless Functions – Run backend code without servers.
Storage – Secure file hosting.
Real-Time – Instant updates on data changes.

To learn more, click here: Nhost.

Pricing

Free – $0/month (basic features for small projects).
Pro – $25/month (more resources & support).
Dedicated Compute – $50/month per vCPU/2GB RAM (for scaling apps).

Check out full details here: Nhost Pricing

Back4apps

Back4App is an open-source BaaS that simplifies backend development. It provides a complete infrastructure for building, hosting, and managing scalable apps. With built-in server-side features, developers can focus on coding without managing servers or databases.

Back4App Features

Database – Manage data with APIs & a visual editor
Authentication – Secure user login & roles
Real-Time – Instant data updates
Push Notifications – Send alerts to users.
Cloud Functions – Run custom backend code.

To learn more, click here: Back4apps.

Pricing

Free – 25K requests, 250MB storage, 1GB transfer/month.
MVP Plan – For launching small apps.
Dedicated Plan – For production apps with more resources.

The MVP Plan in Back4App refers to a Minimum Viable Product (MVP) Plan. It is designed for startups and developers who are launching a small app with essential backend services. This plan provides enough resources to test and validate an idea before scaling up.

While Dedicated Plan in Back4App provides a private server with dedicated resources for apps that need better performance, security, and scalability. It is ideal for production apps with high traffic or specific infrastructure requirements.

Check out full details here: Back4App Pricing.

AWS Amplify

AWS Amplify is a development platform from Amazon Web Services (AWS). It simplifies building and deploying web and mobile apps. It offers tools and services for developers. They can integrate scalable backends, manage frontends, and add features like authentication, storage, and APIs.

AWS Amplify Features

Authentication – Secure login with email, social sign-in, and multi-factor authentication
Database & API – Build real-time APIs with AWS databases
Storage – Manage files and media with Amazon S3
Hosting – Deploy full-stack apps with continuous deployment

To learn more, click here: Aws Amplify

Pricing

Free Tier (First 12 months)
- 1,000 build minutes/month
- 5GB storage
- 15GB bandwidth
- 500K API requests
Pay-As-You-Go (After Free Tier)
- Build & Deploy – $0.01 per build minute
- Storage – $0.023 per GB/month
- Bandwidth – $0.15 per GB served
- API Requests – $0.30 per 1M requests

Full details here: AWS Amplify Pricing

Supabase

Supabase is an open-source alternative to Firebase. It uses PostgreSQL for its database. It has built-in features like authentication, APIs, and real-time subscriptions.

Supabase Features

Database – PostgreSQL with full SQL support.
Authentication – Secure login with email, password, and social logins.
Storage – Store and serve files easily.
Real-Time – Get instant updates when data changes.
Edge Functions – Run serverless backend logic.

To learn more, click here: Supabase.

Pricing

Free – Great for small projects i.e. projects for learning, and experimentation.
Pro – Starts at $25/month (includes $10 compute credits).
Team – Starts at $599/month (for advanced features & support).

Full details here: Supabase Pricing

How to Get Started with BaaS (Quick Example)

Let’s go through a quick example to get started. In this tutorial, I’ll use Firebase as an example.

Go to the Firebase website and sign up using your Google account.
After signing in, create a new Firebase project by following the on-screen instructions.
Go to "Authentication" and enable a sign-in method, like email/password or Google login
In "Firestore Database," create a new database for your app's data.
Install Firebase SDK in your project and integrate authentication, databases, and other Firebase services into your app.

For more detailed instructions on setting up Firebase, check out this article: How to Authenticate Your React App Using Firebase where I explain each step in depth.

Conclusion

Backend as a Service (BaaS) is ideal for developers. It provides an efficient and cost-effective way to handle backend development tasks. BaaS can speed up your development. It lets you avoid server management. You can then focus on building better apps.

If you're new to backend development, check out the BaaS tools in this article. They can simplify your workflow. Try out BaaS today and take your development to the next level!

Have you tried using BaaS for your applications? Share your experiences!

If you found this article helpful, share it with others who may find it interesting.

Stay updated with my projects by following me on Twitter, LinkedIn and GitHub.

Thank you for reading.

How to Become a Web Developer – a Beginner's Guide

Kunal Nalawade — Sun, 22 Dec 2024 21:29:59 +0000

Are you considering a career in web development? If so, then you are making an excellent choice. Web Development is one of the most in-demand skills in the market in 2024. With over 5.038 billion Internet users, web development has a promising future.

In this article, I am going to show you the essentials of getting started with web development. We’ll explore key tech stacks, beginner-friendly project ideas, helpful resources, and some additional tips.

Two years ago, I wrote an article on this topic. Since the demand for web development still remains very high, I am excited to re-visit it with a more detailed guide. So, stick around until the end.

What is a Website?
1. Frontend vs Backend
Frontend Development
Backend Development
Git and GitHub
Build a portfolio of projects
Deployment and Hosting Services
Additional Tips

What is a Website?

A web page is a document displayed in a web browser (like Chrome, Firefox, and so on). It consists of text, images, and other interactive elements. And a website is a collection of web pages that are connected to each other via links.

A website runs on a remote computer referred to as a web server, and is accessed via the internet. Some examples of well-known websites are Wikipedia, Amazon, and YouTube.

Note: When I mention web apps or web applications in this article, I mean the same thing as websites.

A web app has two components, the frontend and the backend. Let’s understand the difference between them.

Frontend vs Backend of a Website

The frontend is the user interface (UI) of the website, which is basically what the user sees on their screen.

The backend refers to the server where the main logic of the website is located. It also includes the database, where all the application’s data is stored.

The frontend and the backend communicate by exchanging data. Let’s take an example of a social media app like Instagram.

When you upload a post, the UI sends the post data to the backend, which processes the data and adds it to the database. Then the next time, when you load the site/app, it fetches all your posts from the backend and displays them on the screen.

In the next two sections, I will show how you can start with frontend and backend development.

Frontend Development

As I mentioned above, frontend development is mainly concerned with the UI – that is, the appearance of the website. To get started with frontend development, you’ll need to learn the following three essentials tools:

HTML

HTML (HyperText Markup Language) is used to write web pages that are displayed by the browser. It defines the structure and content of a web page, making it the backbone of every website.

The content of a web page includes elements such as headings, paragraphs, links, images, lists, and so on. HTML creates and structures all these elements through the use of HTML tags. The browser, in turn, interprets this HTML code and renders it on your screen.

freeCodeCamp’s Responsive Web Design certification in their free curriculum starts off by teaching you HTML basics. You’ll even build your own photo app. So that’s a good place to start and dig deep into HTML.

If you want additional practice, w3schools.com is a helpful resource for beginners as well. It offers clear and step-by-step tutorials for each concept. They also provide an interactive editor for you to practice using HTML tags and see the output web page (as does freeCodeCamp).

Focus on the following areas:

Creating a simple web page
Using HTML tags to render content
Creating Forms

CSS

While HTML defines the structure of the web page, it is not enough as it just creates a skeletal UI of a web page. It just defines what elements are present on the page, and not how they look.

CSS (Cascading Style Sheets) is used to add visual appeal to the web page. It transforms a simple, plain web page into a properly designed user interface.

Here’s how a website looks with plain HTML:

And this is what it looks like when you add CSS:

Much better, right? It actually looks like a proper web page, as compared to the skeletal UI before. This is referred to as “styling” a web page.

CSS styling includes the following:

Colours, fonts, and element backgrounds
Organising content in various layouts (grid, flex box, and so on)
Spacing, that is margins and paddings
Transitions and Animations (Advanced stuff)

As you continue on in the freeCodeCamp curriculum, you’ll learn CSS as well – so that’s a great way to go.

You can also refer to w3schools.com for CSS Tutorials. Play around with each CSS property in their interactive editors.

Note: HTML and CSS are NOT programming languages.

JavaScript

HTML and CSS are only able to create static websites – that is, you cannot interact with any of the elements on a web page created with just HTML and CSS. The website does not update or respond to any user interactions like button clicks or dropdown selections.

JavaScript (JS) is a programming language that makes a website dynamic and interactive. It adds the following functionalities to a website:

Handling user interactions like clicks, hovers, keyboard presses, form filling, and so on.
Updating content dynamically on a web page
Handling form validations and submissions
Interacting with the backend servers

JavaScript has many more capabilities that make your website functional and engaging for end users. As you start your JS learning journey, building strong foundation of the concepts is key. Initially, focus on the following areas:

Basic JavaScript syntax
JS Functions
Interacting with DOM (Document Object Model)
Event Handling
JS Objects and Arrays
Asynchronous JavaScript

Refer to the following resources for JavaScript:

freeCodeCamp for a free, in-depth, JS curriculum

w3schools.com for basic JavaScript tutorial
JavaScript Interview Prep Handbook for important JavaScript concepts.

There are a lot of other resources for JavaScript, but I won’t overwhelm you with too many. These two should be enough to get started.

Once you are familiar with HTML, CSS, and JavaScript, you’ll know how to create a simple web page. Keep practicing by building different web pages, such as to-do lists and forms, and try implementing CRUD (Create, Read, Update, Delete) features.

Learn Frontend Frameworks and Libraries

As your website grows, the JavaScript code becomes more and more complex and harder to maintain. This slows down the development process and makes it really challenging for developers.

To address these issues, you can use frameworks and libraries. Frameworks provide a more structured way to build web apps, encouraging modular design and reusability.

By using frameworks, you can focus on building actual features rather than handling the complexities of JavaScript code, which helps you speed up the development process. So I suggest picking up one JS framework/library to develop bigger projects.

React.js – a good option

React JS is a JavaScript library that makes it easy to create dynamic and interactive web pages. It divides your code into components, making it easy to read and maintain. This reduces code complexity and allows reusability.

React is my personal suggestion since it has a gentler learning curve compared to other frameworks. It is in very high demand among frontend roles, as many web apps are built in React.

To start learning React, your best resource is the React Docs. They are very detailed, and include interactive code editors to play around with.

The freeCodeCamp YouTube channel also has some helpful React courses, like this one from Bob Ziroll, and the Frontend Development Libraries certification has a React section as well.

Apart from React, there are other JavaScript frameworks like Angular, Vue, and the jQuery library. These remain popular as well, and depending on which tools are in demand in your area, you can focus on the one that will meet your needs the best.

Note: Make sure you learn all the basic JavaScript concepts and understand them fully before jumping into any framework.

Learn Responsive Design

Before moving on, let’s talk about a fundamental practice in web development.

Responsive design refers to an approach where your design adjusts to fit screens of all sizes, ranging from desktops to tablets and mobiles. A good responsive design drastically reduces the need to write separate code for different screen sizes.

Here’s an interesting fact: mobile phones make up two-thirds of web usage in the entire world. So, to ensure a good user experience, you need to make the website look good on mobile phones.

Learn more about responsive design in this simple guide, and read more about some best practices here.

And here are some other resources that can help you on your frontend journey:

MDN Docs
WebDevSimplified - YouTube Channel

Backend Development

Backend Development involves building the server-side of web applications. The server-side hosts the business logic of a website, that powers everything behind the scenes. It is also responsible for managing databases and ensuring smooth flow of data between server and the UI.

To dive into backend development, you need to learn a programming language first.

Why should you learn a programming language?

Learning a programming language equips you with the foundations to build these server-side applications. Think of a language as a way to tell the server what you want it to do.

A programming language serves as a tool to solve problems and create robust, working applications. These languages have various capabilities for handling tasks like storing and managing data, communicating with the front end, and ensuring application security.

Learning a programming language isn’t just about learning the syntax and writing code. It’s about understanding how to create systems that drive a successful website. So, getting familiar with a programming language is a key part of backend development.

There are a number of programming languages out there, each with its own features. Let’s understand a few options:

Python

Python is one of the preferred choices for backend development because of its simplicity. It has a concise and readable syntax making it very popular. It offers good features for database connections and setting up web servers. Python also has libraries for data science and machine learning.

Python is has a lot of tutorials and good community support, making it easy to get started with. It is beginner friendly, fun to learn and has a high demand.

Refer to the following resources for learning Python:

Ultimate Python Beginner’s Course on freeCodeCamp’s YouTube channel
GeeksforGeeks Python Tutorial
Python Tutorial for Beginners on YouTube (Hindi)
Machine Learning with Python – freeCodeCamp certification

Golang

Golang (Go) is increasing in popularity because of its simplicity and efficiency. Go code executes quickly and efficiently, making it a good option for high performance needs. This also leads to faster development time. Go also has excellent support for concurrency, which leads to efficient processing.

Go is beginner-friendly and has a clean and concise syntax, making it easy to read and maintain. It also has an extensive standard library offering a many built-in functions and tools, so it’s easy to set up a project without much hassle.

Go is growing in popularity because of its efficiency, and many companies are adopting Go for their projects. This has lead to an increasing demand for Golang developers and it’s expected to go up.

Go offers plenty of resources and a growing community for beginners. To get started with Go, refer to the following resources:

Tour of Go – Interactive Learning with basic Golang concepts
Golang Handbook from Flavio Copes
Go Docs – Very detailed

Java

Java is an Object-Oriented Programming (OOP) language, widely used for backend development. Java is known for its security and robustness, making it a preferred choice for applications that require high reliability such as financial and healthcare systems. Java also offers a great support for concurrency.

Java is a good option for beginners as it has extensive resources and a large developer community. This includes plenty of tutorials and detailed documentation to make life easier for beginners as well as experienced developers.

Java has been around for a while, and many existing systems and enterprise applications currently run on Java. So, there is a huge demand for Java developers among big enterprises.

Lastly, the concepts that you learn while coding in Java stick with you and make you a better developer, even if you switch languages going forward.

The following resources can help you get started with Java:

JavaScript

We already know what JavaScript offers to frontend, but it can also be used for backend development through NodeJS.

NodeJS is a run time environment that allows you to run JS code on the server side. This makes it possible to use JavaScript for both the frontend and the backend.

NodeJS follows an event-driven architecture and asynchronous programming, which enables it to handle multiple tasks without stopping execution for a single one (non-blocking I/O). Node is single threaded, so instead of creating multiple threads to handle tasks, it executes them one by one asynchronously by queuing tasks.

Node also follows a modular architecture, meaning you can break your application into smaller, manageable components. It also includes NPM (Node Package Manager) which provides access to thousands of open-source libraries to add functionality like routing, authentication, or database handling.

Why use Node?

This is a very good option if you are already familiar with JavaScript, as you don’t need to learn any other language.
Node is fast and efficient, making it easy if you want to set up a small server quickly.
Node also has a large ecosystem of libraries through NPM.

However, Node is not ideal for CPU-intensive tasks as they can block the main thread, since it’s single threaded.

How to choose a programming language?

With so many options available, it may feel confusing to choose the right one for you. Each language has its own capabilities and no language is objectively better than the other.

Python and Golang are very beginner-friendly with simple syntax. So, if you value a gentle learning curve, then these two are good options. Java is known for its reliability and robustness, with a lot of enterprise-level applications built using Java.

As for the job opportunities, there’s high demand for each of the above languages, so you can choose any one you want. The most important thing is to develop your problem solving skill and understand how reliable software is built.

The choice of language doesn’t really matter in the long run, since the core fundamentals remain the same. So, my advice is to pick any language, learn its syntax and core capabilities, and start solving problems. You can start with the following:

Learn Data Structures and Algorithms
Start solving problems on LeetCode
Learn language specific frameworks and develop projects (Upcoming section)

Backend Development Frameworks

Programming languages alone are not enough to create robust and secure applications. Frameworks, built on the capabilities of these languages, allow you to create these powerful applications. By providing additional functionalities like routing and database handling, they serve as a platform to put your coding skills to the test and also make the development process faster.

Depending on your language of choice, you can learn the following frameworks:

Django and Flask – Python Based Frameworks
Java Spring Boot
Gin – Golang framework (You can create a simple Golang application without using a framework)

Read more about them if you are interested.

Databases

A database is a structured collection of data and is a crucial part of backend development. It plays an important role in storing and managing the application’s data.

Databases are broadly categorised into two types:

Relational Databases use tables to store data and define relationships between those tables. Examples are MySQL, PostgreSQL, SQLite.
Non-Relational Databases (NoSQL) are designed to handle unstructured or semi-structured data and are often used for hierarchical or document-based data storage. Examples are MongoDB and Cassandra.
- MongoDB: A popular NoSQL database for flexible and scalable data storage.
- Cassandra: Suitable for handling large amounts of distributed data.

To begin with relational databases, learn SQL (Structured Query Language). SQL is used to write queries that perform various operations on the data, such as:

Creating tables and defining their structure.
Reading data using SELECT statements.
Updating existing records.
Deleting unnecessary or outdated data.

Refer to the following resources to learn SQL:

Full handbook on SQL
w3schools.com
GeeksforGeeks – Great resource to learn about database concepts

Once you are familiar with basic SQL syntax and are able to write queries, explore DBMS (DataBase Management System) concepts. These help you understand how databases are designed, managed, and optimised.

As a beginner, I recommend starting with relational databases because they provide a solid foundation in DBMS concepts involving tables and relationships between them. They are much more widely used among enterprises and learning their concepts can benefit you a lot.

These concepts might take some time to study, but don't worry about it. Take your time and keep working on the development stuff in parallel. You'll understand these concepts better as you gain more experience working with databases.

APIs

APIs (Application Programming Interfaces) are an essential part of backend development as they expose the backend logic to the outside world. APIs are a way for two different applications to communicate with each other. In the context of web development, the frontend interacts with the backend services through APIs.

When you build a web application, the frontend often needs to send and receive data from the backend. Let’s take an example of login functionality. When a user logs in, the frontend sends their credentials to the backend through an API call. The backend verifies this information and responds with the result.

To see these API calls, visit any website and open the Network Tab in Developer Tools. Interact with the website, or just reload the page, you’ll see the API calls being made as you use the website.

Read the following articles to understand more about APIs:

At this point, you know how to start with both frontend and backend development. If you've reached this stage, congratulations! You've completed most of the hard work. But there’s one more thing you need to learn before you start developing projects.

Git and GitHub

Git is a version control system that keeps track of changes in a software project. It allows multiple people to work on the project without directly interfering with each other’s work.

GitHub is a remote repository system based on Git. It is like social media, but for your code. GitHub encourages collaboration among developers and keeps track of everyone’s contributions.

GitHub lets you share your project code and view other developers' code, too. This facilitates increased collaboration and learning. I strongly recommend learning Git, especially at the beginning of your development journey.

To get started with Git and GitHub, refer to the following articles:

Build a Portfolio of Projects

Now, you are ready to start working on projects. A strong portfolio of projects is essential for showcasing your skills. It also helps you apply what you have learned so far and improves your problem-solving skills.

Consider the following project ideas:

Todo App
E-Commerce App
Personal Portfolio website
Weather App – Use a public API and create a simple UI
Expense Tracker

You can research more about these ideas and start with some basic features that come to mind. Build either the frontend, the backend, or both, depending on your goals. Share your projects on GitHub to increase their visibility.

Check out GeeksforGeeks for more project ideas.

Deployment and Hosting Platforms

Once you have developed a web project, you can choose to release it to the public. This means that your website will be available on the internet for the public to use. How exciting is that!

Let’s understand the above terms. Deployment refers to the process of uploading your application to a remote system or a server in order to make it live and available to users. Hosting is like renting a space on the Internet for storing your application code. It provides a space to keep your website’s data on the server and displays your website on the internet.

Deploying and Hosting an application mainly follows these steps:

The application code is written, tested locally, and optimised for production
The required configurations and secrets (passwords, API keys, and so on) are written as environment variables
The code is pushed to a version control system like GitHub or GitLab
The code is scanned for any security vulnerabilities and automated tests are run
Hosting platforms pull the code from these repositories and make it accessible on the internet.

Hosting services like Netlify, GitHub Pages, and Heroku offer free and paid services and are easy to use for beginners. Netlify only supports frontend applications while Heroku is good for backend and full stack applications with easy integration of databases. GitHub Pages lets you host right from your repository.

Releasing your website to the public is a great opportunity to showcase your work to recruiters and potential collaborators.

Additional Tips

Don't spend too much time on tutorials, as you might get stuck in "tutorial hell." Tutorials are important for understanding core concepts, but real learning happens when you work hands-on. So start building as soon as you can, even if it’s just small projects at first.
JavaScript might seem overwhelming at first, but start small and practice regularly. Don’t rush to learn multiple things at once, tackle one concept at a time and practice through code for better understanding.
Experiment with different frameworks initially to find one that works for you. Once you choose a framework, stick with it until you learn it well.
Ensure that your programming language concepts are clear before jumping into any framework.
If you feel that a programming language is not working for you, you can switch to a different one, the core fundamentals remain the same.
As a beginner, it's important to have a basic understanding of both the frontend and the backend. Later, you can choose to specialise in one or you can choose to focus on both, becoming a "full stack" developer.
You will face challenges at first, so don't get discouraged. Keep practicing, and you will get better over time.
If you are stuck on any issue, use Chat GPT, Google Search, forums and developer communities, and Stack Overflow as much as possible. I am always available if you need any help.
Lastly, stay updated with the latest trends and technologies in web development. Always look for new or improved ways to solve problems. Learning never stops!

Conclusion

Web Development is divided into two parts: frontend and backend development. The frontend is concerned with the website’s appearance while the backend focuses on the server-side logic and databases.

HTML, CSS, and JavaScript are the essentials in frontend development and form the backbone of a website. In backend development, learning a programming language like Python or Java is key. Both frontend and backend frameworks provide additional capabilities and make the development process faster.

Git is a must-have skill as it allows you to share your work and collaborate with other developers. Building a portfolio of projects and sharing on GitHub showcases your work and makes you a better developer. Lastly, utilise deployment platforms as they make your website available to the general public.

That’s it for today! I hope this article helps you start your web development journey. Let me know what you think. Your feedback is always appreciated!

Connect with me on Twitter for more updates and discussions. If you have any questions or clarifications, feel free to reach out. Thank you for reading, and I look forward to seeing you next time!

References:

Key Golang Concepts You Should Learn as a Beginner Go Developer

Temitope Oyedele — Tue, 12 Nov 2024 20:06:42 +0000

Learning new programming concepts can be hard. So you'll need a guide or a roadmap to help you navigate through the process.

Learning Golang is no exception. And as a beginner, you'll need to work diligently to learn the fundamental building blocks of the language. These key introductory concepts are important as they help you lay the groundwork for more complex development.

In this article, we will explore the main parts of Go that every beginner should learn to build a strong foundation in the language. So if you’re just starting out, this guide will help you solidify your knowledge and start building projects that’ll make you more confident coding in Golang.

Variables and Datatypes
Control Structures
Functions
Pointers
Error Handling
Goroutines and Concurrency
Structs and Inheritance
Go Standard Library
- Accessing and Using a Package from the Standard Library
Testing in Go
That’s a Wrap!

Variables and Datatypes

Variables in Go are used to store and manage data within a program. They act as containers that hold values of specific types. Variables allow you to retrieve and manipulate stored information.

Below is an example of using variables in Go:

package main

import "fmt"

func main(){
 // Variables
    var age int = 30
    var name string = "John"
    salary := 50000.50 // Short variable declaration(only to be used inside a function
  fmt.Println(age)
  fmt.Println(name)
   fmt.Println(salary)
}

To learn more about variables, you can check out my tutorial on them here.

On the other hand, data types define the kind of data a variable can hold. Since Go is a statically typed language, it requires you to specify the data type of each variable.

Some of the main data types in Go include:

Boolean: Represents a true or false value. It's used for logical decisions in the program.
Number: Includes integer types (like int, int32, int64) and floating-point types (like float32, float64) to store whole numbers and decimal values.
String: Represents a sequence of characters (text). It’s used to store words, phrases, or any text-based data.
Array: A collection of fixed-size elements of the same type. Arrays allow you to store multiple values in a single variable.
Slice: Similar to arrays, but with a dynamic size. Slices are more commonly used in Go since they offer greater flexibility.
Map: A collection of key-value pairs. Maps are used when you want to associate values with specific keys for fast lookup.
Struct: A way to group related data together. Structs allow you to define custom data types with multiple fields, each of a different type.
Pointer: Holds the memory address of another variable, allowing for more efficient memory manipulation in certain cases.

Below is an example showing how some of these data types work:

package main

import "fmt"

func main() {

    // Data Types
    var isEmployed bool = true // boolean
    var count int = 42 // integer
    var greeting string = "Hello, Go!" // string

    // Struct
    type Rectangle struct {
        width  float64
        height float64
    }
    rect := Rectangle{width: 10.5, height: 5.2}

    fmt.Println("Is Employed:", isEmployed)
    fmt.Println("Count:", count)
    fmt.Println("Greeting:", greeting)
    fmt.Println("Numbers:", numbers)
    fmt.Printf("Rectangle: width = %.2f, height = %.2f\n", rect.width, rect.height)

}

In this example, we demonstrate the usage of several data types:

bool: The isEmployed variable is declared as a bool and assigned the value true.
int: The count variable is declared as an int and assigned the value 42.
string: The greeting variable is declared as a string and assigned the value "Hello, Go!".
struct: We define a new data type called Rectangle that has two fields: width and height, both of which are float64. We then create a new instance of Rectangle and assign values to its fields.

Variables and data types form the basis of programming in Go. These concepts are essential building blocks that you'll use in almost every program you write. They allow you to store, manipulate, and organize data effectively.

With a strong command of variables and data types, you'll also be better equipped to tackle more advanced Go concepts and write robust, efficient code as you progress in your learning journey.

Control Structures

In Go, control structures are simply constructs that control the flow of execution in a program. They allow you to perform different actions based on conditions or repeatedly execute a block of code.

Some of the main control structures in Go include:

1. If/Else Statements

The if/else statement in Go executes a block of code based on a condition. If the condition returns true, the code inside the if block is performed. If the condition returns false, the else block (if any) is executed.

For example:

package main

import "fmt"

func main() {
    x := 10
    if x > 5 {
        fmt.Println("x is greater than 5")
    } else {
        fmt.Println("x is 5 or less")
    }
}

In the code above, if x is greater than 5, the first block is executed. Otherwise, the else block runs.

2. Switch Statements

The switch statement is a multi-directional branch that allows you to execute different blocks of code depending on the value of an expression. It’s easier to read than multiple if/else statements.

For example:

package main

import "fmt"

func main() {
    day := "Monday"
    switch day {
    case "Monday":
        fmt.Println("Start of the week")
    case "Friday":
        fmt.Println("Almost weekend")
    default:
        fmt.Println("It's another day")
    }
}

In the code above, the output will be "Start of the week" because the day variable matches the Monday case.

3. For Loops

Go only has one looping construct, the for loop. It can be used in various forms: traditional loops, range-based loops (to iterate over slices, maps, and so on), and infinite loops.

Below is an example of a traditional loop:

package main

import "fmt"

func main() {
    for i := 0; i < 5; i++ {
        fmt.Println(i)
    }
}

In the code above, the loop prints the numbers 0 through 4.

The range based loop provides a simplified way for iteration on slices, maps, and others. It makes it easier to access each element directly without needing to manually handle index or length checks. The loop automatically provides both the index and the value during each iteration, improving readability and reducing the chance of off-by-one errors or other indexing issues.

Below is an example of a range-based loop:

package main

import "fmt"

func main() {
    nums := []int{1, 2, 3, 4}
    for i, num := range nums {
        fmt.Println(i, num)
    }
}

In the code above, the for loop is used to iterate over the nums slice. The range keyword returns both the index and the value of each element in the slice. This makes it easy to process all elements of a slice without needing a counter variable.

Functions

A function in Go is a block of code that performs a specific task. Functions help you organize code by allowing you to construct reusable code logic, which is easier to maintain and understand.

Below is an example of a function that adds two numbers:

package main

import "fmt"

func add(a int, b int) int {
    return a + b
}

func main() {
    result := add(2, 3)
    fmt.Println("Result:", result)
}

Here, we have a simple function called add that takes two integer parameters (a and b) and returns their sum, which is also an integer type. The function is then called inside the main function which prints the result.

Functions are the foundation of Go programs, and understanding their structure and capabilities is critical.

Pointers

In Go, a pointer is a variable that stores the memory address of another variable. A pointer "points to" the region in memory where the actual value is stored rather than retaining the value itself.

Pointers are useful when you need to pass references to large structures or when you want to modify a variable's value from inside a function. They are also critical for memory management.

Below is a basic example that illustrates how pointers work in Go:

package main

import "fmt"

func main() {
    var num int = 10

    var ptr *int = &num

    fmt.Println("Value of num:", num)      
    fmt.Println("Pointer address:", ptr)   
    fmt.Println("Value at pointer:", *ptr)

    *ptr = 20
    fmt.Println("Updated value of num:", num) 
}

In the example above, we first declare a variable num and assign it the value of 10. We then create a pointer ptr that stores the memory address of num (using &num) and then print it out to see it. To access the stored pointer value, we use the *ptr. We then modify the value of num through the pointer by setting *ptr to 20, which directly changes num to 20.

This demonstrates how pointers allow you to access and modify variables via their memory addresses, which is useful for more efficient memory handling and function parameter passing in Go.

To get a better understanding of what pointers are, you can check out my article on them here.

Error Handling

In order to write robust and build reliable applications, you’ll need to learn about error handling. Compared to other programming languages, Go takes a unique approach to error handling, encouraging you to handle problems explicitly and immediately rather than relying on exceptions.

In Go, errors are treated as values, which means they are returned from functions just like any other value and must be handled by the developer. This approach helps to promote clarity and also ensures that potential issues are dealt with at the point where they occur.

Below is some example code to illustrate basic error handling in Go:

package main

import (
    "errors"
    "fmt"
)

func divide(a, b float64) (float64, error) {
    if b == 0 {
        return 0, errors.New("cannot divide by zero")
    }
    return a / b, nil
}

func main() {
    result, err := divide(10, 0)
    if err != nil {
        fmt.Println("Error:", err)
    } else {
        fmt.Println("Result:", result)
    }
}

In the code above, we have a divide function that takes two numbers and returns both a result and an error. If the second number is zero, an error is returned because division by zero is not allowed.

We also have a main function where we call divide and check if an error occurred by examining the err value. If an error is present, we handle it by printing an error message. Otherwise, we print the result.

This approach to error handling in Go ensures that errors are caught and dealt with right away, making the program more reliable and easier to troubleshoot.

Goroutines and Concurrency

Goroutines and concurrency are concepts that let your code efficiently execute multiple tasks in parallel.

A goroutine is a function that runs concurrently with other functions. Goroutines are incredibly lightweight, with a small memory footprint, allowing you to run thousands (or even millions) of goroutines simultaneously without overwhelming system resources.

Concurrency, on the other hand, refers to a program's capacity to handle numerous tasks at the same time. It does not necessarily imply that the tasks are executing concurrently (which is parallelism) but rather that they are making progress independently.

Let’s look at an example code to illustrate these concepts:

package main

import (
    "fmt"
    "time"
)

func printNumbers() {
    for i := 1; i <= 5; i++ {
        fmt.Println(i)
        time.Sleep(1 * time.Second)
    }
}

func printLetters() {
    for i := 'A'; i <= 'E'; i++ {
        fmt.Printf("%c\n", i)
        time.Sleep(1 * time.Second)
    }
}

func main() {

    go printNumbers()  // This runs concurrently
    go printLetters()  // This runs concurrently

    // Wait for goroutines to finish
    time.Sleep(6 * time.Second)
    fmt.Println("All tasks completed.")
}

In the code above, we create two functions, printNumbers and printLetters. One prints numbers from 1 to 5, and the other prints letters from 'A' to 'E'. We launch these functions as goroutines by adding the go keyword before calling them in the main function.

Goroutines are lightweight threads that allow functions to run concurrently. This means both printNumbers() and printLetters() can execute at the same time without waiting for each other to complete. The key concept here is concurrency, where multiple tasks (like printing numbers and letters) make progress independently, even though they don't necessarily run in parallel on separate cores.

In this case, both goroutines sleep for one second between prints, but because they are running concurrently, the numbers and letters can be printed almost simultaneously without blocking each other’s execution.

To ensure the program doesn’t exit before the goroutines complete their work, we add a time.Sleep(6 * time.Second) in the main function. This gives enough time for both goroutines to finish printing before the program terminates.

This example illustrates Go's powerful concurrency model through goroutines, enabling efficient multitasking without the complexity of traditional threading.

To dive deeper into goroutines and concurrency, Destiny Erhabor did a fine job in explaining what they are in his article here.

Structs and Inheritance

In Go, a struct is a composite data type that organizes variables (fields) into a single type. These fields can include a variety of data types, making structs suitable for describing complex data structures. Structs in Go function similarly to classes in other programming languages but without the methods of inheritance.

Let’s start with an example of a struct:

type Person struct {
    Name string
    Age  int
}

In this example, Person is a struct with two fields: Name which is a string and Age which is an int. You can create an instance of this struct like so:

p := Person{Name: "Alice", Age: 30}
fmt.Println(p.Name)  // Output: Alice

Go does not have traditional inheritance like some object-oriented languages where one class inherits fields and methods from another. Instead, Go uses composition, which allows you to embed one struct inside another.

Here’s an example code of struct composition:

type Employee struct {
    Person
    Position string
}

e := Employee{
    Person:   Person{Name: "Bob", Age: 25},
    Position: "Developer",
}

fmt.Println(e.Name)     // Output: Bob
fmt.Println(e.Position) // Output: Developer

In the code above, the Employee struct embeds the Person struct, and the fields of Person can be accessed directly like e.Name. This mimics some of the behavior you’d expect from inheritance in other languages, but it’s done through composition.

While Go lacks inheritance, it achieves polymorphism through interfaces. An interface is a type that specifies a set of method signatures. A type is said to implement an interface if it provides the methods declared by that interface.

What makes Go unique is that it uses implicit implementation, meaning that a type does not need to explicitly declare that it implements an interface – it just has to match the method signatures.

Let’s see an example:

type Speaker interface {
    Speak() string
}

type Person struct {
    Name string
}

func (p Person) Speak() string {
    return "Hi, my name is " + p.Name
}

func saySomething(s Speaker) {
    fmt.Println(s.Speak())
}

func main() {
    p := Person{Name: "Alice"}
    saySomething(p)  // Output: Hi, my name is Alice
}

In this example, Person implements the Speaker interface by defining a Speak method. The function saySomething takes any type that implements the Speaker interface, demonstrating polymorphism. Go’s interfaces provide a flexible way to design code that is clean and extensible without needing to rely on traditional inheritance.

Go Standard Library

It’s important to become familiar with Go's standard library. It contains a comprehensive collection of packaged libraries that provide you with a wide range of functionalities for tasks like file handling, network communication, string manipulation, data structures, cryptography, testing, and more. This allows you to perform common programming tasks without the need to install external packages.

Accessing and Using a Package from the Standard Library

To access a package from the standard library, you simply import it into your Go file using the import statement. Then, you can use the package's functions directly.

Some of the packages you can import from the standard library include:

fmt for formatted I/O
net/http for building web servers
io for I/O operations
strings for string manipulation
time for date and time operations

For example, let’s look at the fmt package, which is used for formatted input and output. Here's a basic example of how to use the fmt package to print formatted output:

package main

import "fmt"

func main() {
    name := "Alice"
    age := 30
    fmt.Printf("Hello, my name is %s and I am %d years old.\n", name, age)
}

In this example:

The import "fmt" line allows us to access the fmt package from the standard library.
We use fmt.Printf to format and print a string that includes a name (%s for strings) and an age (%d for integers).

Each package in the standard library is well-documented, with plenty of examples, so it’s a good idea to explore the official Go documentation to better understand how to use these packages in your projects. You can find the documentation for the Go standard library here.

Testing in Go

Testing is a first-class citizen in Go. This means that Go treats testing as a core, integral part of the development process.

Go’s testing framework is built around the testing package, which provides the tools necessary to write tests. You write your tests in separate files, which are automatically detected and executed by the Go tool.

To write a test, you need to add the suffix _test.go. For example, if your main code file is math, the tests for that file would go into math_test.go.

Let’s see how to write a simple test. Let’s say we have a simple function in math.go:


package math

func Add(a, b int) int {
    return a + b
}

To test the Add function, you need to create a test file called math_test.go:


package math

import "testing"

func TestAdd(t *testing.T) {
    result := Add(2, 3)
    expected := 5
    if result != expected {
        t.Errorf("Add(2, 3) = %d; want %d", result, expected)
    }
}

In the test above:

The TestAdd function is defined to test the Add function.
The t.Errorf is used to report an error if the result doesn't match the expected value.

To run the test, you use this command in your terminal:

go test

You can also get more verbose output by adding the -v flag like so:

go test -v

This is just the basics, as there are other types of tests, such as table-driven tests and benchmarking.

That’s a Wrap!

In this article, we took a look at nine key concepts to learn as a beginner getting started with Golang.

And keep in mind that this isn’t everything you’ll need to know when you’re learning Go – these are just what I consider to be the most important basics. And they should help you get your foot in the door of the world of Go.

If you think I missed a key concept, I’d love it if you’d share it with me so I can update the article. Thank you!

Understand How Express.js Works by Building Your Own Server Multiplexer from Scratch

Sifundo — Thu, 03 Oct 2024 15:31:46 +0000

Kata Machines have become the go-to method for mastering tough concepts, and it's hard to find a better tool for deliberate practice.

If you haven’t come across a kata yet, trust me—you will soon enough.

There’s a reason why developers love katas, whether they use them to sharpen their skills for personal projects or prepare for interviews.

A kata is all about deliberate practice. It comes from martial arts like Karate and Judo, and, according to Wikipedia, it’s defined as a pre-determined sequence of movements, techniques, and patterns that follow a specific order (source: wikipedia).

Kata Machines come from this idea: learning through drills and deliberate, conscious (choreographed) practice.

I realized just how perfect katas are when I was learning Haskell back in the day. If you know, you know. Haskell was a beast to learn for me back then!

So, I thought, why not do the same for the backend? Just pick one high-level concept, and drill down on it repeatedly and deliberately to its core and first principles.

In this article, I picked server-side frameworks. We're going to pick apart the idea of a "framework" using Express as an example.

We’re going to take high-level Express:

const express = require("express");

const PORT = 3000;
const app = express();

app.get("/", (req, res) => {
  res.send("Hello, world!");
});

app.listen(PORT, () => console.log("Server listening on port:", PORT));

And drill all the way down, repeatedly, until we touch Node.js native code:

void TCPWrap::New(const FunctionCallbackInfo& args) {
  CHECK(args.IsConstructCall());
  CHECK(args[0]->IsInt32());
  Environment* env = Environment::GetCurrent(args);
  int type_value = args[0].As()->Value();
  TCPWrap::SocketType type = static_cast(type_value);
  ProviderType provider;
  switch (type) {
    case SOCKET:
      provider = PROVIDER_TCPWRAP;
      break;
    case SERVER:
      provider = PROVIDER_TCPSERVERWRAP;
      break;
    default:
      UNREACHABLE();
  }
  new TCPWrap(env, args.This(), provider);
}

And having gained this new intuition, we’ll build back up with a custom "Express" implementation:

function serverMux() {
  function hook(req, res) {
    // To be implemented
  }
  return {
    hook
  };
}

const app = serverMux();

const server = http.createServer((req, res) => {
  app.hook(req, res);
});

It’s going to be quite the journey – and a rewarding one at that!

I'm assuming you have some backend knowledge and classify yourself as an advanced beginner who’s looking to level up.

If that sounds like you, we’re ready to proceed.

Here’s what we’ll cover:

Form 1: Server-Side Frameworks
Form 2: Implementing a Custom Server Mux

Wrapping Up

Form 1: Server-Side Frameworks

The term "server-side framework" is broad. Think about it: mysql2 could be considered a framework depending on how you classify frameworks and libraries. Even sharp.js for image editing could fit under the umbrella of server-side frameworks, right?

But the question is, what type of framework is Express.js?

Express is a multiplexer—specifically, a server multiplexer (server mux). I promise, the term isn’t as complex as it sounds. The implementation, though—that’s a whole different story.

In simple terms, a server mux is a router. Of course, Express and other server muxes handle more than just routing, but that’s the core idea.

Express takes in request and response objects from the server and routes them. Don’t worry, we’ll dive into routing soon.

Here’s an interesting point: if Express isn’t the server, then what exactly is the server?

To answer that, we need to look at the Express.js source code, which you can clone from GitHub:

git clone https://github.com/expressjs/express.git

Once you’re set, we can dive right in with our first deep dive.

First Drill: Unpacking Express.js

Open your Express source code in an editor. You’ll find the entry file express.js in the lib folder.

You can skim the file, but we’re going to focus on lines 42 and 43—the heart of it all:

mixin(app, EventEmitter.prototype, false);
mixin(app, proto, false);

What you’re looking at is object composition: a design pattern where an object is created by combining the properties and methods of other objects.

Our target object here is proto, which is imported from application.js, the core of Express.

Let’s open that file. There’s a lot of code, but remember, our goal is to figure out where the server is within Express.

If there’s one function in Express that everyone likely knows, it’s listen. The essence of a server is to "listen" over a network. So, do a quick Ctrl+F for "listen," and you’ll find the definition on line 633:

app.listen = function listen() {
  var server = http.createServer(this);
  return server.listen.apply(server, arguments);
};

There it is, the famous listen function. Did we just find the server?

var server = http.createServer(this);

We’ve already seen a version of this in the intro:

const server = http.createServer((req, res) => {
  app.hook(req, res);
});

This confirms that Express is indeed a server mux, and the actual server is returned by the Node.js createServer function from the http package.

That’s some solid progress!

We’ve peeled back a layer, but we can go deeper. What exactly does createServer do, and what is this server object?

The Server

A server is the basic unit of the backend. At its core, the concept is simple: how can two or more processes communicate over a network?

This is the fundamental idea behind network programming. We have devices equipped with IP addresses for identification and ports for data exchange over a network.

The communication itself is complex, which is where protocols come in to facilitate the process.

The most common protocols are UDP and TCP:

UDP is a connectionless protocol and does not guarantee reliable communication, but allows for low-latency and efficient data transfer. This is ideal for time-sensitive applications such as video conferencing, online gaming, and voice over IP (VoIP) (source Wikipedia).
TCP is a connection-oriented protocol with reliable, ordered, and error-checked data transmission between applications on networked devices. It’s a major part of internet applications (source Wikipedia).

TCP is the most widely used protocol due to its reliability, and most server-side applications you’ll work with, including Express, are TCP-based.

Although I love the quirks and power of UDP, we’ll focus on TCP, tracing its roots in Node.js.

We’ve already seen a glimpse of this:

void TCPWrap::New(const FunctionCallbackInfo& args) {
  // some code
  new TCPWrap(env, args.This(), provider);
}

Before we dig into that, we need to answer a key question: What does it really mean to be a server process?

Without getting too deep into file descriptors, sockets, or network layers, a server is an OS-level object responsible for handling communication between nodes. When you call:

const http = require("node:http");

const something = http.createServer({});

You’re creating an OS-level object, commonly known as a socket. This socket facilitates network communication between devices, along with handling data encoding and decoding.

In short, createServer abstracts and returns this socket object.

And, yes, we can implement this socket in Node.js. Remember, Node.js has native access to the OS, allowing JavaScript to function at the system level.

The Socket in Node.js

Here’s some code that creates a server socket:

// Using Node v20
const net = require('node:net');

const server = net.createServer((c) => {
  console.log('client connected', c.remoteAddress);
  c.write("Hello; world");

  c.on('end', () => {
    console.log('client disconnected');
  });
});

server.on('error', (err) => {
  throw err;
});

server.listen(3000, () => {
  console.log('server bound');
});

While net.createServer((c) is still a high-level abstraction like http.createServer, it returns the raw socket.

The c object represents the client that made the connection (dial). Beyond writing to it, we can do much more.

For instance, here’s a simple write operation:

c.write("Hello world");

Our socket is running on localhost:3000. If you make a request (or use curl):

curl localhost:3000

The OS-level network stack encodes not only your data but also information about who you are and where to find you—in the form of a response, among other things.

This is what the server receives, and it’s important to know where to send the response (like IP, and so on).

So, the c object represents all of that!

We’ve covered a lot of the surface-level concepts, but before we wrap up this part, here’s a bonus challenge:

Try writing a class on the server to manage multiple connections. You could store these connections in a data structure and periodically send data to them while the connection remains open.

We’re about three layers deep now, but the journey isn’t over. Remember the goal?

Now it’s time to clone the Node.js source code. Don’t worry, we’ll only focus on the relevant parts.

git clone https://github.com/nodejs/node.git

The Socket in Node.js Source Code

Let the tracing begin! Node.js is a massive codebase – it’s an entire engine that does way more than just handle sockets. But we only care about the networking part today.

First, navigate to the lib folder, and inside you’ll find a file called net.js. This is where most of the work happens for network applications. If you scroll down to line 210, you’ll see a familiar sight:

function createServer(options, connectionListener) {
  return new Server(options, connectionListener);
}

That’s it! Every time we create a server, it calls this function and returns a Server object. Anytime you see new in JavaScript, you should have a lightbulb moment—it means a new object or class (blueprint) is being created.

So we can trace and find the Server definition:

On line 1737

At first glance, it might seem like nothing special is happening. But JavaScript has a sneaky way of hiding complexity.

Here’s the thing: JavaScript is a prototype-based language. This means that objects can inherit features from other objects through prototypes. On line 1791, we see this in action:

ObjectSetPrototypeOf(Server.prototype, EventEmitter.prototype);

In plain English: our Server object is inheriting all the behavior from other objects like EventEmitter, for example. This is a common pattern in JavaScript libraries – remember the mixin in Express?

At this point, if you’ve never worked with prototypes or Object-Oriented JavaScript (OOJS), this might feel like advanced territory. But don’t worry – the good folks at MDN have an excellent guide on prototypes to get you up to speed.

Now, what’s one thing we know for sure about a Node.js server? It has a listen function. We use it all the time in server-side code (even in frameworks like Express). So, let’s check if our Server object has a listen function.

Scroll down a bit more, and there it is on line 2006:

Server.prototype.listen = function(...args) {}

This function handles a lot of stuff—like validating the port number—but the key part starts around line 2016, where the comment clearly tells us:

// start TCP server listening on host:port

We know what TCP is!

The important functions here are lookupAndListen and listenInCluster. They are responsible for starting the actual TCP server:

// start TCP server listening on host:port
if (options.host) {
  lookupAndListen(this, options.port | 0, options.host, backlog, options.exclusive, flags);
} else {
  listenInCluster(this, null, options.port | 0, 4, backlog, undefined, options.exclusive);
}

Digging into lookupAndListen (line 2156), we find that it calls listenInCluster, which leads us to another function: server._listen2 (yep, more tracing!):

server._listen2(address, port, addressType, backlog, fd, flags);

As the comments explain, this is all about backward compatibility:

// _listen2 sets up the listened handle, it is still named like this
// to avoid breaking code that wraps this method

I know this might feel like a wild goose chase, but trust me, tracing through a large codebase like Node.js requires patience. We’re getting close.

So, ._listen2 is defined in our Server object’s prototype and points to a function called setupListenHandle (line 1856). This function is the real hub where everything comes together.

Around line 1870 and 1883, you’ll find the function createServerHandle:

function createServerHandle(address, port, addressType, fd, flags) {
   handle = new TCP(TCPConstants.SERVER);
   isTCP = true;
   return handle;
}

Finally! We’ve hit the core: the TCP object. This is where the actual TCP server is created, the core. We could stop here, satisfied that we’ve found the TCP server, but why not dig deeper?

Remember that new TCP is creating an object, so we need to figure out what TCP actually represents.

Go back up to line 68, where you’ll see the following import:

const {
  TCP,
  TCPConnectWrap,
  constants: TCPConstants,
} = internalBinding('tcp_wrap');

This is where things get interesting. You might wonder: “What kind of import is that? It’s not your regular require or import statement.” That’s because JavaScript alone can’t handle TCP servers—it needs help from C++.

Node.js, which is built on the V8 engine, relies on C++ bindings to do the heavy lifting. These bindings are like a bridge, allowing JavaScript to communicate with low-level system functions (like creating a TCP server). internalBinding('tcp_wrap') is one of these bridges.

To truly trace things to their source, we need to dive into the Node.js C++ code. You’ll find tcp_wrap.cc in the src folder (among others like crypto, streams, async, fs). Open it, and you’ll find this function:

void TCPWrap::New(const FunctionCallbackInfo& args) {
  CHECK(args.IsConstructCall());
  CHECK(args[0]->IsInt32());
  Environment* env = Environment::GetCurrent(args);
  int type_value = args[0].As()->Value();
  TCPWrap::SocketType type = static_cast(type_value);
  ProviderType provider;
  switch (type) {
    case SOCKET:
      provider = PROVIDER_TCPWRAP;
      break;
    case SERVER:
      provider = PROVIDER_TCPSERVERWRAP;
      break;
    default:
      UNREACHABLE();
  }
  new TCPWrap(env, args.This(), provider);
}

This is where the TCP server is actually created. You can see more familiar functions like bind, and everything JavaScript does is just a mirror of these lower-level operations.

We’ve traced our way from high-level JavaScript all the way down to C++—the true beginning of a TCP server in Node.js.

We've completed the first part of the introduction: "And drill all the way down repeatedly until we touch Node.js native code" and now it's time to build up.

Form 2: Implementing a Custom Server Mux

Before diving into the code, the goal isn’t to focus on the complexity of mux (multiplexer) development (because that can get complicated). Instead, it’s to show how the server and mux fit together.

If anything, this is the key takeaway: the flow from the server to the router, and ultimately to the caller (the client that made the request).

Remember, we've already seen a similar concept in Express:

// Express app inherits from Node's EventEmitter
mixin(app, EventEmitter.prototype, false);
// Implements the server mux with functions like listen, handle, middleware
mixin(app, proto, false);

Behind the scenes, a lot of complex code is abstracted away. This helps simplify things and makes the code cleaner, but for teaching purposes, we’ll take a more verbose approach. This way, you can see how everything connects.

Creating Our Custom Router

Let's start simple and build a basic server. You probably already know how to create a native server in Node.js:

const server = http.createServer((req, res) => {
   app.hook(req, res);
});

Here, we’re introducing an object app with a hook function (which we’ll implement shortly). This is where the server redirects the req and res to our custom router. This hook is the meeting point—the interaction between the server and the router (mux).

Basic Mux Skeleton

Let's start by creating the structure of our mux:

function serverMux() {
  function hook(req, res) {
    // To be implemented
  }
  return {
    hook
  };
}

const app = serverMux();

const server = http.createServer((req, res) => {
  app.hook(req, res);
});

The Hook Function

The hook function is our middleman between the server and the mux. It receives the request (req) and response (res) objects from the server and passes them to our mux:

function hook(req, res) {
    requestsQueue.push(requestWrapper(req, res));
    console.log("new request!");
    processRequests();
}

Here, we introduced a few new things:

requestWrapper: A function to wrap the req and res.
processRequests: A function to handle the request processing.
requestsQueue: A basic JavaScript array that will act as our queue for handling requests.

Let's update serverMux to reflect this:

function serverMux() {
    const requestsQueue = [];

    async function processRequests() {
      // To be implemented
    }

    function hook(req, res) {
      requestsQueue.push(requestWrapper(req, res));
      console.log("new request!");
      processRequests();
    }

    return {
      hook
    };
}

Why Use a Queue?

You might be wondering why we’re using a queue instead of handling requests immediately like Express does with app.handle. Well, storing requests in a queue helps simulate an event loop. This will give us better visibility into how requests are processed, one at a time.

Queue Operations

A queue is a first-in, first-out (FIFO) data structure. Just like a line at the store, the request that arrives first gets processed first.

In our case, the requestsQueue is an array. Here’s how we’ll handle enqueueing and dequeueing:

Enqueue (push): We push requests into the queue with requestsQueue.push(requestWrapper(req, res));
Dequeue (shift): We pull the next request out of the queue with const c = requestsQueue.shift();

Request Wrapper

The requestWrapper function is a simple utility that wraps the incoming req and res objects, and extracts some useful information:

function requestWrapper(req, res) {
    return {
      url: req.url,
      method: req.method,
      req,
      res
    };
}

In more advanced frameworks like Hono.js, the request wrapper might add additional functionality, such as helper methods for setting headers or parsing body content. For now, we’re keeping things simple and just returning the request and response with the URL and method.

Testing the Queue

Let’s test this out by logging the request queue on every new request. Update your hook function:

function hook(req, res) {
    requestsQueue.push(requestWrapper(req, res));
    console.log("New request queued!", requestsQueue);
    processRequests();
}

Start the server with:

node index.js

Now, open another terminal and make a request to the server:

curl http://localhost:3000

You should see the queue logged in the console. The terminal might look like it's hanging because we haven’t responded to the request yet. You can exit the process manually for now.

Processing Requests

Here’s the full processRequests function:

async function processRequests() {
    while (requestsQueue.length > 0) {
        const c = requestsQueue.shift();
        if (c) {
            const handler = lookupTable[c.url] || lookupTable["/notfound"];
            if (handler) {
                (async function() {
                    handler(c.req, c.res);
                })();
            } else {
                console.log("Missing not found handler!");
            }
        }
    }
}

Let’s break it down:

Queue processing: We loop through the queue, dequeueing each request one by one.
Handler lookup: For each request, we check if a handler exists in the lookupTable for the URL. If it doesn’t, we fall back to a /notfound handler.
Handler execution: We execute the handler, passing the request and response objects.

Lookup Table and Handlers

We need a way to map URLs to their respective handlers. This is where the lookupTable comes in:

const lookupTable = {
    "/": (req, res) => {
      res.writeHead(200, { 'Content-Type': 'text/plain' });
      res.end('Hello, World!\n');
    },
    "/notfound": (req, res) => {
      res.writeHead(404, { 'Content-Type': 'text/plain' });
      res.end('404 Not Found\n');
    }
};

When a request comes in, we check if the URL matches an entry in the table. If it does, we call the corresponding handler function.

For example, calling curl http://localhost:3000 will hit the / route and return "Hello, World!". If you hit a non-existent route like /random, it will trigger the 404 handler.

Registering Handlers

Finally, let’s add a method to register new handlers dynamically:

function serverMux() {
  const lookupTable = {};

  function registerHandler(path, handler) {
    if (typeof path !== 'string' || !path) {
      throw new Error("Path must be a non-empty string");
    }
    if (typeof handler !== 'function') {
      throw new Error("Handler must be a function");
    }
    lookupTable[path] = handler;
  }

  return {
    hook,
    registerHandler
  };
}

Now, we can dynamically register new routes with their handlers:

app.registerHandler("/", (req, res) => {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('Home Page\n');
});

app.registerHandler("/about", (req, res) => {
    res.writeHead(200, { 'Content-Type': 'text/plain' });
    res.end('About Us\n');
});

Here's a full example of registerHandler in action:

const app =  serverMux()


app.registerHandler("/", (req, res)=> {

  res.writeHead(200, { 'Content-Type': 'text/plain' });

  res.end('Hello, World!');



})

app.registerHandler("/about", (req, res)=> {

    res.writeHead(200, { 'Content-Type': 'text/html' });

    res.end('About Us
');

})


app.registerHandler("/contact", (req, res)=> {

  res.writeHead(200, { 'Content-Type': 'text/html' });

  res.end('Contact Us
');

})


app.registerHandler("/api/data", (req, res)=> {

  res.writeHead(200, { 'Content-Type': 'application/json' });

  res.end(JSON.stringify({ message: 'Some data from the API' }));

})


app.registerHandler("/notfound", (req, res)=> {

   res.writeHead(404, { 'Content-Type': 'text/plain' });

    res.end('404 Not Found');

})



const server = http.createServer((req, res) => {

   console.log(req.url)

   app.hook(req, res)

});

Notice how similar it is to Express?

app.get("/", (req, res)=> {

})

Just a bit more verbose!

Now, run the server and put it to the test by pasting this into your terminal:

for /l %i in (1,1,100) do curl -X GET http://localhost:3000

This will send 100 requests. Try opening two or more terminals and running the same command concurrently to see how your server handles the load.

Congratulations! You’ve built a basic server multiplexer (mux). It may not revolutionize the world, but it's a solid starting point to understand how routing works in web frameworks.

Wrapping Up

In this article, we took a deep dive into the concept of server-side frameworks, using Express as our primary example. We traced it from its high-level abstractions all the way down to the native TCP server built in C++. Then, to cement these ideas, we built our own simple server mux.

It’s a powerful learning exercise, because we stripped away the magic and dug into the core of how things work. While this example is just the tip of the iceberg, it gives you the tools to explore even deeper. For a challenge, look into how Express handles pattern matching and registering routes—try improving our simple mux!

I left out more advanced topics like updating our queue with a linked list and simulating concurrent requests, so this is something you can explore.

Thanks for reading! I hope you enjoyed this exploration as much as I did writing it. If you have any thoughts, questions, or just want to connect I am on x, feel free to reach out.

And of course, enjoy your timezone!

Skills You Need to Become a Backend Developer – A Roadmap for Beginners

Programming with Shahan — Tue, 03 Sep 2024 14:38:36 +0000

In this guide, I'll discuss the basic skills you need and steps you can take to become a backend developer.

I'll give you a simple roadmap that'll help you know where you are and where you should go next.

Front End vs Back End

Every website has two parts: a front end and a back end.

The front end is the part that you see in the browser and interact with. Basically, it represents all the visual aspects of a website.

The back end is the part that powers the front end. It's behind the scenes, and it's mainly concerned with storing and accessing data using databases, APIs, and so on.

Because of this, web development jobs fall into three main categories:

👨‍💻 Front-end development
🛟 Back-end development
🚢 Full-stack development (involves both front-end and back-end development)

👷‍♂️What Exactly Does a Backend Developer Do?

Backend developers are responsible for constructing the foundational systems that drive the functionality of applications that users engage with.

This includes various tasks such as designing the architecture, and then implementing and maintaining these critical systems.

Their responsibilities typically involve working with:

💾 Databases, such as MySQL,
✂️ Frameworks, like Laravel or Ruby on Rails, and
🧨 APIs (Application Programming Interfaces).

Backend devs ensure that the backend infrastructure operates seamlessly, enabling smooth interactions between the user interface and the underlying data and processes.

Here are some of the core responsibilities of a backend developer:

Understanding the website's performance needs and goals
Developing and managing APIs for the website
Developing systems for secure data storage and processing (especially special use cases like payments)
Writing code, testing it, maintaining it, and developing solutions to bugs
Designing the site's architecture using established methodologies like Agile/Scrum and various frameworks
Organizing the system logic effectively
Providing solutions to address system-related issues and challenges

🥷 What You Need to Know to Become a Backend Developer

Becoming a proficient backend developer requires mastering various skills and technologies.

Below, I've outline a comprehensive roadmap to help you achieve this goal as quickly as possible.

Just keep in mind that everyone learns at a different pace, so it's hard to set a specific timeframe on how long this will take. Some might be able to get through it in six months or less, but others might take significantly more time.

Step 1: Foundations of Backend Development 🏗️

Understand the role and responsibilities of a back-end developer. Familiarize yourself with databases, frameworks, and APIs.
Join a comprehensive course on databases. Learn about SQL and relational databases like MySQL. Practice creating and managing databases.
Learn about server-side programming languages. Start with PHP or Python. Learn basic syntax, control structures, and data types.

Step 2: Advanced Backend Concepts 🔋

Deepen your understanding of databases. Explore NoSQL databases like MongoDB. Learn about data modeling and optimization.
Master server-side programming languages. Dig deeper into PHP or Python. Learn about functions, classes, and object-oriented programming.
Get hands-on projects with frameworks. Choose a popular framework like Laravel for PHP or Django for Python. Build simple applications to understand the model-view-controller (MVC) architecture.

Step 3: API Development and Integration 🛠️

Learn about APIs and their importance in backend development. Explore RESTful API design principles.
Start building your own APIs. Use tools like Postman to test and debug API endpoints. Learn about different authentication methods as well as security.
Focus on API integration. Learn how to consume APIs in web and mobile applications. Practice integrating third-party APIs into your projects.

Step 4: Troubleshooting and Optimization 🔮

Master troubleshooting techniques. Learn how to diagnose and debug backend applications effectively.

Jump into performance optimization techniques. Learn about lazy loading, caching, load balancing, and database indexing.
Learn monitoring and analytics tools. Use tools like New Relic or Datadog to monitor application performance. Learn how to generate and analyze performance metrics.

Step 5: Cloud Infrastructure and Deployment 🌥️

Understand cloud computing concepts. Learn about popular cloud providers like AWS or Azure.
Take a course on cloud services for backend development. Learn about serverless computing, containerization, and microservices.
Practice deploying applications to the cloud. Learn about CI/CD pipelines and automated deployment strategies.

Step 6: Advanced Topics and Projects 🚀

Now you're ready to learn more advanced backend topics. Choose topics like message queues, event-driven architecture, and real-time communication.
Work on a capstone project. Choose a challenging project that integrates various backend technologies and concepts.
Finally, reflect on your learning journey. Review your projects and identify areas for improvement. Prepare for job interviews by practicing coding challenges.

Back-end Development Tools and Software

Here's a breakdown of the tools and software commonly used in back-end development:

1. Database Frameworks:

MySQL: A relational database management system used for storing and retrieving data. It's widely used for web applications and offers scalability and reliability.
MongoDB: A NoSQL database program that uses JSON-like documents with schema flexibility. It's suitable for applications with rapidly changing data models or unstructured data.

2. Web Servers:

Node.js: Node.js is a cross-platform (meaning it supports multiple operating systems) JavaScript Runtime Environment that uses an event-driven, non-blocking I/O model. That means it can handle a large number of users at the same time. If you already know JavaScript, Node.js will likely be your best choice for building real-time applications like chat apps, online gaming apps, and collaborative tools.
Apache HTTP Server 2: An open-source web server software that delivers web content across the internet. It's known for its stability, security, and flexibility, making it a popular choice for hosting websites and web applications.

If your team is used to using languages like PHP or Python, Apache might be easier for them to work with.
LiteSpeed: LiteSpeed is another web server software that helps websites handle lots of visitors without slowing down or crashing. It’s great for businesses that expect a lot of traffic, like during busy times. LiteSpeed also works with many programming languages like Python, Ruby, PHP, and Java.

Imagine you're a company with 20 servers that can each handle 500 websites. Together, that’s 10,000 websites. But if you switch to LiteSpeed, you might be able to handle 20,000 websites with the same number of servers. That’s a big improvement!

When choosing a web server, it depends on what kind of website or app you’re building. If you’re just starting out, try Node.js first. It’s simple because you only need to learn one language (JavaScript) to build both the front part and the back part of your website.

3. Security Protocols:

SSL/TLS Certificates: These are cryptographic protocols that provide secure communication over a computer network. SSL (Secure Sockets Layer) and its successor TLS (Transport Layer Security) encrypt data between the server and client, ensuring confidentiality and integrity in online connections.

4. Version Control System:

Next on your journey is to become familiar with version control systems. These systems help track project history and enable collaborative work with others.

🔌 Learn Git: Git is the most widely used version control system, employed by over 70% of software development teams. It's a must-know tool, and allocating around two weeks for learning Git is advisable.

Back-end developers usually utilize these tools and software to build robust, scalable, and secure web applications that handle data storage, server hosting, and encryption of online communications.

If you seek more knowledge regarding backend development, you can check out this in-depth visual guide to backend development.

✅ Printable Roadmap

You can also use this printable Backend Developer Notion template to track your progress.

Beginner developers often struggle with choosing the right tech stack which leads them to waste their time and often lose motivation.

To help you out, I created this easy-to-follow 6-month backend developer roadmap in Notion so that you can track your progress and stick with your goals in real-time.

This roadmap:

🛤️ Provides a clear path to avoid confusion.
🎯 Helps you stay motivated by outlining where to start and end.
📚 Follows a structured plan, similar to a school syllabus.
📅 Organizes your learning with weekly goals for tools and languages.
⏳ Helps you get through it in 6 months, covering everything you need.
👀 Features a beautiful design for easy navigation.

👏 Conclusion

By following this roadmap and dedicating consistent effort to learning, you can acquire valuable backend development skills and be well-prepared to land a job in backend development.

If you're curious about becoming a front-end developer as well, you can read this article as well. 👉 The Roadmap to Become a Frontend Developer in 6 Months

If you ever wonder about the future of frontend development, you can also take a look at this article.

Stay tuned for more valuable content, and if you find it helpful, you may also like my YouTube Channel.

🤳My Social: X

Thanks for taking the time to read this article.

How to Create a CRUD API – NodeJS and Express Project for Beginners

Victor Yakubu — Fri, 08 Mar 2024 11:04:53 +0000

An API is a technology that powers communication between software applications on the internet. API stands for Application Programming Interface, and it is basically a set of rules and protocols that define how different software can interact with each other.

Imagine having two different programs: program A and program B. For these two programs to communicate together, an API is needed, and a set of rules ensure that they know what to expect when they interact with each other.

As a backend developer, your responsibilities involve building server-side applications, handling data storage, and providing the necessary functionalities to do all these through APIs.

There are different types of APIs like REST, GraphQL, gRPC, SOAP, and WebSockets. However, when it comes to web development, one is more popular, and that is the REST API.

In this article, you will learn how to create a CRUD API with Node.js and Express using the REST architecture, and by the end of this article, you should have a fully functional API that is able to perform CRUD operations.

So, let's dive into the world of backend development with Node.js and Express and get started on our journey to building a CRUD API.

What is a CRUD API?
What is Node.js?
Why Node?
How to Install Node.js
What is Express?
Why do You Need Express?
Prerequisites
How to Set Up Your Development Environment
How to Set up a server for Your CRUD Restful API with Node.js and Express
CRUD API Example
How to Create API Routes
How to Create Your own CRUD API
How to Create the GET /users Endpoint
How to Test Your API GET Request
How to Create the POST /users Endpoint
How to Test the POST Request
How to Create the GET /users/:id Endpoint
How to Test the GET Request
How to Create the DELETE /users/:id
How to Test the DELETE Request
How to Create the PATCH /users/:id Endpoint
How to Test the PATCH Request
Wrapping UP

What is a CRUD API?

In web development, CRUD operations are the bread and butter of backend systems. This is because they allow you to "Create", "Read", "Update", and "Delete" data through your API.

Here's a quick overview of the four primary HTTP methods associated with CRUD operations:

GET: Used for reading or retrieving data from the server.
POST: Used for creating new data on the server.
PUT: Used for updating existing data on the server.
DELETE: Used for removing data from the server.

Virtually every web application interacts with a database to perform these four core operations. Whether it's a social media platform, an e-commerce website, or a weather app, they all rely on creating, reading, updating, and deleting data.

For example, WhatsApp recently added an edit feature to the application, allowing users to make corrections to an already-sent message. That's one part of the CRUD operation in action (updating).

In the context of building a web API, these operations become the backbone of how your application interacts with data. Your API provides endpoints that allow client applications (like web or mobile apps) to perform these operations on the server.

This communication between the client and the server is the essence of web development, and understanding how to create a CRUD API is a crucial skill for a web developer.

What is Node.js?

Node.js is an open-source and cross-platform runtime environment for executing JavaScript code outside of a browser. Quite often, we use it to build back-end services, also called APIs. Node is ideal for building highly scalable, data-intensive and real-time back-end services that power our client applications

Why Node?

It is one of the most popular choices for building the backend.
You get to write JavaScript across your entire stack, making it easier to transition from either a frontend developer to a backend developer and vice versa.
It allows for easy scaling of applications, making it a good choice for large-scale professional projects.
It is fast and non-blocking. This is because of the asynchronous event-driven nature of Node.js.
Node.js has a vibrant community and a rich ecosystem of packages and libraries.

How to Install Node.js

Installation steps:

Download the Mac/Windows installer from the Node.js website.
Choose the Long-Term Support (LTS) version that’s shown on the left
After downloading, install/run the installer, and then follow the prompts. (You will have to click the NEXT button a bunch of times and accept the default installation settings
To confirm that Node has been successfully installed, open your terminal and run the command. (For Windows, you might need to restart your command before running it.)

node –version

What is Express?

Express is a fast, unopinionated, and minimalist web backend or server-side web framework for Node.js. Basically, it gives you the ability to build your APIs how you want them, with less code.

It is a framework built on top of Node.js that allows you to create your Backend with ease. You can use Express in combination with frontend frameworks like React, Angular, or Vue to build full-stack applications.

Why do You Need Express?

It makes building web applications with Node.js much easier.
It is extremely light, fast and free.
It is used for both server-rendered apps as well as API/Microservices.
It is the most popular Node.
It gives you full control over requests and responses.

Prerequisites

To follow along, you'll need to have the following:

Basic knowledge of JavaScript
Download and install Node.js and Postman on your computer

See the complete code for this tutorial on Github.

How to Set Up Your Development Environment

Before diving into creating your API, let's go through the process of creating a basic server on your local computer.

Here are the steps to follow:

Step #1 – Create Directory

Create a directory/folder on your computer. Open the folder in a code editor

Step #2 – Create index.js File

Create an index.js file inside the folder using this command:

touch index.js

Step #3 – Initialize NPM

Initialize NPM inside the folder by running this command in your terminal:

npm init -y

The command will create a package.json file with default values.

Step #5 – Install Express

Use the command below to install Express.js

npm install express

After installing Express, go to the package.json file and include “type”: “module”. This declaration will tell Node that this project will be using ES6 module syntax (import/export) instead of common.js, which is the default in Node.

package.json file with type:module

How to Set up a server for Your CRUD Restful API with Node.js and Express

To create a CRUD restful API, you first need to set up your server. You can do this by following these steps:

Step #1 – Write your Server Application Code inside the index.js file

Basically, a server code will look like this:

import express from 'express';
import bodyParser from ‘body-parser

const app = express();
const PORT = 5000

app.use(bodyParser.json());

app.listen(PORT, () => console.log(`Server running on port: http://localhost:${PORT}`));

Here's an explanation for the code above:

In the first line, we imported express from the Express module that we installed.
The bodyParser comes with Express, and it allows us to take in the incoming POST request body.
Next, we created an app using the express object.
We then specified the port for the application – it was set to 5000 (if you get an error using this port, it might be that the port is currently being used by a different app, so you can either change your port or stop the other app from using the port).
Next, we specified that JSON data will be used in the application.
Once that was created, we used the listen method on the app to make our application listen for incoming requests. The method accepts two things: the PORT, which is where we would be listening for requests from our client side, and a callback function that will be triggered when our server is set up.

Step #2 – Start the Server

Now you can start your server by running this command. If you used a different file, replace index.js with the file name where the server is located.

node index.js

Now your server should be running on port 5000. You can verify that your server is running on your terminal

Step #3 – Install Nodemon (Optional)

At the moment, anytime you make changes to your server file, you will need to restart the server before your changes can reflect (you can try it and see). So to take care of this challenge, you can use Nodemon. Run the command to install it:

npm install nodemon

To use Nodemon, head over to your package.json file and set up a script. Replace your start script with this instead:

"start": "nodemon index.js"

Note that index.js is the file where the server code is written.

Now you can start your server by running this command:

npm start

With this code, we've set up a server that listens on port 5000 and prints a message when it starts. But this is just the beginning because our server needs to do much more.

Let's explore how to handle API requests next.

CRUD API Example

Let's start by defining the API routes for each CRUD operation. These routes will serve as the entry points for your API, and they will map to specific actions we want to perform on our data.

In our case, these actions are creating, reading, updating, and deleting data.

How to Create API Routes

When the port (http://localhost:5000/) is opened in a browser, you will get an error.

localhost:5000 in the browser without any routes

This is because Node.js and Express are all about routing, and we don't any routes yet.

You can define API routes using the app.get(), app.post(), app.put(), and app.delete() methods in your Express application (in the index.js file).

Here's how to create a route to handle GET requests using the app.get() function:

app.get('/', (req, res)

The app.get() function accepts two parameters. The first is used to specify the path (in this case, it is '/').

The next parameter is a callback function where you define what happens when the GET request is called. The function also has two parameters: the request body (req), which can contain information such as the request query string, parameters, body, and HTTP headers. While the response object (res) contains the information you want to send

Here is the complete code:

import express from 'express';
import bodyParser from 'body-parser'
const app = express();

const PORT = 5000;

app.use(bodyParser.json());

app.get('/', (req, res) => {
    console.log('[GET ROUTE]');
    res.send('HELLO FROM HOMEPAGE');
})

app.listen(PORT, () => console.log(`Server running on port: http://localhost:${PORT}`));

When you head back to http://localhost:5000/ and refresh it, you shouldn’t get an error anymore.

localhost:5000 in the browser with GET route

How to Create Your own CRUD API

For this API, you will be handling a set of users. Handling users in a database is a great example because it is a common use case in most applications.

Here are the API endpoints you will be creating:

GET /users - find all users
POST /users - creates a user
GET /users/:id - finds a specific user
DELETE /users/:id - deletes a specific user
PATCH /users/:id - updates a specific user.

How to Create the GET /users Endpoint

Reading data is one of the most common operations in an API. In this example, you will be getting the list of all the users in your mock database. This information will be presented in JSON format.

To define a route to retrieve users' data from the database, follow these steps:

Create a new folder called routes
Create a new file called users.js inside the routes folder.
Write the code to set up the GET router.

import express from 'express';
const router = express.Router();

// Mock database
const users = [
  {
    first_name: 'John',
    last_name: 'Doe',
    email: 'johndoe@example.com',
  },
  {
    first_name: 'Alice',
    last_name: 'Smith',
    email: 'alicesmith@example.com',
  },
];

// Getting the list of users from the mock database
router.get('/', (req, res) => {
    res.send(users);
})

export default router

In this code snippet:

import express from 'express'; imports the Express.js framework
const router = express.Router(); creates a fresh router instance, stored in the variable router.
The users variable serves as the mock database containing an array of users.
The router.get() function sets up a route that responds to HTTP GET requests.
The second part of the code (req, res) => { ... } is a callback function. It gets executed when a request is made to the GET route.
Inside the callback function, we used res.send(users) to send a response back to the client. In this example, we're sending the users variable as the response. So when a user hits the GET URL, the server responds by sending the data inside the users variable in JSON format to the client.

Save your changes in the users.js file. Then do the following in the index.js file:

import your user routes from user.js:

import userRoutes from './routes/users.js'

Use the app.use method, and specify the path and router handler:

app.use('/users', userRoutes);

When a user visits http://localhost:5000/users, the router is triggered. It effectively acts as a filter, determining when a specific set of routes should be applied.

Here is the complete code for the index.js file:

import express from 'express';
import bodyParser from 'body-parser'
const app = express();
import userRoutes from './routes/users.js'

const PORT = 5000;

app.use(bodyParser.json());

app.use('/users', userRoutes);

app.get('/', (req, res) => res.send('HELLO FROM HOMEPAGE'))

app.get('/', (req, res));

app.listen(PORT, () => console.log(`Server running on port: http://localhost:${PORT}`));

How to Test Your API GET Request

You can use either a browser (browsers can only be used to perform GET requests) or Postman to test the GET request.

So copy your API URL, http://localhost:5000/users, and paste it either on Postman or in your browser. If you are using Postman, you will first need to make a GET request, then paste your API URL, and then click on send. After that, you will see the list of users in your Postman console.

postman GET route test

How to Create the POST /users Endpoint

You can use the POST request to add data to your database. It accepts input from the client and stores it in the database. To create data in our API, we'll define a route that accepts POST requests and saves the data to the mock database you have set up.

But before that, you'll need the UUID package. Use this command to install it:

npm install uuid

This package will help generate a unique ID for each user you will be creating. This will be useful when you are implementing the GET, DELETE, and PATCH user by ID requests, where you will need a way to identify a specific user.

So in the users.js file, do the following:

Import the uuid package:

import { v4 as uuidv4 } from 'uuid';

Secondly, you'll have to implement the code for the POST request.

Here is what it looks like:

router.post('/', (req, res) => {
    const user = req.body;

    users.push({ ...user, id: uuidv4() });

    res.send(`${user.first_name} has been added to the Database`);
})

In the code snippet:

The router.post() function sets up a route that responds to HTTP POST requests. This means that when a client sends a POST request to the root URL of your application, this route will be triggered.
Within the callback function (req, res) => { ... }, we access the req object, which represents the incoming request made by the client. Specifically, we're interested in the req.body property. This property contains the data (first name, last name, and email) that the client will send as part of the POST request's body.
With const user = req.body we extract this data from req.body and store it in the user variable.
Next, we added the user data to an array called users. To ensure each user has a unique identifier, we generate a universally unique identifier (UUID) using a function like uuidv4() and include it as id in the user object. This helps in keeping user records distinct and identifiable.
Finally, we used res.send() to send a response back to the client. In this case, we're sending a simple message notifying the client that the user has been successfully added to the database. The message includes the user's first name for a personalized touch.

Here is the complete code

import express from 'express';
import { v4 as uuidv4 } from 'uuid';

const router = express.Router();

const users = [];

// Adding users to our mock database

router.post('/', (req, res) => {
    const user = req.body;

    users.push({ ...user, id: uuidv4() });

    res.send(`${user.first_name} has been added to the Database`);
})  

export default router

How to Test the POST Request

Here are the steps to follow to make a POST request on Postman:

Go to Postman
Open a new request tab
Select "POST" from the list of available HTTP methods
In the URL field, enter the full URL where you want to send the POST request (http://localhost:5000/users)
Click on the "Body" tab in the request window.
Choose the format in which you want to send your POST data (choose JSON).
Enter the data you want to send in the request body. This data should match the format expected by the server.
Finally, click on “Send”

postman POST route test

If it is successful, you will get a response saying, “Daanny has been added to the database."

To confirm if it has been added, make a GET request, and you should see the new user added to your database. (Note: This user information will be lost when your server restarts since it’s not saved to an actual database).

postman GET route test

How to Create the GET /users/:id Endpoint

Fetching specific data based on a unique identifier, such as a user ID, is a common operation. It's often essential for building features like user profiles or retrieving individual records from a database.

In this section, you’ll explore how to use this endpoint (users/:id) to retrieve user information based on a provided user ID.

Let's get started!

router.get('/:id', (req, res) => {
    const { id } = req.params;

    const foundUser = users.find((user) => user.id === id)

    res.send(foundUser)
});

Here's what this code snippet does:

The router.get() function sets up a route that responds to HTTP GET requests. In this example, we've defined a route with ('/:id'). The :id part is a route parameter, which allows us to capture a dynamic value from the URL. In this case, it represents the user ID we want to retrieve.
Within the callback function (req, res) => { ... }, we can access the req object, which represents the incoming request made by the client. Specifically, we're interested in req.params, which holds the values of route parameters. In this case, we destructure id from req.params, effectively extracting the user ID from the URL. For example, if a client makes a GET request to /users/123, the id will be '123'.
We used the .find() method to search through this data based on the user ID (id) captured from the URL. This method attempts to find a user whose ID matches the provided id.
Once we located the user data (if it exists), we send it as the response using res.send(foundUser). The foundUser variable contains the user object that matches the requested ID.

How to Test the GET Request

In testing the API, follow these steps:

Go to your POST request tab and make as many requests as you want to add a new user to the database.
Go to your GET request tab and make a request to see the list of users you have added

postman GET route test

Copy the id of any of the users on the list
Create a new GET request tab, copy in the base API URL, and add the id of any of the users to it. It should be in a format like this: http://localhost:5000/users/734a9e75-b3f5-415f-82fb-79b4fdf1a593
Click on “Send”. If it’s successful, you will see the user information of the user id you used for the request

postman GET route test

How to Create the DELETE /users/:id

Sometimes, it's necessary to remove user accounts or specific records from a database for various reasons, such as account deactivation.

Here, you will explore how to use this endpoint to delete user data based on a provided user ID.

To delete data, we'll define a route that accepts DELETE requests and removes the data from the database.

See the code to delete a user from a database below

router.delete('/:id', (req, res) => {
  const { id } = req.params;

  users = users.filter((user) => user.id !== id)

  res.send(`${id} deleted successfully from database`);
});

Here's what this code does:

The router.delete() function sets up a route that responds to HTTP DELETE requests. In this example, we've defined a route with ('/:id'), where :id is a route parameter. It captures a dynamic value from the URL, representing the user ID we want to delete.
In the callback function (req, res) => { ... }, we can access the req object, which represents the incoming request made by the client. Specifically, we're interested in req.params, which holds the values of route parameters. Here, we destructure id from req.params, extracting the user ID from the URL. For instance, if a client sends a DELETE request to /users/123, id will be '123'.
Assuming that we have an array or database (users) containing user data, we employ the .filter() method to create a new array that excludes the user with the matching ID (id). This effectively removes the user from the data store.
After successfully deleting the user, we send a response back to the client using res.send(). The response contains a message confirming the deletion, including the user's ID that was deleted from the database.

How to Test the DELETE Request

Here are the steps to delete a user from the database on Postman:

Go to Postman
Open a new request tab
Select "DELETE" from the list of available HTTP methods
Enter the URL. It should contain the id of the user you want to delete (for example: http://localhost:5000/users/734a9e75-b3f5-415f-82fb-79b4fdf1a593). If you don’t have a user in your database, you will need to create one and copy the id.
Click on “Send”.

postman DELETE route test

How to Create the PATCH /users/:id Endpoint

There are times when you don't need to update an entire resource or object. Instead, you'd want to make partial modifications or adjustments. This is where the HTTP PATCH request comes into play.

For example, after creating a new user, you can change either the first name, last name, or email of that user using the PATCH request. Let’s see how to do that.

router.patch('/:id', (req, res) => {
  const { id } = req.params;

  const { first_name, last_name, email} = req.body;

  const user = users.find((user) => user.id === id)

  if(first_name) user.first_name = first_name;
  if(last_name) user.last_name = last_name;
  if(email) user.email = email;

  res.send(`User with the ${id} has been updated`)

});

Here's what this code snippet accomplishes:

The router.patch() function sets up a route that responds to HTTP PATCH requests. In this example, we've defined a route with ('/:id'), where :id is a route parameter. It captures the dynamic value from the URL, representing the user ID we want to update.
Within the callback function (req, res) => { ... }, we can access the req object, which represents the incoming request made by the client. Specifically, we're interested in req.params, which hold the values of route parameters (id in this case), and req.body, which contains the data to be updated.
Next, we used .find() to locate the user object with the matching ID (id). Once found, we can proceed to modify the user's data based on the content of req.body. We also checked if first_name, last_name, or email properties exist in req.body. If they do, we can update the corresponding properties of the user object with the new values. This allows us to make selective changes to the user's data without affecting other attributes.
After successfully applying the requested modifications, we send a response back to the client using res.send(). The response includes a message confirming the successful update of the user data, along with the user's ID.

How to Test the PATCH Request

Follow these steps to make a PATCH request in Postman:

Go to Postman
Open a new request tab
Select "PATCH" from the list of available HTTP methods
Enter URL; the URL will contain the id of the user you want to delete (for example: http://localhost:5000/users/734a9e75-b3f5-415f-82fb-79b4fdf1a593). If you don’t have a user in your database, you will need to create one and copy the id.
Click on the "Body" tab in the request window and choose the format in which you want to send your PATCH data (for example: JSON, form-data, x-www-form-urlencoded).
Enter the data you want to send in the request body. This data should only include the specific changes you want to make to the resource.
Then click the "Send" button. Postman will send the PATCH request to the specified URL with the provided data.

postman PATCH route test

Wrapping Up

APIs are the linchpin connecting various software components and enabling them to work together seamlessly. They allow them to communicate, share data, and perform tasks. In the context of web development, APIs enable the web to function as we know it today.

In this tutorial, we explored backend development by creating a CRUD API with Node.js and Express. We covered various concepts, like how to set up a development environment, create a server with Express and Node.js, and most importantly, how to handle CRUD operations and test your API using Postman.

While we've covered fundamental aspects of API creation in this tutorial, there is still a vast landscape of knowledge to explore as regards to APIs. These include how to secure your API by adding authentication, how to use middleware, database interaction, API deployment, and a lot more.

Communication Design Patterns for Backend Development

freeCodeCamp — Tue, 12 Sep 2023 16:25:03 +0000

By Chinwendu Enyinna

When you’re building the backend of an application, you’ve got to figure out how all the different components are going to talk to each other.

It’s like setting up a communication network for your app’s brain, and the way you do it can seriously impact how well your app performs.

But here’s the fun part: there’s no one-size-fits-all answer. The communication pattern you choose depends on what your application needs to do.

So, in this tutorial, we’re going to take a look at five different ways backend systems like to chat it up. We’ll explore what they’re good at and when you should invite them to the conversation. Let’s dive right in!

What is a Design Pattern?

Before we get into all the cool patterns, let’s talk about what a design pattern is, shall we?

Design patterns are nifty, tried-and-true reusable solutions to common problems encountered during software design and development. You can call them the cheat codes of software design and development.

They can help provide a structured approach to solving recurring challenges, offering a set of guidelines and best practices that can be adapted for various scenarios.

Now that we’ve got that down, let’s explore five of these emerging design patterns used for backend communication.

Request-Response Pattern

The request-response pattern is a fundamental building block for how the front-end and back-end of web applications chat with each other. This pattern is like a conversation between the client (say your browser) and the server, where they take turns speaking. Imagine it as a “ping-pong” of data.

Here's a diagram illustrating what that looks like:

request/response communication model

How does the Request-Response pattern work?

This pattern is all about synchronization. The client sends a request to the server, kind of like raising your hand to ask a question in class. Then it patiently waits for the server to respond before it can move on.

It’s like a polite conversation — one speaks, the other listens, and then they swap roles.

You’ve probably heard of RESTful APIs, right? Well, they’re a prime example of the request-response model in action.

When your app needs some data or wants to do something on the server, it crafts an HTTP request — say GET, POST, PUT, or DELETE (like asking nicely for a page), and sends it to specific endpoints (URLs) on the server. The server then processes your request and replies with the data you need or performs the requested action.

It’s like ordering your favorite dish from a menu — you ask, and the kitchen (server) cooks it up for you.

Interestingly, there’s more than one way to have this conversation. Besides REST, there’s GraphQL, an alternative that lets you ask for exactly the data you want. It’s like customizing your order at a restaurant — you get to pick and choose your ingredients.

It’s important to note that this pattern isn’t just limited to web applications. You’ll spot it in Remote Procedure Calls (RPCs), database queries (with the server being the client and the database, the server), and network protocols (HTTP, SMTP, FTP) to name a few. It’s like the language of communication for the web.

Benefits of the Request-Response pattern

Ease of Implementation and Simplicity: The way communication flows in this model is pretty straightforward, making it a go-to choice for many developers, especially when they’re building apps with basic interaction needs.

Flexibility and Adaptability (One Size Fits Many): The request-response pattern seamlessly fits into a wide range of contexts. You can use it for making API calls, rendering web pages on the server, fetching data from databases, and more.

Scalability: Each request from the client is handled individually, so the server can easily manage multiple requests at once. This is highly beneficial for high-traffic websites, APIs that get tons of calls, or cloud-based services.

Reliability: Since the server always sends back a response, the client can be sure its request is received and processed. This helps maintain data consistency and ensures that actions have been executed as intended even during high-traffic scenarios.

Ease of Debugging: If something goes wrong, the server kindly sends an error message with a status code stating what happened. This makes error handling easy.

Limitations of the Request-Response Pattern

Latency Problem: Because it’s a back-and-forth conversation, there’s often a waiting period. This amounts to idle periods and amplifies latency, especially when the request requires the server to perform time-consuming computing tasks.

Data Inconsistency in Case of Failures: If a failure occurs after the server has processed the request but before the response is delivered to the client, data inconsistency may result.

Complexity in Real-Time Communication: For applications that need lightning-fast real-time communication (like live streaming, gaming, or chat apps), this pattern can introduce delays and is therefore unsuitable for these use-cases.

Inefficiency in Broadcasting: In scenarios where you need to send the same data to multiple clients at once (broadcast), this pattern can be a bit inefficient. It’s like mailing individual letters instead of sending one group message.

Here’s a code example that shows the request-response pattern using Nodejs.

First, we have the server.js file. Here we’ve set up the server to listen for incoming requests from the client.

const http = require("http");

const server = http.createServer((req, res) => {
 res.statusCode = 200;
 res.setHeader("Content-Type", "text/plain");

 //check request method and receive data from client
 if (req.method === "POST") {
  let incomingMessage = "";
  req.on("data", (chunk) => {
   incomingMessage += chunk;
  });

  //write back message received from the client on the console
  req.on("end", () => {
   console.log(`Message from client: ${incomingMessage}`);
   res.end(`Hello client, message received!`);
  });
 } else {
  res.end("Hey there,  Client!\n");
 }
});

const PORT = 3030;
server.listen(PORT, () => {
 console.log(
  `Server is listening for incoming request from client on port:${PORT}`
 );
});

And here's the client.js file:

const http = require("http");

const options = {
 method: "POST",
 hostname: "localhost",
 port: 3030,
 path: "/",
};

//message to server
let messageToServer = "Hey there, server!";

//send a http request to the server
const req = http.request(options, (res) => {
 let incomingData = "";

 res.on("data", (chunk) => {
  incomingData += chunk;
 });

 res.on("end", () => {
  console.log(`Response from the server: ${incomingData}`);
 });
});

req.on("error", (error) => {
 console.log(`Error message: ${error.message}`);
});

//send message to the server
req.write(messageToServer);

//end your request
req.end();

Here's the output:

code output on the terminal

The Publish/Subscribe Pattern

Publish/Subscribe is another communication design pattern you may see on the backend. It is used in distributed systems for asynchronous communication between several (usually decoupled) components. It’s perfect for when you’ve got a bunch of components that need to work together but want to keep their distance.

Here's a diagram showing how it works:

Pub/Sub communication model

How does the Publish/Subscribe pattern work?

This pattern involves the use of message queues (often called message brokers) which serve as intermediaries between the publishers and subscribers. These message brokers group messages into something called channels (or topics).

Publishers are the components that create and send messages while subscribers are the components that receive and consume the messages.

The publishers simply toss messages (or events) into specific channels/topics within the message broker. These channels act as the distribution point for the messages. The subscribers then indicate interest by subscribing to those channels within the message broker and whenever a message or event is published to that channel, they receive a copy.

Message queuing tools like Apache Kafka and MQTT use the publish/subscribe communication pattern under the hood.

Benefits of the Publish/Subscribe pattern

Asynchronous Communication: Unlike the request-response model, pub/sub is designed to be asynchronous by nature. This makes it ideal for building real-time applications with reduced latency bottlenecks.

Loose Coupling of Components: The components in a publish/subscribe model are loosely coupled. This means that they are not tied together and can freely interact by triggering and responding to events.

Highly Scalable: There is no limit to the number of subscribers a publisher can publish events to. Also, there’s no limit to the number of publishers subscribers can subscribe to.

Independent of Language and Protocol: The Pub/Sub model can be easily integrated into any tech stack because it is language-agnostic. Also, it often supports a wide range of environments and platforms making it cross-platform compatible.

Load Balancing: In cases where multiple subscribers subscribe to a particular event, the pub/sub model can distribute the events evenly among the subscribers, providing load-balancing capabilities out of the box.

Limitations of the Publish/Subscribe pattern

Complexity in Implementation: Setting up a Pub/Sub system can be more complex than simpler communication models like the request-response pattern. You need to configure and manage message brokers, channels, and subscriptions, which can add overhead to your system.

Message Duplication: Depending on the configuration and network issues, messages can be duplicated. Subscribers might receive the same message more than once, which can lead to redundancy and extra processing.

Scalability Challenges: While Pub/Sub is highly scalable, managing extremely high volumes of messages and subscribers can become complex. You might need to consider how to distribute messages efficiently and handle massive numbers of subscribers.

Complex Error Handling: Dealing with errors in a Pub/Sub system can be challenging. Handling situations like message delivery failures or subscriber errors requires careful consideration and design.

When should you use it?

In building features that require real-time and low latency responsiveness for the end-users, for example live chat or gaming applications for multiple players.
In event notification systems
In building distributed systems that rely on logging and caching

Here’s a code snippet showing a simple pub/sub implementation using the npm package pubsub-js.

Here are the contents of the pubsub.js file:

const PubSub = require("pubsub-js");

/*
create  a topic of choice.
Any subscriber that subscribes to this topic
receives the published messages
*/
const TOPIC = "chat";

//a function to publish a messgae to subscribers
function publishMessageToSubscribers(message) {
 PubSub.publish(TOPIC, message);
 console.log(`Message Published: ${message}`);
}

let subscriber1 = function (msg, data) {
 console.log(msg, data);
 console.log("subscriber1 received: ", data);
};

let subscriber2 = function (msg, data) {
 console.log(msg, data);
 console.log("subscriber2 received: ", data);
};

// Subscriber1 subscribes to the topic
PubSub.subscribe(TOPIC, subscriber1);

//subscriber2 subscribes to topic
PubSub.subscribe(TOPIC, subscriber2);

publishMessageToSubscribers("Hello subscriber!");

console.log("Subscriber 1 is listening....");

console.log("Subscriber 2 is listening....");

Here's the output:

The Short Polling Pattern

Short polling is another communication pattern that facilitates data exchange between client and server. It uses the pull-based communication mechanism (which is basically the client pulling data from the backend) to continuously poll the server for new updates.

How does Short Polling work?

Imagine you’re waiting for a message from a friend, and you don’t want to miss it. What do you do? You keep asking, “Got a message for me yet?”

So, the client makes a request to the server at regular (fixed) intervals (say every x unit of time) to check for new data or updates. The server sends back a response and if there’s new available data or an update, the server includes the data in the response.

Here’s what the client-server communication would look like:

Client: “Hey, server, any new messages?”
Server: “Nah, nothing yet.”
Client: “Alright, I’ll check back later.”
[Some time passes…]
Client: “Hey, server, any new messages?”
Server: “Bingo! Here they are!”

Although polling has some similarities with the request-response model we discussed earlier, a key difference between them is timing.

While polling occurs at regular, predefined intervals regardless of whether updates are available, the request-response model allows clients to request data or actions on-demand when needed, reducing unnecessary communication.

Benefits of Polling

Simplicity: Polling is easy to understand and implement and it’s completely stateless between the client and server. This makes it perfect for scenarios where the complexities need to be minimized.

Compatibility: It can be used with a wide range of technologies and protocols which makes it highly compatible with a number of platforms and environments.

Some example use cases of polling pattern includes simple dashboards, weather apps that require periodic updates, resource monitoring, or in cases where you’re considering cross-platform compatibility.

Limitations of Polling

Latency: Polling introduces latency, as clients must wait for predefined intervals before receiving updates. This can lead to delays in accessing real-time data or receiving notifications.

Inefficiency: Constantly polling the server for updates can be inefficient and can result in unnecessary network and server overhead.

Scaling: Handling a large number of simultaneous clients using polling can be resource-intensive for the server. It may require significant server resources to manage numerous concurrent polling requests.

Here's a code snippet to illustrate short polling. Here, the client sends a periodic request(polls ) to the server to check for upload progress.

The app.js file:

const express = require("express");

const app = express();

//create a dictionary to store the upload progress
const uploadStatus = {};

//simulate upload of file
app.post("/upload", (req, res) => {
 //create a unique request id for incoming request
 const requestId = Math.floor(Math.random() * 1000000);
 uploadStatus[requestId] = 0;

 simulateUploadProgress(requestId);

 res.json({ requestId });
});

//endpoint to check upload progress
app.get("/status/:requestId", (req, res) => {
 const requestId = parseInt(req.params.requestId);

 if (!isNaN(requestId) && uploadStatus[requestId] !== undefined) {
  if (uploadStatus[requestId] === 100) {
   //upload completed
   res.json({ progress: 100, message: "UPLOAD COMPLETED!" });
  } else {
   // upload still in progress
   res.json({ progress: uploadStatus[requestId] });
  }
 } else {
  res.status(404).json({ error: "Request ID not found" });
 }
});

//update upload progress by 10% every 5 secs
function simulateUploadProgress(requestId) {
 if (uploadStatus[requestId] < 100) {
  setTimeout(() => {
   uploadStatus[requestId] += 10;
   simulateUploadProgress(requestId);
  }, 5000);
 }
}
const PORT = 4000;

app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

Here's the output in the terminal

The Long Polling Pattern

Long polling is like polling but uses a push-based communication mechanism. In long polling, instead of the client asking the server, “Any updates?” all the time, it says, “Let me know when something’s up.”

Here’s how Long Polling works:

The client pings the server, just like in regular polling, but this time, the server doesn’t immediately answer. It holds the connection open, like keeping a phone line on hold. When there’s something to share, it responds. It’s like the server’s saying, “Hey, I’ll call you when I have news.”

It is used in web applications to achieve real-time or near-real-time updates between a client and a server.

Benefits of the Long Polling Pattern

Low Latency: Long polling provides low latency when compared to traditional polling as data is immediately sent back to clients as soon as they are available.

Real-Time Updates: With the long polling technique, applications can achieve real-time updates without the need for constantly polling the server.

Limitations of the Long Polling Pattern

Resource Intensive: Long polling requires keeping many connections open. As a result, it can be resource-consuming on both the server and client side.

Increased Latency: Although long polling helps cut down on frequent polling, it can still introduce latency when compared to other real-time communication protocols like Web Socket. Clients may experience delays between updates since they have to wait for the server to respond.

Difficult to Scale: When dealing with a large number of concurrent clients, long polling can strain server resources. As more clients establish long-polling connections, the server may struggle to manage and respond to all these connections efficiently.

Typical use cases of long polling include real-time updates, event-driven applications, and event notification systems.

Here's a code snippet to illustrate the concept of long polling where the client waits for updates from the server.

const express = require("express");
const app = express();

const uploadStatus = {};

app.post("/upload", (req, res) => {
 const requestId = Math.floor(Math.random() * 1000000);
 uploadStatus[requestId] = 0;

 updateUploadProgress(requestId, uploadStatus[requestId]);

 res.json({ requestId });
});

app.get("/status/:requestId", async (req, res) => {
 const requestId = parseInt(req.params.requestId);

 //simulate long polling (the server will not respond until done)
 while ((await checkUploadComplete(requestId)) === false);
 res.end("\n\n Upload status: completed " + uploadStatus[requestId]);
});

//update progress by 20% after every 5 seconds
function updateUploadProgress(requestId, progress) {
 uploadStatus[requestId] = progress;
 console.log(`Updated progress to ${progress}`);
 if (progress === 100) return;
 this.setTimeout(() => updateUploadProgress(requestId, progress + 20), 5000);
}

//check upload status
function checkUploadComplete(requestId) {
 return new Promise((resolve, reject) => {
  if (uploadStatus[requestId] < 100) {
   this.setTimeout(() => resolve(false), 1000);
  } else {
   resolve(true);
  }
 });
}
const PORT = 4000;

app.listen(PORT, () => {
 console.log(`Server is running on port ${PORT}`);
});

And here's the output in the terminal.

code output

The Push Pattern

Push is a communication model that is used to deliver real-time updates to connected clients.

In this model, the client opens a connection to the server and awaits messages or updates from the server. Whenever there is a new update or message, the server immediately pushes that update to the client without the client explicitly requesting it — as long it is connected.

This model allows for bidirectional communication between the client and server. Web sockets, a popular protocol, uses the push model as its underlying data exchange method.

The Push model provides the most real-time or near real-time end-user experience when compared to other closely related paradigms such as polling and long polling.

To explain how the push pattern works, imagine a chat room with multiple connected users as an example. The client and server conversation will look like this:

user1 (client): “Hello…”
Server: “Oh, user1 has a message!” Instantly sends it to everyone else in the room.
User2 (client): Gets the message from user1 without doing anything.
User3 (client): Also receives the message from user1 — no need to refresh or ask for it.

In the example above, the push pattern enables real-time communication such that whenever there’s a message from a client in the chat room, it pushes the message to all other connected clients in that room without them having to continuously poll or request updates.

Popular technologies that uses the push pattern are RabbitMQ and WebSocket.

Benefits of Push Pattern

Real-Time Updates: The push model allows clients to receive updates from the server as soon as they are available. This is key, especially in applications where real-time updates are crucial.

Reduced Latency: Since the server pushes updates to the client as soon as they are available, it potentially reduces the latency.

Efficiency: Because there is no need for continuous polling or frequent client requests, there is efficient use of network resources and reduced server load.

While the push model is widely adopted as the best fit for providing real-time updates, it does have its own disadvantages.

Limitations of Push Pattern

Scalability: It can become difficult to scale as the number of connected clients increases. At this point, it becomes resource-intensive, especially on the server side since the server needs to maintain open connections with multiple clients.

Client support: Some clients might not be able to handle pushed messages as not all client platforms support push technologies. This may lead to compatibility issues and may need some sort of fallback mechanism for unsupported clients.

Some example use cases of the push pattern include chat and messaging apps, notification systems, IoT data streaming, and online gaming, to name a few.

Here’s a simple Node.js code example to illustrate the push pattern using Web Sockets. To run the code, you have to install the ws library using npm.

//create a server - this server will push updates to the clients at intervals
const WebSocket = require("ws");
const server = new WebSocket.Server({ port: 8080 });

server.on("connection", (client) => {
 console.log("Client connected to server");

 //Simulate client to receive real-time updates from the server every 2 seconds
 const interval = setInterval(() => {
  const message = `Message received at: ${new Date().toLocaleTimeString()}`;
  client.send(message);
 }, 2000);

 // Handle client disconnection
 client.on("close", () => {
  clearInterval(interval);
  console.log("Client disconnected");
 });
});

// Client (listens for updates from server)
const clientSocket = new WebSocket("ws://localhost:8080");

clientSocket.onmessage = (event) => {
 console.log(`Message from server: "${event.data}"`);
};

Wrapping Up

In this tutorial, we explored five key communication design patterns: Request-Response, Publish/Subscribe, Short Polling, Long Polling, and Push.

Each pattern has its unique strengths and limitations which makes them suitable for various use cases.

Depending on your application’s goal, understanding these patterns will help you design efficient backend systems.

backend - freeCodeCamp.org

From Flutter to Backend: How to Build Production-Grade REST APIs with Dart and Serverpod

Table of Contents

Prerequisites

How Serverpod Differs from Shelf

Installing Serverpod

Creating the Project

Understanding the Project Structure

Serverpod Core Concepts

Endpoints and the Session Object

Model Files and Code Generation

The Built-in ORM

Migrations

Starting the Development Server

Defining the Models

The User Model

The Profile Model

Running Code Generation

Creating and Applying Migrations

Building the API

The Auth Endpoint

The User Endpoint

The Profile Endpoint

Authentication

Password Hashing and JWT

Protecting Endpoints

Error Handling in Serverpod

Testing the API

Deployment

Deploying with Docker and Fly.io

Deploying with Serverpod Cloud

Conclusion

How to Use PostgreSQL as a Cache, Queue, and Search Engine

Table of Contents

Prerequisites

The Setup

Benchmark 1: Caching with UNLOGGED Tables

Results (200 Virtual Users)

Under Stress (1,000 Virtual Users, No Sleep)

The Verdict

Benchmark 2: Job Queues with SKIP LOCKED

Results (100 Producers + 50 Consumers)

The Verdict

Benchmark 3: Full-Text Search with tsvector

Results (500 Virtual Users)

Under Stress (500 Virtual Users, No Sleep)

The Verdict

Benchmark 4: Pub/Sub with LISTEN/NOTIFY

Results (200 Virtual Users)

The Verdict

The Combined Workload: The Honest Test

Results (All Four Workloads Running Together)

What I Learned

How to Build Real-Time Update Systems with MQTT and Express.js

Table of Contents

What You Will Learn

Prerequisites

Understanding the Architecture

What is MQTT and Why Use It?

MQTT Topic Design

Why Server-Sent Events Instead of WebSockets?

Project Setup

How to Set Up the MQTT Broker

Option 1: Docker (Recommended)

Option 2: Local Install

Option 3: Public Test Broker

How to Build the Express Server

How to Implement the Match Routes

How to Bridge MQTT to the Browser with Server-Sent Events

How to Build the Admin Upload Interface

Admin HTML Structure and Styles

Create New Match

Update Score

Add Match Event (Goal, Card, etc.)

Active Matches

Admin JavaScript Logic

How to Build the Live Viewer Interface

Live Feed

Viewer JavaScript Logic

How to Build the Home Page

Solving the Problem with `select_related` and `prefetch_related`