Prerequisites

This guide explains the Morgan library’s code to help you understand how it works under the hood. This is helpful if you have experience with Express and you're interested in understanding the inner workings that produce Morgan log lines. An understanding of closures in JavaScript is helpful for this guide but not necessary.

Table of Contents

• What is an Express Middleware?

• A Brief Overview of How Morgan Works

• What is a Morgan Token?

• What Happens When Morgan is Initialised?

• What Happens When Morgan Captures a Request?

• Next Steps

What is an Express Middleware?

According to Express documentation, a middleware is a function that has access to the request and response objects and the next function of an Express request cycle. They are generally used to intercept requests to execute side-effects before or after the request is handled by its route handler.

A middleware can be used to:

• Make changes to the request and the response objects: It can make changes to the request and response objects by attaching properties like headers and cookies to them.

• Terminate the request-response cycle: It can terminate a request and send a response to the client before or after the request is handled by its route handler.

• Execute the next middleware in the stack: It can trigger the execution of the middleware after it via the next function argument.

A function called next is usually the third argument of a middleware and it is used to pass the request to the next middleware. If the next function is not executed in a middleware and the request is not explicitly terminated by sending a response to the client, the request will be left hanging.

The interface of a middleware is shown in the code snippet below:

function middleware(request, response, next) {
    // operations to be performed when this middleware is executed
    next() // execute the next middleware
}

A middleware can intercept and handle cases where preceding middleware or route handlers throw unhandled errors. These middlewares are usually called error handler middlewares and accept four arguments as shown below:

function errorHandlerMiddleware(error, request, response, next) {}

The error argument represents the unhandled error.

Some middlewares like Morgan and cors are higher-order functions. They accept configuration arguments when initialised and return a middleware function, executed by Express when hit by a request.

function initialise(...configArgs) {
    // make use of configArgs here
    return function middleware(request, response, next) {
        // can also make use of configArgs here
        // operations to perform when this middleware is hit by a request
        next() // execute the next middleware
    }
}

A Brief Overview of How Morgan Works

import morgan from "morgan"
// morgan(format, [options])
morgan("tiny") // initialise morgan and return a middleware
// Sample output: GET /tiny 200 2 - 0.188 ms

Morgan is initialised by executing it with a required format argument and an optional options argument. The format argument may be:

• A predefined Morgan format name

• A format string containing predefined tokens (a token set)

• A custom format function that returns a log output in the form of a string

The options argument is optional. It is an object with three properties:

• immediate (boolean): If true, the log output will be created on receiving requests and not when a response is sent. It defaults to false.

• skip (function): The function accepts the request and response objects as arguments and returns a boolean value based on the logic in it. If the value returned is true, the log line for a request is not logged. skip defaults to false.

• stream (WritableStream): Output stream for writing log lines. It defaults to process.stdout but it could be a file.

When Morgan is initialised, it stores its initialisation arguments in closure variables and returns a middleware function. The function is executed when a request hits it and it outputs a log line for the request. The format and where the log line is output to are determined by the initialisation arguments.

What is a Morgan Token?

A Morgan token is a string prefixed by a colon, corresponding to property of the request or response objects or a user-generated value. For example, the request method’s token is ':method' and the response status code’s token is ':status'. A token can also accept an argument to customise its behaviour. For instance, in ':date[format]', format can be replaced with clf, iso or web to set the format of the date that would be in the log line. An understanding of Morgan tokens is crucial to understanding how Morgan works.

You can create new tokens using the morgan.token function. The code snippet below creates a new token called ':type' which corresponds to the response Content-Type header:

morgan.token('type', function (req, res) {
    return res.headers['content-type']
})

Morgan has predefined named format (tiny, dev, short, combined, common) strings containing a set of tokens and each named format has its specific token set and configuration. The token set for tiny is ':method :url :status :res[content-length] - :response-time ms'. Morgan can accept these named formats as the value of the format argument.

Aside from accepting named formats, Morgan can also accept a token set (for example ':method :url :status :res[content-length] - :response-time ms') as the format argument. A third argument type that Morgan accepts as the format argument is a format function. A format function accepts three arguments and returns a string that forms the log line for each request. For example, the format function described below:

morgan(function (tokens, req, res) {
    return `method: ${tokens.method(req, res)}
path: ${tokens.url(req, res)}
code: ${tokens.status(req, res)}`
})

This will produce a log line output like:

method: GET
path: /
code: 200

tokens.method, tokens.url and tokens.status are examples of functions on the morgan object that can generate values to be logged. To illustrate, the table below shows sample token methods, their token and sample output values:

The next sections of this article explain how Morgan works under the hood. To follow along, open up Morgan’s index.js file on GitHub.

What Happens When Morgan is Initialised?

When Morgan is initialised, it makes a copy of the arguments provided to it. For arguments that were not provided, Morgan sets default values for them. For instance, if no format string argument was provided, Morgan uses the 'default' named format and logs a deprecation notice afterwards with a suggested fix.

Morgan then sets up the formatLine function - the function that creates and returns the log line for a request when executed. How does it create the log line?

First, Morgan checks if format is a format function. If it is, the format function is assigned to formatLine and next, Morgan sets up the output stream. If format is not a function, it is passed as an argument to getFormatFunction. getFormatFunction accepts format and looks up Morgan’s object store to check if format is:

• One of Morgan’s named formats or a user-defined named format created via morgan.format

• A token set

If it is neither of the two, Morgan uses the default named format.

function getFormatFunction (name) { // `name` is also `format`
  var fmt = morgan[name] || name || morgan.default

  return typeof fmt !== 'function'
    ? compile(fmt)
    : fmt
}

If the named format corresponds to a format function after the lookup, Morgan returns the format function, which is then assigned to formatLine, else, it corresponds to a token set. Morgan compiles the token set into a format function through the compile function - one of the most important functions in the Morgan package.

How the compile Function Works

The compile function accepts a token set and returns a function that has the interface of a format function. How does it do this?

With the JavaScript replace method, it uses a RegEx to search for all occurrences of a token in the token set and replaces each occurrence. If the token set is ':method :res[content-length] - :response-time ms' , the RegEx replace method replaces the tokens as illustrated in the table below:

The result of the RegEx replace is prefixed with "use strict"\n return "" and ends up producing the string below:

"use strict" 
    return "" +  
    (tokens["method"](req, res) || "-") + " " +   
    (tokens["res"](req, res, "content-length") || "-") + " - " + 
    (tokens["response-time"](req, res) || "-") + " ms"

The string above is used to create a format function using the Function constructor and returned as:

function (tokens, req, res) {
  "use strict"
  return "" +
    (tokens["method"](req, res) || "-") + " " +
    (tokens["res"](req, res, "content-length") || "-") + " - " +
    (tokens["response-time"](req, res) || "-") + " ms"
}

The format function is eventually stored in formatLine.

When formatLine is executed with morgan as the tokens argument, it will create a log line. In the case of the sample token set, it will create a log line that will look like GET 20 - 1.233 ms.

After creating the formatLine function, Morgan uses the createBufferStream function to set up the streaming of the log lines created to the preferred output if set by options.stream. If options.stream is not set, it uses process.stdout.

Morgan does all this setting up so that it can create log lines quickly on capturing a request. It will be inefficient to do all of these for each request.

What Happens When Morgan Captures a Request?

When Morgan captures a request, it stores the IP address of the client using the getip function. Next, it stores the time that the request was triggered in the startAt property of the request object.

Then Morgan tries to generate the log line for the request and log it by executing the logRequest function. Morgan checks if the log line should be output on request, and if it should, Morgan executes logRequest and executes next thereafter to pass the request to the next middleware.

if (immediate) {
  logRequest()
} else {
  onHeaders(res, recordStartTime)
  onFinished(res, logRequest)
}

next()

If the log output should be created on response, Morgan registers two functions on the response object event listeners:

• An function to be run when headers start to be written to the response object: When this listener is triggered, it records the time when headers start to be written to the response object as _startAt and startTime. These values are used to calculate the response time and the total time of the request.

• A function to be run when the request closes, finishes or errors: It executes logRequest when this event occurs.

  Just when Node.js starts sending a response to the client, a _startAt property – the time when the response starts getting sent – is attached to the response object. The absolute difference between _startAt on the request object and _startAt on the response object is the response time of the request and can be seen through ":response-time".

  ":total-time" is the total time taken from when the request was received to when the response was completely sent. In practice, total-time will be equal to or slightly greater than response-time, depending on how long it takes to write the response body to the stream after the response has started.

Within logRequest, Morgan checks the value of the skip option. If it is a function, it is executed and if it returns true, Morgan doesn’t create a log output for the request and it exits.

function logRequest () {
  if (skip !== false && skip(req, res)) {
    debug('skip request')
    return
  }

  var line = formatLine(morgan, req, res)

  if (line == null) {
    debug('skip line')
    return
  }

  stream.write(line + '\n')
};

If skip is false or executing it evaluates to false, Morgan generates the log line for the request using formatLine. If the log line is null, Morgan exits, else it sends the log line to the output medium and exits.

Next Steps

You have learned how the Morgan Express middleware outputs logs. You now have foundational skills to pick up another middleware or Node.js library like helmet or cors that you use and study it to see how it works. Choose one, study it, write about it, and share it with others.

If you have any questions, you can connect with me on LinkedIn. I’ll be happy to respond.

Imagine having two different programs: program A and program B. For these two programs to communicate together, an API is needed, and a set of rules ensure that they know what to expect when they interact with each other.

As a backend developer, your responsibilities involve building server-side applications, handling data storage, and providing the necessary functionalities to do all these through APIs.

There are different types of APIs like REST, GraphQL, gRPC, SOAP, and WebSockets. However, when it comes to web development, one is more popular, and that is the REST API.

In this article, you will learn how to create a CRUD API with Node.js and Express using the REST architecture, and by the end of this article, you should have a fully functional API that is able to perform CRUD operations.

So, let's dive into the world of backend development with Node.js and Express and get started on our journey to building a CRUD API.

Table of Contents

• What is a CRUD API?
• What is Node.js?
• Why Node?
• How to Install Node.js
• What is Express?
• Why do You Need Express?
• Prerequisites
• How to Set Up Your Development Environment
• How to Set up a server for Your CRUD Restful API with Node.js and Express
• CRUD API Example
• How to Create API Routes
• How to Create Your own CRUD API
• How to Create the GET /users Endpoint
• How to Test Your API GET Request
• How to Create the POST /users Endpoint
• How to Test the POST Request
• How to Create the GET /users/:id Endpoint
• How to Test the GET Request
• How to Create the DELETE /users/:id
• How to Test the DELETE Request
• How to Create the PATCH /users/:id Endpoint
• How to Test the PATCH Request
• Wrapping UP

What is a CRUD API?

In web development, CRUD operations are the bread and butter of backend systems. This is because they allow you to "Create", "Read", "Update", and "Delete" data through your API.

Here's a quick overview of the four primary HTTP methods associated with CRUD operations:

• GET: Used for reading or retrieving data from the server.
• POST: Used for creating new data on the server.
• PUT: Used for updating existing data on the server.
• DELETE: Used for removing data from the server.

Virtually every web application interacts with a database to perform these four core operations. Whether it's a social media platform, an e-commerce website, or a weather app, they all rely on creating, reading, updating, and deleting data.

For example, WhatsApp recently added an edit feature to the application, allowing users to make corrections to an already-sent message. That's one part of the CRUD operation in action (updating).

In the context of building a web API, these operations become the backbone of how your application interacts with data. Your API provides endpoints that allow client applications (like web or mobile apps) to perform these operations on the server.

This communication between the client and the server is the essence of web development, and understanding how to create a CRUD API is a crucial skill for a web developer.

What is Node.js?

Node.js is an open-source and cross-platform runtime environment for executing JavaScript code outside of a browser. Quite often, we use it to build back-end services, also called APIs. Node is ideal for building highly scalable, data-intensive and real-time back-end services that power our client applications

Why Node?

• It is one of the most popular choices for building the backend.
• You get to write JavaScript across your entire stack, making it easier to transition from either a frontend developer to a backend developer and vice versa.
• It allows for easy scaling of applications, making it a good choice for large-scale professional projects.
• It is fast and non-blocking. This is because of the asynchronous event-driven nature of Node.js.
• Node.js has a vibrant community and a rich ecosystem of packages and libraries.

How to Install Node.js

Installation steps:

1. Download the Mac/Windows installer from the Node.js website.
2. Choose the Long-Term Support (LTS) version that’s shown on the left
3. After downloading, install/run the installer, and then follow the prompts. (You will have to click the NEXT button a bunch of times and accept the default installation settings
4. To confirm that Node has been successfully installed, open your terminal and run the command. (For Windows, you might need to restart your command before running it.)

node –version

What is Express?

Express is a fast, unopinionated, and minimalist web backend or server-side web framework for Node.js. Basically, it gives you the ability to build your APIs how you want them, with less code.

It is a framework built on top of Node.js that allows you to create your Backend with ease. You can use Express in combination with frontend frameworks like React, Angular, or Vue to build full-stack applications.

Why do You Need Express?

• It makes building web applications with Node.js much easier.
• It is extremely light, fast and free.
• It is used for both server-rendered apps as well as API/Microservices.
• It is the most popular Node.
• It gives you full control over requests and responses.

Prerequisites

To follow along, you'll need to have the following:

• Basic knowledge of JavaScript
• Download and install Node.js and Postman on your computer

See the complete code for this tutorial on Github.

How to Set Up Your Development Environment

Before diving into creating your API, let's go through the process of creating a basic server on your local computer.

Here are the steps to follow:

Step #1 – Create Directory

Create a directory/folder on your computer. Open the folder in a code editor

Step #2 – Create index.js File

Create an index.js file inside the folder using this command:

touch index.js

Step #3 – Initialize NPM

Initialize NPM inside the folder by running this command in your terminal:

npm init -y

The command will create a package.json file with default values.

Step #5 – Install Express

Use the command below to install Express.js

npm install express

After installing Express, go to the package.json file and include “type”: “module”. This declaration will tell Node that this project will be using ES6 module syntax (import/export) instead of common.js, which is the default in Node.

package.json file with type:module

How to Set up a server for Your CRUD Restful API with Node.js and Express

To create a CRUD restful API, you first need to set up your server. You can do this by following these steps:

Step #1 – Write your Server Application Code inside the index.js file

Basically, a server code will look like this:

import express from 'express';
import bodyParser from ‘body-parser

const app = express();
const PORT = 5000

app.use(bodyParser.json());

app.listen(PORT, () => console.log(`Server running on port: http://localhost:${PORT}`));

Here's an explanation for the code above:

• In the first line, we imported express from the Express module that we installed.
• The bodyParser comes with Express, and it allows us to take in the incoming POST request body.
• Next, we created an app using the express object.
• We then specified the port for the application – it was set to 5000 (if you get an error using this port, it might be that the port is currently being used by a different app, so you can either change your port or stop the other app from using the port).
• Next, we specified that JSON data will be used in the application.
• Once that was created, we used the listen method on the app to make our application listen for incoming requests. The method accepts two things: the PORT, which is where we would be listening for requests from our client side, and a callback function that will be triggered when our server is set up.

Step #2 – Start the Server

Now you can start your server by running this command. If you used a different file, replace index.js with the file name where the server is located.

node index.js

Now your server should be running on port 5000. You can verify that your server is running on your terminal

Step #3 – Install Nodemon (Optional)

At the moment, anytime you make changes to your server file, you will need to restart the server before your changes can reflect (you can try it and see). So to take care of this challenge, you can use Nodemon. Run the command to install it:

npm install nodemon

To use Nodemon, head over to your package.json file and set up a script. Replace your start script with this instead:

"start": "nodemon index.js"

Note that index.js is the file where the server code is written.

Now you can start your server by running this command:

npm start

With this code, we've set up a server that listens on port 5000 and prints a message when it starts. But this is just the beginning because our server needs to do much more.

Let's explore how to handle API requests next.

CRUD API Example

Let's start by defining the API routes for each CRUD operation. These routes will serve as the entry points for your API, and they will map to specific actions we want to perform on our data.

In our case, these actions are creating, reading, updating, and deleting data.

How to Create API Routes

When the port (http://localhost:5000/) is opened in a browser, you will get an error.

localhost:5000 in the browser without any routes

This is because Node.js and Express are all about routing, and we don't any routes yet.

You can define API routes using the app.get(), app.post(), app.put(), and app.delete() methods in your Express application (in the index.js file).

Here's how to create a route to handle GET requests using the app.get() function:

app.get('/', (req, res)

The app.get() function accepts two parameters. The first is used to specify the path (in this case, it is '/').

The next parameter is a callback function where you define what happens when the GET request is called. The function also has two parameters: the request body (req), which can contain information such as the request query string, parameters, body, and HTTP headers. While the response object (res) contains the information you want to send

Here is the complete code:

import express from 'express';
import bodyParser from 'body-parser'
const app = express();

const PORT = 5000;

app.use(bodyParser.json());

app.get('/', (req, res) => {
    console.log('[GET ROUTE]');
    res.send('HELLO FROM HOMEPAGE');
})

app.listen(PORT, () => console.log(`Server running on port: http://localhost:${PORT}`));

When you head back to http://localhost:5000/ and refresh it, you shouldn’t get an error anymore.

localhost:5000 in the browser with GET route

How to Create Your own CRUD API

For this API, you will be handling a set of users. Handling users in a database is a great example because it is a common use case in most applications.

Here are the API endpoints you will be creating:

1. GET /users - find all users
2. POST /users - creates a user
3. GET /users/:id - finds a specific user
4. DELETE /users/:id - deletes a specific user
5. PATCH /users/:id - updates a specific user.

How to Create the GET /users Endpoint

Reading data is one of the most common operations in an API. In this example, you will be getting the list of all the users in your mock database. This information will be presented in JSON format.

To define a route to retrieve users' data from the database, follow these steps:

• Create a new folder called routes
• Create a new file called users.js inside the routes folder.
• Write the code to set up the GET router.

import express from 'express';
const router = express.Router();

// Mock database
const users = [
  {
    first_name: 'John',
    last_name: 'Doe',
    email: 'johndoe@example.com',
  },
  {
    first_name: 'Alice',
    last_name: 'Smith',
    email: 'alicesmith@example.com',
  },
];

// Getting the list of users from the mock database
router.get('/', (req, res) => {
    res.send(users);
})

export default router

In this code snippet:

• import express from 'express'; imports the Express.js framework
• const router = express.Router(); creates a fresh router instance, stored in the variable router.
• The users variable serves as the mock database containing an array of users.
• The router.get() function sets up a route that responds to HTTP GET requests.
• The second part of the code (req, res) => { ... } is a callback function. It gets executed when a request is made to the GET route.
• Inside the callback function, we used res.send(users) to send a response back to the client. In this example, we're sending the users variable as the response. So when a user hits the GET URL, the server responds by sending the data inside the users variable in JSON format to the client.

Save your changes in the users.js file. Then do the following in the index.js file:

import your user routes from user.js:

import userRoutes from './routes/users.js'

Use the app.use method, and specify the path and router handler:

app.use('/users', userRoutes);

When a user visits http://localhost:5000/users, the router is triggered. It effectively acts as a filter, determining when a specific set of routes should be applied.

Here is the complete code for the index.js file:

import express from 'express';
import bodyParser from 'body-parser'
const app = express();
import userRoutes from './routes/users.js'

const PORT = 5000;

app.use(bodyParser.json());

app.use('/users', userRoutes);

app.get('/', (req, res) => res.send('HELLO FROM HOMEPAGE'))

app.get('/', (req, res));

app.listen(PORT, () => console.log(`Server running on port: http://localhost:${PORT}`));

How to Test Your API GET Request

You can use either a browser (browsers can only be used to perform GET requests) or Postman to test the GET request.

So copy your API URL, http://localhost:5000/users, and paste it either on Postman or in your browser. If you are using Postman, you will first need to make a GET request, then paste your API URL, and then click on send. After that, you will see the list of users in your Postman console.

postman GET route test

How to Create the POST /users Endpoint

You can use the POST request to add data to your database. It accepts input from the client and stores it in the database. To create data in our API, we'll define a route that accepts POST requests and saves the data to the mock database you have set up.

But before that, you'll need the UUID package. Use this command to install it:

npm install uuid

This package will help generate a unique ID for each user you will be creating. This will be useful when you are implementing the GET, DELETE, and PATCH user by ID requests, where you will need a way to identify a specific user.

So in the users.js file, do the following:

Import the uuid package:

import { v4 as uuidv4 } from 'uuid';

Secondly, you'll have to implement the code for the POST request.

Here is what it looks like:

router.post('/', (req, res) => {
    const user = req.body;

    users.push({ ...user, id: uuidv4() });

    res.send(`${user.first_name} has been added to the Database`);
})

In the code snippet:

• The router.post() function sets up a route that responds to HTTP POST requests. This means that when a client sends a POST request to the root URL of your application, this route will be triggered.
• Within the callback function (req, res) => { ... }, we access the req object, which represents the incoming request made by the client. Specifically, we're interested in the req.body property. This property contains the data (first name, last name, and email) that the client will send as part of the POST request's body.
• With const user = req.body we extract this data from req.body and store it in the user variable.
• Next, we added the user data to an array called users. To ensure each user has a unique identifier, we generate a universally unique identifier (UUID) using a function like uuidv4() and include it as id in the user object. This helps in keeping user records distinct and identifiable.
• Finally, we used res.send() to send a response back to the client. In this case, we're sending a simple message notifying the client that the user has been successfully added to the database. The message includes the user's first name for a personalized touch.

Here is the complete code

import express from 'express';
import { v4 as uuidv4 } from 'uuid';

const router = express.Router();

const users = [];

// Adding users to our mock database

router.post('/', (req, res) => {
    const user = req.body;

    users.push({ ...user, id: uuidv4() });

    res.send(`${user.first_name} has been added to the Database`);
})  

export default router

How to Test the POST Request

Here are the steps to follow to make a POST request on Postman:

• Go to Postman
• Open a new request tab
• Select "POST" from the list of available HTTP methods
• In the URL field, enter the full URL where you want to send the POST request (http://localhost:5000/users)
• Click on the "Body" tab in the request window.
• Choose the format in which you want to send your POST data (choose JSON).
• Enter the data you want to send in the request body. This data should match the format expected by the server.
• Finally, click on “Send”

postman POST route test

If it is successful, you will get a response saying, “Daanny has been added to the database."

To confirm if it has been added, make a GET request, and you should see the new user added to your database. (Note: This user information will be lost when your server restarts since it’s not saved to an actual database).

postman GET route test

How to Create the GET /users/:id Endpoint

Fetching specific data based on a unique identifier, such as a user ID, is a common operation. It's often essential for building features like user profiles or retrieving individual records from a database.

In this section, you’ll explore how to use this endpoint (users/:id) to retrieve user information based on a provided user ID.

Let's get started!

router.get('/:id', (req, res) => {
    const { id } = req.params;

    const foundUser = users.find((user) => user.id === id)

    res.send(foundUser)
});

Here's what this code snippet does:

• The router.get() function sets up a route that responds to HTTP GET requests. In this example, we've defined a route with ('/:id'). The :id part is a route parameter, which allows us to capture a dynamic value from the URL. In this case, it represents the user ID we want to retrieve.
• Within the callback function (req, res) => { ... }, we can access the req object, which represents the incoming request made by the client. Specifically, we're interested in req.params, which holds the values of route parameters. In this case, we destructure id from req.params, effectively extracting the user ID from the URL. For example, if a client makes a GET request to /users/123, the id will be '123'.
• We used the .find() method to search through this data based on the user ID (id) captured from the URL. This method attempts to find a user whose ID matches the provided id.
• Once we located the user data (if it exists), we send it as the response using res.send(foundUser). The foundUser variable contains the user object that matches the requested ID.

How to Test the GET Request

In testing the API, follow these steps:

• Go to your POST request tab and make as many requests as you want to add a new user to the database.
• Go to your GET request tab and make a request to see the list of users you have added

postman GET route test

• Copy the id of any of the users on the list
• Create a new GET request tab, copy in the base API URL, and add the id of any of the users to it. It should be in a format like this: http://localhost:5000/users/734a9e75-b3f5-415f-82fb-79b4fdf1a593
• Click on “Send”. If it’s successful, you will see the user information of the user id you used for the request

postman GET route test

How to Create the DELETE /users/:id

Sometimes, it's necessary to remove user accounts or specific records from a database for various reasons, such as account deactivation.

Here, you will explore how to use this endpoint to delete user data based on a provided user ID.

To delete data, we'll define a route that accepts DELETE requests and removes the data from the database.

See the code to delete a user from a database below

router.delete('/:id', (req, res) => {
  const { id } = req.params;

  users = users.filter((user) => user.id !== id)

  res.send(`${id} deleted successfully from database`);
});

Here's what this code does:

• The router.delete() function sets up a route that responds to HTTP DELETE requests. In this example, we've defined a route with ('/:id'), where :id is a route parameter. It captures a dynamic value from the URL, representing the user ID we want to delete.
• In the callback function (req, res) => { ... }, we can access the req object, which represents the incoming request made by the client. Specifically, we're interested in req.params, which holds the values of route parameters. Here, we destructure id from req.params, extracting the user ID from the URL. For instance, if a client sends a DELETE request to /users/123, id will be '123'.
• Assuming that we have an array or database (users) containing user data, we employ the .filter() method to create a new array that excludes the user with the matching ID (id). This effectively removes the user from the data store.
• After successfully deleting the user, we send a response back to the client using res.send(). The response contains a message confirming the deletion, including the user's ID that was deleted from the database.

How to Test the DELETE Request

Here are the steps to delete a user from the database on Postman:

• Go to Postman
• Open a new request tab
• Select "DELETE" from the list of available HTTP methods
• Enter the URL. It should contain the id of the user you want to delete (for example: http://localhost:5000/users/734a9e75-b3f5-415f-82fb-79b4fdf1a593). If you don’t have a user in your database, you will need to create one and copy the id.
• Click on “Send”.

postman DELETE route test

postman DELETE route test

How to Create the PATCH /users/:id Endpoint

There are times when you don't need to update an entire resource or object. Instead, you'd want to make partial modifications or adjustments. This is where the HTTP PATCH request comes into play.

For example, after creating a new user, you can change either the first name, last name, or email of that user using the PATCH request. Let’s see how to do that.

router.patch('/:id', (req, res) => {
  const { id } = req.params;

  const { first_name, last_name, email} = req.body;

  const user = users.find((user) => user.id === id)

  if(first_name) user.first_name = first_name;
  if(last_name) user.last_name = last_name;
  if(email) user.email = email;

  res.send(`User with the ${id} has been updated`)

});

Here's what this code snippet accomplishes:

• The router.patch() function sets up a route that responds to HTTP PATCH requests. In this example, we've defined a route with ('/:id'), where :id is a route parameter. It captures the dynamic value from the URL, representing the user ID we want to update.
• Within the callback function (req, res) => { ... }, we can access the req object, which represents the incoming request made by the client. Specifically, we're interested in req.params, which hold the values of route parameters (id in this case), and req.body, which contains the data to be updated.
• Next, we used .find() to locate the user object with the matching ID (id). Once found, we can proceed to modify the user's data based on the content of req.body. We also checked if first_name, last_name, or email properties exist in req.body. If they do, we can update the corresponding properties of the user object with the new values. This allows us to make selective changes to the user's data without affecting other attributes.
• After successfully applying the requested modifications, we send a response back to the client using res.send(). The response includes a message confirming the successful update of the user data, along with the user's ID.

How to Test the PATCH Request

Follow these steps to make a PATCH request in Postman:

• Go to Postman
• Open a new request tab
• Select "PATCH" from the list of available HTTP methods
• Enter URL; the URL will contain the id of the user you want to delete (for example: http://localhost:5000/users/734a9e75-b3f5-415f-82fb-79b4fdf1a593). If you don’t have a user in your database, you will need to create one and copy the id.
• Click on the "Body" tab in the request window and choose the format in which you want to send your PATCH data (for example: JSON, form-data, x-www-form-urlencoded).
• Enter the data you want to send in the request body. This data should only include the specific changes you want to make to the resource.
• Then click the "Send" button. Postman will send the PATCH request to the specified URL with the provided data.

postman PATCH route test

Wrapping Up

APIs are the linchpin connecting various software components and enabling them to work together seamlessly. They allow them to communicate, share data, and perform tasks. In the context of web development, APIs enable the web to function as we know it today.

In this tutorial, we explored backend development by creating a CRUD API with Node.js and Express. We covered various concepts, like how to set up a development environment, create a server with Express and Node.js, and most importantly, how to handle CRUD operations and test your API using Postman.

While we've covered fundamental aspects of API creation in this tutorial, there is still a vast landscape of knowledge to explore as regards to APIs. These include how to secure your API by adding authentication, how to use middleware, database interaction, API deployment, and a lot more.

Putting together all of the pieces of a full stack JavaScript application can be a complex endeavor.

In this tutorial, we're going to build a multiplayer tabletop game simulator using Vue, Phaser, Node/Express, and Socket.IO to learn several concepts that will be useful in any full stack app.

You can follow along with this video tutorial as well (1 hour 16 minute watch):

All of the project files for this tutorial are available on GitHub.

Project Overview

Our project will feature a Phaser game instance that will allow us to create tokens and cards on screen, and move them around on a digital game board.

The Phaser instance will be wrapped in a Vue component that will handle things like multiplayer chat and commands. Together, Phaser and Vue will comprise our front end (referred to from here on as the "client"), and we'll use Socket.IO to communicate with other players and tie together the front and back ends of our app.

The back end (referred to from here on as the "server") will be a simple Express server that receives Socket.IO events from the client and acts accordingly. The whole application will run on Node as its runtime.

You don't need to be an expert in any of the above frameworks to complete this project, but it would be a good idea to have a solid foundation in basic JavaScript and HTML/CSS before trying to tackle the specifics. You can also follow along with my series on Learning JavaScript by Making Digital Tabletop Games and Web Apps.

You'll also want to make sure that you have Node and Git installed, along with your favorite code editor and a command line interface (you can follow my tutorial on setting up an IDE here if you need help).

Let's get started!

Part 1: Client Basics

We'll begin building our client by installing the Vue CLI, which will help us with some tooling and allow us to make changes to our files without having to reload our web browser.

In a command line, type in the following to install the Vue CLI globally:

npm install -g @vue/cli

Navigate to a desired directory and create a new folder for our project:

mkdir tabletop-project
cd tabletop-project

Now we can use the Vue CLI to template a front end project for us:

vue create client

You can just hit "enter" at the ensuing prompts unless you have specific preferences.

The Vue CLI has helpfully templated a front end project for us, which we can view in our code editor:

Let's navigate to our new client folder in our CLI and run the template app:

cd client
npm run serve

After a little work, the Vue CLI should begin displaying our app in a web browser at the default http://localhost:8080:

Cool! We have the basic structure of our client. Let's break it by creating two new components in the /components folder, called Game.vue and Chat.vue (you can go ahead and delete HelloWorld.vue and anything in the assets folder if you're obsessed with tidiness like I am).

Replace the code in App.vue with the following:

<template>
    <div id="app">
        <div id="game">
            <Game />
        </div>
        <div id="border" />
        <div id="input">
            <Chat />
        </div>
    </div>
</template>

<script>
    import Chat from './components/Chat.vue';
    import Game from './components/Game.vue';

    export default {
        name: 'App',
        components: {
            Chat,
            Game
        }
    }
</script>

<style>
    #app {
        font-family: 'Trebuchet MS';
        text-align: left;
        background-color: black;
        color: cyan;
        display: flex;
    }
    #game {
        width: 50vw;
        height: 100vh;
    }
    #input {
        width: 50vw;
        height: 100vh;
    }
    #border {
        border-right: 2px solid cyan;
    }
    @media (max-width: 1000px) {
        #app {
            flex-direction: column;
        }
        #game {
            width: 100vw;
            height: 50vh;
        }
        #input {
            width: 100vw;
            height: 50vh;
        }
    }
</style>

As you can see, a Vue component ordinarily has three sections: Template, Script, and Style, which contain any HTML, JavaScript, and CSS for that component, respectively. We've just imported our Game and Chat components here and added a little styling to give it a cyberpunk feel when it's all up and running.

That's actually all that we need to do to set up our App.vue component, which will house everything else in our client. Before we can actually do anything with it, we'll need to get our server working!

Part 2: Server Basics

At our root directory (tabletop-project, above /client), initialize a new project in a new command line interface by typing:

npm init

Like with our client, you can go ahead and press "enter" at the prompts unless there are specifics that you'd like to designate at this time.

We'll need to install Express and Socket.IO, along with Nodemon to watch our server files for us and reboot as necessary:

npm install --save express socket.io nodemon

Let's open up the new package.json file in that root directory and add a "start" command in the "scripts" section:

  "scripts": {
    "start": "nodemon server.js"
  },

Create a new file called server.js in this directory, and enter the following code:

const server = require('express')();
const http = require('http').createServer(server);
const io = require('socket.io')(http);

io.on('connection', function (socket) {
    console.log('A user connected: ' + socket.id);

    socket.on('send', function (text) {
        let newText = "<" + socket.id + "> " + text;
        io.emit('receive', newText);
    });

    socket.on('disconnect', function () {
        console.log('A user disconnected: ' + socket.id);
    });
});

http.listen(3000, function () {
    console.log('Server started!');
});

Excellent! Our simple server will now listen at http://localhost:3000, and use Socket.IO to log to the console when a user connects and disconnects, with their socket ID.

When the server receives a "send" event from a client, it will create a new text string that includes the socket ID of the client that emitted the event, and emit its own "receive" event to all clients with the text that it received, interpolated with the socket ID.

We can test the server by returning to our command line and starting it up :

npm run start

The command console should now display:

Cool! Let's return to the Chat component of our client to start building out our front end functionality.

Part 3: Chat

Let's open a separate command line interface and navigate to the /client directory. Within that directory, install the client version of Socket.IO:

npm install --save socket.io-client

In /client/src/components/Chat.vue, add the following code:

<template>
    <div id="container">
        <div id="output">
            <h1>STRUCT</h1>
            <p v-for="(text, index) in textOutput" :key="index">{{text}}</p>
        </div>
        <div id="input">
            <form>
                <input type="text" v-model="textInput" :placeholder="textInput" />
                <input type="submit" value="Send" v-on:click="submitText" />
            </form>
        </div>
    </div>
</template>

<script>
    import io from 'socket.io-client';
    let socket = io('http://localhost:3000');

    export default {
        name: 'Chat',
        data: function () {
            return {
                textInput: null,
                textOutput: []
            }
        },
        methods: {
            submitText: function (event) {
                event.preventDefault();
                socket.emit('send', this.textInput);
            }
        },
        created: function () {
            socket.on('connect', () => {
                console.log('Connected!');
            });
            socket.on('receive', (text) => {
                this.textOutput.push(text);
                this.textInput = null;
            });
        }
    }
</script>

<style scoped>
    #container {
        text-align: left;
        display: flex;
        flex-direction: column;
        margin-left: 1vw;
        min-height: 100vh;
    }
    h1 {
        text-align: center;
    }
    .hotpink {
        color: hotpink;
    }
    #input {
        position: fixed;
        margin-top: 95vh;
    }
    input[type=text] {
        height: 20px;
        width:  40vw;
        border: 2px solid cyan;
        background-color: black;
        color: hotpink;
        padding-left: 1em;
    }
    input[type=submit]{
        height: 25px;
        width: 5vw;
        background-color: black;
        color: cyan;
        border: 2px solid cyan;
        margin-right: 2vw;
    }
    input[type=submit]:focus{
        outline: none;
    }
    input[type=submit]:hover{
        color: hotpink;
    }
    @media (max-width: 1000px) {
        #container {
            border-left: none;
            border-top: 2px solid cyan;
            min-height: 50vh;
        }
        #input {
            margin-top: 43vh;
        }
        #output {
            margin-right: 10vw;
        }
        input[type=text] {
            width: 60vw;
        }
        input[type=submit] {
            min-width: 10vw;
        }
    }
</style>

Let's examine the above from bottom to top before moving forward. Between the

Have you ever wondered how chat applications work behind the scenes? Well, today I am going to walk you through how to make a REST + Sockets-based application built on top of NodeJS/ExpressJS using MongoDB.

I have been working on the content for this article for over a week now – I really hope it helps someone out there.

Prerequisites

• Set up Mongodb on your machine [Installation guide written here]
• For windows users, you can find the installation guide [here]
• For macOS users, you can find the installation guide [here][To the point installation that I wrote]
• For Linux users, you can find the installation guide [here]
• Install Node/NPM on your machine [Installation link here] (I am using Node version v12.18.0)

Topics we'll cover

General

• Create an express server
• How to do API validations
• Create basic skeleton for the entire application
• Setting up MongoDB (installation, setup in express)
• Creating users API + Database (Create a user, Get a user by id, Get all users, Delete a user by id)
• Understanding what a middleware is
• JWT (JSON web tokens) authentication (decode/encode) - Login middleware
• Web socket class that handles events when a user disconnects, adds its identity, joins a chat room, wants to mute a chat room
• Discussing chat room & chat message database model

For the API

• Initiate a chat between users
• Create a message in chat room
• See conversation for a chat room by its id
• Mark an entire conversation as read (similar to Whatsapp)
• Get recent conversation from all chats (similar to Facebook messenger)

Bonus - API

• Delete a chat room by id along with all its associated messages
• Delete a message by id

Before we begin, I wanted to touch on some basics in the following videos.

Understanding the basics of ExpressJS

What are routes? Controllers? How do we allow for CORS (cross origin resource sharing)? How do we allow enduser to send data in JSON format in API request?

I talk about all this and more (including REST conventions) in this video:

Also, here's a GitHub link to the entire source code of this video [Chapter 0]

Do have a look at the README.md for "Chapter 0" source code. It has all the relevant learning links I mention in the video along with an amazing half hour tutorial on postman.

Adding API validation to your API end-point

In the below video, you'll learn how to write your own custom validation using a library called "make-validation":

Here's the GitHub link to the entire source code of this video [Chapter 0].

And here's the make-validation library link [GitHub][npm][example].

The entire source code of this tutorial can be found here. If you have any feedback, please just reach out to me on http://twitter.com/adeelibr. If you like this tutorial kindly leave a star on the github repository**.**

Let's begin now that you know the basics of ExpressJS and how to validate a user response.

Getting started

Create a folder called chat-app:

mkdir chat-app;
cd chat-app;

Next initialize a new npm project in your project root folder by typing the following:

npm init -y

and install the following packages:

npm i cors @withvoid/make-validation express jsonwebtoken mongoose morgan socket.io uuid --save;
npm i nodemon --save-dev;

And in your package.json scripts section add the following 2 scripts:

"scripts": {
    "start": "nodemon server/index.js",
    "start:server": "node server/index.js"
},

Your package.json now should look something like this:

{
  "name": "chapter-1-chat",
  "version": "0.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "nodemon server/index.js",
    "start:server": "node server/index.js"
  },
  "dependencies": {
    "@withvoid/make-validation": "1.0.5",
    "cors": "2.8.5",
    "express": "4.16.1",
    "jsonwebtoken": "8.5.1",
    "mongoose": "5.9.18",
    "morgan": "1.9.1",
    "socket.io": "2.3.0",
    "uuid": "8.1.0"
  },
  "devDependencies": {
    "nodemon": "2.0.4"
  }
}

Awesome!

Now in your project's root folder create a new folder called server:

cd chat-app;
mkdir server;
cd server;

Inside your server folder create a file called index.js and add the following content to it:

import http from "http";
import express from "express";
import logger from "morgan";
import cors from "cors";
// routes
import indexRouter from "./routes/index.js";
import userRouter from "./routes/user.js";
import chatRoomRouter from "./routes/chatRoom.js";
import deleteRouter from "./routes/delete.js";
// middlewares
import { decode } from './middlewares/jwt.js'

const app = express();

/** Get port from environment and store in Express. */
const port = process.env.PORT || "3000";
app.set("port", port);

app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

app.use("/", indexRouter);
app.use("/users", userRouter);
app.use("/room", decode, chatRoomRouter);
app.use("/delete", deleteRouter);

/** catch 404 and forward to error handler */
app.use('*', (req, res) => {
  return res.status(404).json({
    success: false,
    message: 'API endpoint doesnt exist'
  })
});

/** Create HTTP server. */
const server = http.createServer(app);
/** Listen on provided port, on all network interfaces. */
server.listen(port);
/** Event listener for HTTP server "listening" event. */
server.on("listening", () => {
  console.log(`Listening on port:: http://localhost:${port}/`)
});

Let's add the routes for indexRouter userRouter chatRoomRouter & deleteRouter.

In your project's root folder create a folder called routes. Inside the routes folder add the following files:

• index.js
• user.js
• chatRoom.js
• delete.js

Let's add content for routes/index.js first:

import express from 'express';
// controllers
import users from '../controllers/user.js';
// middlewares
import { encode } from '../middlewares/jwt.js';

const router = express.Router();

router
  .post('/login/:userId', encode, (req, res, next) => { });

export default router;

Let's add content for routes/user.js next:

import express from 'express';
// controllers
import user from '../controllers/user.js';

const router = express.Router();

router
  .get('/', user.onGetAllUsers)
  .post('/', user.onCreateUser)
  .get('/:id', user.onGetUserById)
  .delete('/:id', user.onDeleteUserById)

export default router;

And now let's add content for routes/chatRoom.js:

import express from 'express';
// controllers
import chatRoom from '../controllers/chatRoom.js';

const router = express.Router();

router
  .get('/', chatRoom.getRecentConversation)
  .get('/:roomId', chatRoom.getConversationByRoomId)
  .post('/initiate', chatRoom.initiate)
  .post('/:roomId/message', chatRoom.postMessage)
  .put('/:roomId/mark-read', chatRoom.markConversationReadByRoomId)

export default router;

Finally, let's add content for routes/delete.js:

import express from 'express';
// controllers
import deleteController from '../controllers/delete.js';

const router = express.Router();

router
  .delete('/room/:roomId', deleteController.deleteRoomById)
  .delete('/message/:messageId', deleteController.deleteMessageById)

export default router;

Awesome now that our routes are in place let's add the controllers for each route.

Create a new folder called controllers. Inside that folder create the following files:

• user.js
• chatRoom.js
• delete.js

Let's start of with controllers/user.js:

export default {
  onGetAllUsers: async (req, res) => { },
  onGetUserById: async (req, res) => { },
  onCreateUser: async (req, res) => { },
  onDeleteUserById: async (req, res) => { },
}

Next let's add content in controllers/chatRoom.js:

export default {
  initiate: async (req, res) => { },
  postMessage: async (req, res) => { },
  getRecentConversation: async (req, res) => { },
  getConversationByRoomId: async (req, res) => { },
  markConversationReadByRoomId: async (req, res) => { },
}

And finally let's add content for controllers/delete.js:

export default {
  deleteRoomById: async (req, res) => {},
  deleteMessageById: async (req, res) => {},
}

So far we have added empty controllers for each route, so they don't do much yet. We'll add functionality in a bit.

Just one more thing – let's add a new folder called middlewares and inside that folder create a file called jwt.js. Then add the following content to it:

import jwt from 'jsonwebtoken';

export const decode = (req, res, next) => {}

export const encode = async (req, res, next) => {}

I will talk about what this file does in a bit, so for now let's just ignore it.

We are done with our basic boilerplate of the code base

We have ended up doing the following:

• Created an Express server that listens on port 3000
• Added cross-origin-resource (CORS) to our server.js
• Added a logger to our server.js
• And also added route handlers with empty controllers.

Nothing fancy so far that I haven't covered in the videos above.

Let's setup MongoDB in our application

Before we add MongoDB to our code base, make sure it is installed in your machine by running one of the following:

• For Windows users installation guide [here]
• For macOS users installation guide [here][To the point installation that I wrote]
• For Linux users installation guide [here]

If you are having issues installing MongoDB, just let me know at https://twitter.com/adeelibr and I'll write a custom guide for you or make an installation video guide. :)

I am using Robo3T as my MongoDB GUI.

Now you should have your MongoDB instance running and Robo3T installed. (You can use any GUI client that you like for this. I like Robo3T a lot so I'm using it. Also, it's open source.)

Here is a small video I found on YouTube that gives a 6 minute intro to Robo3T:

Once your MongoDB instance is up and running let's begin integrating MongoDB in our code as well.

In your root folder create a new folder called config. Inside that folder create a file called index.js and add the following content:

const config = {
  db: {
    url: 'localhost:27017',
    name: 'chatdb'
  }
}

export default config

Usually the default port that MongoDB instances will run on is 27017.

Here we set info about our database URL (which is in db) and the name of database which is chatdb (you can call this whatever you want).

Next create a new file called config/mongo.js and add the following content:

import mongoose from 'mongoose'
import config from './index.js'

const CONNECTION_URL = `mongodb://${config.db.url}/${config.db.name}`

mongoose.connect(CONNECTION_URL, {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

mongoose.connection.on('connected', () => {
  console.log('Mongo has connected succesfully')
})
mongoose.connection.on('reconnected', () => {
  console.log('Mongo has reconnected')
})
mongoose.connection.on('error', error => {
  console.log('Mongo connection has an error', error)
  mongoose.disconnect()
})
mongoose.connection.on('disconnected', () => {
  console.log('Mongo connection is disconnected')
})

Next import config/mongo.js in your server/index.js file like this:

.
.
// mongo connection
import "./config/mongo.js";
// routes
import indexRouter from "./routes/index.js";

If you get lost at any time, the entire source code for this tutorial is right here.

Let's discuss what we are doing here step by step:

We first import our config.js file in config/mongo.js. Next we pass in the value to our CONNECTION_URL like this:

const CONNECTION_URL = `mongodb://${config.db.url}/${config.db.name}`

Then using the CONNECTION_URL we form a Mongo connection, by doing this:

mongoose.connect(CONNECTION_URL, {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

This tells mongoose to make a connection with the database with our Node/Express application.

The options we are giving Mongo here are:

• useNewUrlParser: MongoDB driver has deprecated their current connection string parser. useNewUrlParser: true tells mongoose to use the new parser by Mongo. (If it's set to true, we have to provide a database port in the CONNECTION_URL.)
• useUnifiedTopology: False by default. Set to true to opt in to using MongoDB driver's new connection management engine. You should set this option to true, except for the unlikely case that it prevents you from maintaining a stable connection.

Next we simply add mongoose event handlers like this:

mongoose.connection.on('connected', () => {
  console.log('Mongo has connected succesfully')
})
mongoose.connection.on('reconnected', () => {
  console.log('Mongo has reconnected')
})
mongoose.connection.on('error', error => {
  console.log('Mongo connection has an error', error)
  mongoose.disconnect()
})
mongoose.connection.on('disconnected', () => {
  console.log('Mongo connection is disconnected')
})

• connected will be called once the database connection is established
• disconnected will be called when your Mongo connection is disabled
• error is called if there is an error connecting to your Mongo database
• reconnected event is called when the database loses connection and then makes an attempt to successfully reconnect.

Once you have this in place, simply go in your server/index.js file and import config/mongo.js. And that is it. Now when you start up your server by typing this:

npm start;

You should see something like this:

Logs when you start your server

If you see this you have successfully added Mongo to your application.

Congratulations!

If you got stuck here for some reason, let me know at twitter.com/adeelibr and I will try to sort it out for you. :)

Let's setup our first API section for users/

The setup of our API for users/ will have no authentication token for this tutorial, because my main focus is to teach you about the Chat application here.

User Modal Scheme

Let's create our first model (database scheme) for the user collection.

Create a new folder called models. Inside that folder create a file called User.js and add the following content:

import mongoose from "mongoose";
import { v4 as uuidv4 } from "uuid";

export const USER_TYPES = {
  CONSUMER: "consumer",
  SUPPORT: "support",
};

const userSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    firstName: String,
    lastName: String,
    type: String,
  },
  {
    timestamps: true,
    collection: "users",
  }
);

export default mongoose.model("User", userSchema);

Let's break this down into pieces:

export const USER_TYPES = {
  CONSUMER: "consumer",
  SUPPORT: "support",
};

We are basically going to have 2 types of users, consumer and support. I have written it this way because I want to programmatically ensure API and DB validation, which I will talk about later.

Next we create a schema on how a single document (object/item/entry/row) will look inside our user collection (a collection is equivalent to a MySQL table). We define it like this:

const userSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    firstName: String,
    lastName: String,
    type: String,
  },
  {
    timestamps: true,
    collection: "users",
  }
);

Here we are telling mongoose that for a single document in our users collection we want the structure to be like this:

{
    id: String // will get random string by default thanks to uuidv4
        firstName: String,
        lastName: String,
        type: String // this can be of 2 types consumer/support
}

In the second part of the schema we have something like this:

{
    timestamps: true,
    collection: "users",
}

Setting timestamps to true will add 2 things to my schema: a createdAt and a updatedAt date value. Every time when we create a new entry the createdAt will be updated automatically and updatedAt will update once we update an entry in the database using mongoose. Both of these are done automatically by mongoose.

The second part is collection. This shows what my collection name will be inside my database. I am assigning it the name of users.

And then finally we'll export the object like this:

export default mongoose.model("User", userSchema);

So mongoose.model takes in 2 parameters here.

• The name of the model, which is User here
• The schema associated with that model, which is userSchema in this case

Note: Based on the name of the model, which is User in this case, we don't add collection key in the schema section. It will take this User name and append an s to it and create a collection by its name, which becomes user.

Great, now we have our first model.

If you've gotten stuck anywhere, just have a look at the source code.

Create a new user API [POST request]

Next let's write our first controller for this route: .post('/', user.onCreateUser).

Go inside controllers/user.js and import 2 things at the top:

// utils
import makeValidation from '@withvoid/make-validation';
// models
import UserModel, { USER_TYPES } from '../models/User.js';

Here we are importing the validation library that I talked about in the video at the very top. We are also importing our user modal along with the USER_TYPES from the same file.

This is what USER_TYPES represents:

export const USER_TYPES = {
  CONSUMER: "consumer",
  SUPPORT: "support",
};

Next find the controller onCreateUser and add the following content to it:

onCreateUser: async (req, res) => {
    try {
      const validation = makeValidation(types => ({
        payload: req.body,
        checks: {
          firstName: { type: types.string },
          lastName: { type: types.string },
          type: { type: types.enum, options: { enum: USER_TYPES } },
        }
      }));
      if (!validation.success) return res.status(400).json(validation);

      const { firstName, lastName, type } = req.body;
      const user = await UserModel.createUser(firstName, lastName, type);
      return res.status(200).json({ success: true, user });
    } catch (error) {
      return res.status(500).json({ success: false, error: error })
    }
  },

Let's divide this into 2 sections.

First we validate the user response by doing this:

const validation = makeValidation(types => ({
  payload: req.body,
  checks: {
    firstName: { type: types.string },
    lastName: { type: types.string },
    type: { type: types.enum, options: { enum: USER_TYPES } },
  }
}));
if (!validation.success) return res.status(400).json({ ...validation });

Please make sure that you have seen the video (above) on validate an API request in Node using custom validation or by using make-validation library.

Here we are using the [make-validation](https://www.npmjs.com/package/@withvoid/make-validation) library (that I ended up making while writing this tutorial). I talk about it's usage in the video at the start of this tutorial.

All we are doing here is passing req.body to payload. Then in the checks we're adding an object where against each key we are telling what are the requirements for each type, for example:

firstName: { type: types.string },

Here we are telling it that firstName is of type string. If the user forgets to add this value while hitting the API, or if the type is not string, it will throw an error.

The validation variable will return an object with 3 things: {success: boolean, message: string, errors: object}.

If validation.success is false we simply return everything from the validation and give it to the user with a status code of 400.

Once our validation is in place and we know that the data we are getting are valid, then we do the following:

const { firstName, lastName, type } = req.body;
const user = await UserModel.createUser(firstName, lastName, type);
return res.status(200).json({ success: true, user });

Then we destruct firstName, lastName, type from req.body and pass those values to our UserModel.createUser. If everything goes right, it simply returns success: true with the new user created along with a status 200.

If anywhere in this process anything goes wrong, it throws an error and goes to the catch block:

catch (error) {
  return res.status(500).json({ success: false, error: error })
}

There we simply return an error message along with the HTTP status 500.

The only thing we are missing here is the UserModel.createUser() method.

So let's go back into our models/User.js file and add it:

userSchema.statics.createUser = async function (
    firstName, 
        lastName, 
        type
) {
  try {
    const user = await this.create({ firstName, lastName, type });
    return user;
  } catch (error) {
    throw error;
  }
}


export default mongoose.model("User", userSchema);

So all we are doing here is adding a static method to our userSchema called createUser that takes in 3 parameters: firstName, lastName, type.

Next we use this:

const user = await this.create({ firstName, lastName, type });

Here the this part is very important, since we are writing a static method on userSchema. Writing this will ensure that we are using performing operations on the userSchema object

One thing to note here is that userSchema.statics.createUser = async function (firstName, lastName, type) => {} won't work. If you use an => arrow function the this context will be lost and it won't work.

If you want to learn more about static methods in mongoose, see this very short but helpful doc example here.

Now that we have everything set up, let's start our terminal by running the following command in the project's root folder:

npm start;

Go into postman, set up a POST request on this API http://localhost:3000/users, and add the following body to the API:

{
    firstName: 'John'
        lastName: 'Doe',
        type: 'consumer'
}

Like this:

You can also get the entire postman API collection from here so that you don't have to write the APIs again and again.

Awesome – we just ended up creating our first API. Let's create a couple more user APIs before we move to the chat part because there is no chat without users (unless we have robots, but robots are users as well ?).

Get a user by its ID API [GET request]

Next we need to write an API that gets us a user by its ID. So for our route .get('/:id', user.onGetUserById) let's write down its controller.

Go to controllers/user.js and for the method onGetUserById write this:

onGetUserById: async (req, res) => {
  try {
    const user = await UserModel.getUserById(req.params.id);
    return res.status(200).json({ success: true, user });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Cool, this looks straightforward. Let's add UserModel.getUserById() in our models/User.js file.

Add this method below the last static method you wrote:

userSchema.statics.getUserById = async function (id) {
  try {
    const user = await this.findOne({ _id: id });
    if (!user) throw ({ error: 'No user with this id found' });
    return user;
  } catch (error) {
    throw error;
  }
}

We pass in an id parameter and we wrap our function in try/catch. This is very important when you are using async/await. The lines to focus on here are these 2:

const user = await this.findOne({ _id: id });
if (!user) throw ({ error: 'No user with this id found' });

We use mongoose's findOne method to find an entry by id. We know that only one item exists in the collection by this id because the id is unique. If no user is found we simply throw an error with the message No user with this id found.

And that is it! Let's start up our server:

npm start;

Open up postman and create a GET request http://localhost:3000/users/:id.

Note: I am using the ID of the last user we just created.

Nicely done! Good job.

Two more API's to go for our user section.

Get all users API [GET request]

For our router in .get('/', user.onGetAllUsers) let's add information to its controller.

Go to controllers/user.js and add code in the onGetAllUsers() method:

onGetAllUsers: async (req, res) => {
  try {
    const users = await UserModel.getUsers();
    return res.status(200).json({ success: true, users });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Next let's create the static method for getUsers() in the models/User.js file. Below the last static method you wrote in that file, type:

userSchema.statics.getUsers = async function () {
  try {
    const users = await this.find();
    return users;
  } catch (error) {
    throw error;
  }
}

We use the mongoose method called await this.find(); to get all the records for our users collection and return it.

Note: I am not handling pagination in our users API because that's not the main focus here. I'll talk about pagination once we move towards our chat APIs.

Let's start our server:

npm start;

Open up postman and create a GET request for this route http://localhost:3000/users:

I went ahead and ended up creating a couple more users. ?

Delete a user by ID API [DELETE request] (More of a bonus section, you can skip this if you want)

Let's create our final route to delete a user by their ID. For the route .delete('/:id', user.onDeleteUserById) go to its controller in controllers/user.js and write this code in the onDeleteUserById() method:

onDeleteUserById: async (req, res) => {
  try {
    const user = await UserModel.deleteByUserById(req.params.id);
    return res.status(200).json({ 
      success: true, 
      message: `Deleted a count of ${user.deletedCount} user.` 
    });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Let's add the static method deleteByUserById in models/User.js:

userSchema.statics.deleteByUserById = async function (id) {
  try {
    const result = await this.remove({ _id: id });
    return result;
  } catch (error) {
    throw error;
  }
}

We pass in the id here as a parameter and then use the mongoose method called this.remove to delete a record item from a specific collection. In this case, it's the users collection.

Let's start up our server:

npm start;

Go to postman and create a new DELETE route:

With this we'll conclude our USER API section.

Next we will cover how to authenticate routes with an authentication token. This is the last thing I want to touch on before moving on to the chat section – because all of the chat APIs will be authenticated.

What are middlewares in ExpressJS?

How can we write them? By adding JWT middleware in your application:

And here's the GitHub link to the entire source code of this video [Chapter 0].

And again, all the relevant info can be found in the READ.ME.

Coming back to our code base, let's create a JWT middleware to authenticate our routes. Go to middlewares/jwt.js and add the following:

import jwt from 'jsonwebtoken';
// models
import UserModel from '../models/User.js';

const SECRET_KEY = 'some-secret-key';

export const encode = async (req, res, next) => {
  try {
    const { userId } = req.params;
    const user = await UserModel.getUserById(userId);
    const payload = {
      userId: user._id,
      userType: user.type,
    };
    const authToken = jwt.sign(payload, SECRET_KEY);
    console.log('Auth', authToken);
    req.authToken = authToken;
    next();
  } catch (error) {
    return res.status(400).json({ success: false, message: error.error });
  }
}

export const decode = (req, res, next) => {
  if (!req.headers['authorization']) {
    return res.status(400).json({ success: false, message: 'No access token provided' });
  }
  const accessToken = req.headers.authorization.split(' ')[1];
  try {
    const decoded = jwt.verify(accessToken, SECRET_KEY);
    req.userId = decoded.userId;
    req.userType = decoded.type;
    return next();
  } catch (error) {

    return res.status(401).json({ success: false, message: error.message });
  }
}

Let's discuss the encode method first:

export const encode = async (req, res, next) => {
  try {
    const { userId } = req.params;
    const user = await UserModel.getUserById(userId);
    const payload = {
      userId: user._id,
      userType: user.type,
    };
    const authToken = jwt.sign(payload, SECRET_KEY);
    console.log('Auth', authToken);
    req.authToken = authToken;
    next();
  } catch (error) {
    return res.status(400).json({ 
        success: false, message: error.error 
    });
  }
}

Let's go through it step by step.

We get the userId from our req.params. If you remember from the video earlier, req.params is the /:<identifier> defined in our routes section.

Next we use the const user = await UserModel.getUserById(userId); method we just created recently to get user information. If it exists, that is – otherwise this line will throw an error and it will directly go to the catch block where we will return the user with a 400 response and and an error message.

But if we get a response from the getUserById method we then make a payload:

const payload = {
      userId: user._id,
      userType: user.type,
};

Next we sign that payload in JWT using the following:

const authToken = jwt.sign(payload, SECRET_KEY);

Once we have the JWT signed we then do this:

req.authToken = authToken;
next();

Set it to our req.authToken and then forward this information as next().

Next let's talk about the decode method:

export const decode = (req, res, next) => {
  if (!req.headers['authorization']) {
    return res.status(400).json({ success: false, message: 'No access token provided' });
  }
  const accessToken = req.headers.authorization.split(' ')[1];
  try {
    const decoded = jwt.verify(accessToken, SECRET_KEY);
    req.userId = decoded.userId;
    req.userType = decoded.type;
    return next();
  } catch (error) {

    return res.status(401).json({ success: false, message: error.message });
  }
}

Let's break this down:

if (!req.headers['authorization']) {
  return res.status(400).json({ 
      success: false, 
        message: 'No access token provided' 
  });
}

First we check if the authorization header is present or not. If not we simply return an error message to user.

Then we do this:

const accessToken = req.headers.authorization.split(' ')[1];

It's being split(' ') by space and then we are getting the second index of the array by accessing its [1] index because the convention is authorization: Bearer <auth-token>. Want to read more on this? Check out this nice thread on quora.

Then we try to decode our token:

try {
  const decoded = jwt.verify(accessToken, SECRET_KEY);
  req.userId = decoded.userId;
  req.userType = decoded.type;
  return next();
} catch (error) {
  return res.status(401).json({ 
      success: false, message: error.message 
  });
}

If this is not successful jwt.verify(accessToken, SECRET_KEY) will simply throw an error and our code will go in the catch block immediately. If it is successful, then we can decode it. We get userId and type from the token and save it as req.userId, req.userType and simply hit next().

Now, moving forward, every route that goes through this decode middleware will have the current user's id & it's type.

This was it for the middleware section. Let's create a login route so that we can ask a user for their information and give a token in return (because moving forward they'll need a token to access the rest of chat APIs).

Creating a login route [POST request]

Go to your routes/index.js file and paste the following content:

import express from 'express';
// middlewares
import { encode } from '../middlewares/jwt.js';

const router = express.Router();

router
  .post('/login/:userId', encode, (req, res, next) => {
    return res
      .status(200)
      .json({
        success: true,
        authorization: req.authToken,
      });
  });

export default router;

So all we are doing is adding the encode middleware to our http://localhost:3000/login/:<user-id> [POST] route. If everything goes smoothly the user will get an authorization token.

Note: I am not adding a login/signup flow, but I still wanted to touch on JWT/middleware in this tutorial.

Usually authentication is done in a similar way. The only addition here is that the user doesn't provide their ID. They provide their username, password (which we verify in the database), and if everything checks out we give them an authorization token.

If you got stuck anywhere up to this point, just write to me at twitter.com/adeelibr, so that way I can improve the content. You can also write to me if you would like to learn something else.

As a reminder, the entire source code is available here. You don't have to code along with this tutorial, but if you do the concepts will stick better.

Let's just check our /login route now.

Start your server:

npm start;

Let's run postman. Create a new POST request http://localhost:3000/login/<user-id>:

When the user ID is correct

When the user ID is invalid

With this we are done with our login flow as well.

This was a lot. But now we can focus only on our chat routes.

Create a web socket class

This web socket class will handle events when a user disconnects, joins a chat room, or wants to mute a chat room.

So let's create a web-socket class that will manage sockets for us. Create a new folder called utils. Inside that folder create a file called WebSockets.js and add the following content:

class WebSockets {
  users = [];
  connection(client) {
    // event fired when the chat room is disconnected
    client.on("disconnect", () => {
      this.users = this.users.filter((user) => user.socketId !== client.id);
    });
    // add identity of user mapped to the socket id
    client.on("identity", (userId) => {
      this.users.push({
        socketId: client.id,
        userId: userId,
      });
    });
    // subscribe person to chat & other user as well
    client.on("subscribe", (room, otherUserId = "") => {
      this.subscribeOtherUser(room, otherUserId);
      client.join(room);
    });
    // mute a chat room
    client.on("unsubscribe", (room) => {
      client.leave(room);
    });
  }

  subscribeOtherUser(room, otherUserId) {
    const userSockets = this.users.filter(
      (user) => user.userId === otherUserId
    );
    userSockets.map((userInfo) => {
      const socketConn = global.io.sockets.connected(userInfo.socketId);
      if (socketConn) {
        socketConn.join(room);
      }
    });
  }
}

export default new WebSockets();

The WebSockets class has three major things here:

• users array
• connection method
• subscribing members of a chat room to it. subscribeOtherUser

Let's break this down.

We have a class:

class WebSockets {

}

export default new WebSocket();

We create a class and export an instance of that class.

Inside the class we have an empty users array. This array will hold a list of all the active users that are online using our application.

Next we have a connection method, the core of this class:

connection(client) {
  // event fired when the chat room is disconnected
  client.on("disconnect", () => {
    this.users = this.users.filter((user) => user.socketId !== client.id);
  });
  // add identity of user mapped to the socket id
  client.on("identity", (userId) => {
    this.users.push({
      socketId: client.id,
      userId: userId,
    });
  });
  // subscribe person to chat & other user as well
  client.on("subscribe", (room, otherUserId = "") => {
    this.subscribeOtherUser(room, otherUserId);
    client.join(room);
  });
  // mute a chat room
  client.on("unsubscribe", (room) => {
    client.leave(room);
  });
}

The connection method takes in a parameter called client (client here will be our server instance, I will talk more about this in a bit).

We take the param client and add some event to it

• client.on('disconnect') // when a user connection is lost this method will be called
• client.on('identity') // when user logs in from the front end they will make a connection with our server by giving their identity
• client.on('subscribe') // when a user joins a chat room this method is called
• client.on('unsubscribe') // when a user leaves or wants to mute a chat room

Let's talk about disconnect:

client.on("disconnect", () => {
  this.users = this.users.filter((user) => user.socketId !== client.id);
});

As soon as the connection is disconnected, we run a filter on users array. Where we find user.id === client.id we remove it from our sockets array. ( client here is coming from the function param.)

Let's talk about identity:

client.on("identity", (userId) => {
  this.users.push({
    socketId: client.id,
    userId: userId,
  });
});

When a user logs in through he front end application web/android/ios they will make a socket connection with our backend app and call this identity method. They'll also send their own user id.

We will take that user id and the client id (the user's own unique socket id that socket.io creates when they make a connection with our BE).

Next we have unsubscribe:

client.on("unsubscribe", (room) => {
  client.leave(room);
});

The user passes in the room id and we just tell client.leave() to remove the current user calling this method from a particular chat room.

Next we have subscribe:

client.on("subscribe", (room, otherUserId = "") => {
  this.subscribeOtherUser(room, otherUserId);
  client.join(room);
});

When a user joins a chat room, they will tell us about the room they want to join along with the other person who is part of that chat room.

Note: We will see later that when we initiate a chat room we get all the users associated with that room in the API response.

In my opinion: Another thing we could have done here was when the user sends in the room number, we can make a DB query to see all the members of the chat room and make them join if they are online at the moment (that is, in our users list).

The subscribeOtherUser method is defined like this:

subscribeOtherUser(room, otherUserId) {
  const userSockets = this.users.filter(
    (user) => user.userId === otherUserId
  );
  userSockets.map((userInfo) => {
    const socketConn = global.io.sockets.connected(userInfo.socketId);
    if (socketConn) {
      socketConn.join(room);
    }
  });
}

We pass in room and otherUserId as params to this function.

Using the otherUserId we filter on our this.users array and all the results that match are stored in userSockets array.

You might be thinking – how can one user have multiple presences in the user array? Well, think of a scenario where the same user is logged in from both their web application and mobile phone. It will create multiple socket connections for the same user.

Next we map on userSockets. For each item in this array we pass it into this method: const socketConn = global.io.sockets.connected(userInfo.socketId)

I will talk more about this global.io.sockets.connected in a bit. But what this initially does is it takes in userInfo.socketId and if it exists in our socket connection, it will return the connection, otherwise null.

Next we simply see if socketConn is available. If so, we take that socketConn and make this connection join the room passed in the function:

if (socketConn) {
    socketConn.join(room);
}

And this is it for our WebSockets class.

Let's import this file in our server/index.js file:

import socketio from "socket.io";
// mongo connection
import "./config/mongo.js";
// socket configuration
import WebSockets from "./utils/WebSockets.js";

So just import socket.io and import WebSockets somewhere at the top.

Next where we are creating our server add the content below this:

/** Create HTTP server. */
const server = http.createServer(app);
/** Create socket connection */
global.io = socketio.listen(server);
global.io.on('connection', WebSockets.connection)

The server was created and we do two things:

• assign global.io to socketio.listen(server) (As soon as a port starts listening on the server, sockets starts listening for events happening on that port as well.)
• then we assign global.io.on('connection', WebSockets.connection) method. Every time someone from the front end makes a socket connection, the connection method will be called which will invoke our Websockets class and inside that class the connection method.

global.io is equivalent to windows object in browser. But since we don't have windows in NodeJS we use global.io. Whatever we put in global.io is available in the entire application.

This is the same global.io we used in the WebSockets class inside the subscribeOtherUser method.

If you got lost here is the entire source code of this chat application. Also free to drop me a message with your feedback and I will try to improve the content of this tutorial.

Discussing chat room & chat message database model

Before starting off with Chat, I think it is really important to discuss the database model on which we will create our chat application. Have a look at the below video:

Now that you have a clear idea about what our chat structure will be like, let's start off by making our chat room model.

Go inside your models folder and create the following ChatRoom.js. Add the following content to it:

import mongoose from "mongoose";
import { v4 as uuidv4 } from "uuid";

export const CHAT_ROOM_TYPES = {
  CONSUMER_TO_CONSUMER: "consumer-to-consumer",
  CONSUMER_TO_SUPPORT: "consumer-to-support",
};

const chatRoomSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    userIds: Array,
    type: String,
    chatInitiator: String,
  },
  {
    timestamps: true,
    collection: "chatrooms",
  }
);

chatRoomSchema.statics.initiateChat = async function (
    userIds, type, chatInitiator
) {
  try {
    const availableRoom = await this.findOne({
      userIds: {
        $size: userIds.length,
        $all: [...userIds],
      },
      type,
    });
    if (availableRoom) {
      return {
        isNew: false,
        message: 'retrieving an old chat room',
        chatRoomId: availableRoom._doc._id,
        type: availableRoom._doc.type,
      };
    }

    const newRoom = await this.create({ userIds, type, chatInitiator });
    return {
      isNew: true,
      message: 'creating a new chatroom',
      chatRoomId: newRoom._doc._id,
      type: newRoom._doc.type,
    };
  } catch (error) {
    console.log('error on start chat method', error);
    throw error;
  }
}

export default mongoose.model("ChatRoom", chatRoomSchema);

We have three things going on here:

• We have a const for CHAT_ROOM_TYPES which has only two types
• We define our ChatRoom schema
• We add a static method to initiate chat

Chat related APIs

Initiate a chat between users (/room/initiate [POST request])

Let's discuss our static method defined in models/ChatRoom.js called initiateChat:

chatRoomSchema.statics.initiateChat = async function (userIds, type, chatInitiator) {
  try {
    const availableRoom = await this.findOne({
      userIds: {
        $size: userIds.length,
        $all: [...userIds],
      },
      type,
    });
    if (availableRoom) {
      return {
        isNew: false,
        message: 'retrieving an old chat room',
        chatRoomId: availableRoom._doc._id,
        type: availableRoom._doc.type,
      };
    }

    const newRoom = await this.create({ userIds, type, chatInitiator });
    return {
      isNew: true,
      message: 'creating a new chatroom',
      chatRoomId: newRoom._doc._id,
      type: newRoom._doc.type,
    };
  } catch (error) {
    console.log('error on start chat method', error);
    throw error;
  }
}

This function takes in three parameters:

• userIds (array of users)
• type (type of chatroom)
• chatInitiator (the user who created the chat room)

Next we are doing two things here: either returning an existing chatroom document or creating a new one.

Let's break this one down:

const availableRoom = await this.findOne({
  userIds: {
    $size: userIds.length,
    $all: [...userIds],
  },
  type,
});
if (availableRoom) {
  return {
    isNew: false,
    message: 'retrieving an old chat room',
    chatRoomId: availableRoom._doc._id,
    type: availableRoom._doc.type,
  };
}

First using the this.findOne() API in mongoose, we find all the chatrooms where the following criteria is met:

userIds: { $size: userIds.length, $all: [...userIds] },
type: type,

You can read more on the $size operator here, and more on the $all operator here.

We're checking to find a chatroom document where an item exists in our chatrooms collection where

1. the userIds are the same as the one we are passing to this function (irrespective of the user ids order), and
2. the length of the userIds is the same as that my userIds.length that we are passing through the function.

Also we're checking that the chat room type should be the same.

If something like this is found, we simply return the existing chatroom.

Otherwise we create a new chat room and return it by doing this:

const newRoom = await this.create({ userIds, type, chatInitiator });
return {
  isNew: true,
  message: 'creating a new chatroom',
  chatRoomId: newRoom._doc._id,
  type: newRoom._doc.type,
};

Create a new room and return the response.

We also have an isNew key where, if it's retrieving an old chatroom, we set it to false otherwise true.

Next for your route created in routes/chatRoom.js called post('/initiate', chatRoom.initiate) go to its appropriate controller in controllers/chatRoom.js and add the following in the initiate method:

initiate: async (req, res) => {
  try {
    const validation = makeValidation(types => ({
      payload: req.body,
      checks: {
        userIds: { 
          type: types.array, 
          options: { unique: true, empty: false, stringOnly: true } 
        },
        type: { type: types.enum, options: { enum: CHAT_ROOM_TYPES } },
      }
    }));
    if (!validation.success) return res.status(400).json({ ...validation });

    const { userIds, type } = req.body;
    const { userId: chatInitiator } = req;
    const allUserIds = [...userIds, chatInitiator];
    const chatRoom = await ChatRoomModel.initiateChat(allUserIds, type, chatInitiator);
    return res.status(200).json({ success: true, chatRoom });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

We are using the [make-validation](https://www.npmjs.com/package/@withvoid/make-validation) library here to validate the user's request. For the initiate API, we expect the user to send an array of users and also define the type of the chat-room that is being created.

Once the validation passes, then:

const { userIds, type } = req.body;
const { userId: chatInitiator } = req;
const allUserIds = [...userIds, chatInitiator];
const chatRoom = await ChatRoomModel.initiateChat(allUserIds, type, chatInitiator);
return res.status(200).json({ success: true, chatRoom });

One thing to notice here is userIds, type is coming from req.body while userId that is being aliased as chatInitiatorId is coming from req thanks to our decode middleware.

If you remember, we attached app.use("/room", decode, chatRoomRouter); in our server/index.js file. This means this route /room/initiate is authenticated. So const { userId: chatInitiator } = req; is the id of the current user logged in.

We simply call our initiateChat method from ChatRoomModel and pass it allUserIds, type, chatInitiator. Whatever result comes we simply pass it to the user.

Let's run this and see if it works (here is a video of me doing it):

Create a message in chat room (/:roomId/message) [POST request]

Let's create a message for the chat room we just created with pikachu.

But before we create a message we need to create a model for our chatmessages. So let's do that first. In your models folder create a new file called ChatMessage.js and add the following content to it:

import mongoose from "mongoose";
import { v4 as uuidv4 } from "uuid";

const MESSAGE_TYPES = {
  TYPE_TEXT: "text",
};

const readByRecipientSchema = new mongoose.Schema(
  {
    _id: false,
    readByUserId: String,
    readAt: {
      type: Date,
      default: Date.now(),
    },
  },
  {
    timestamps: false,
  }
);

const chatMessageSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    chatRoomId: String,
    message: mongoose.Schema.Types.Mixed,
    type: {
      type: String,
      default: () => MESSAGE_TYPES.TYPE_TEXT,
    },
    postedByUser: String,
    readByRecipients: [readByRecipientSchema],
  },
  {
    timestamps: true,
    collection: "chatmessages",
  }
);

chatMessageSchema.statics.createPostInChatRoom = async function (chatRoomId, message, postedByUser) {
  try {
    const post = await this.create({
      chatRoomId,
      message,
      postedByUser,
      readByRecipients: { readByUserId: postedByUser }
    });
    const aggregate = await this.aggregate([
      // get post where _id = post._id
      { $match: { _id: post._id } },
      // do a join on another table called users, and 
      // get me a user whose _id = postedByUser
      {
        $lookup: {
          from: 'users',
          localField: 'postedByUser',
          foreignField: '_id',
          as: 'postedByUser',
        }
      },
      { $unwind: '$postedByUser' },
      // do a join on another table called chatrooms, and 
      // get me a chatroom whose _id = chatRoomId
      {
        $lookup: {
          from: 'chatrooms',
          localField: 'chatRoomId',
          foreignField: '_id',
          as: 'chatRoomInfo',
        }
      },
      { $unwind: '$chatRoomInfo' },
      { $unwind: '$chatRoomInfo.userIds' },
      // do a join on another table called users, and 
      // get me a user whose _id = userIds
      {
        $lookup: {
          from: 'users',
          localField: 'chatRoomInfo.userIds',
          foreignField: '_id',
          as: 'chatRoomInfo.userProfile',
        }
      },
      { $unwind: '$chatRoomInfo.userProfile' },
      // group data
      {
        $group: {
          _id: '$chatRoomInfo._id',
          postId: { $last: '$_id' },
          chatRoomId: { $last: '$chatRoomInfo._id' },
          message: { $last: '$message' },
          type: { $last: '$type' },
          postedByUser: { $last: '$postedByUser' },
          readByRecipients: { $last: '$readByRecipients' },
          chatRoomInfo: { $addToSet: '$chatRoomInfo.userProfile' },
          createdAt: { $last: '$createdAt' },
          updatedAt: { $last: '$updatedAt' },
        }
      }
    ]);
    return aggregate[0];
  } catch (error) {
    throw error;
  }
}

export default mongoose.model("ChatMessage", chatMessageSchema);

There are a couple of things happening here:

• We have a MESSAGE_TYPES object which has only one type called text
• We are defining our schema for chatmessage and readByRecipient
• Then we are writing our static method for createPostInChatRoom

I know this is a lot of content, but just bear with me. Let's just write the controller for the route that creates this message.

For the route defined in our routes/chatRoom.js API called .post('/:roomId/message', chatRoom.postMessage) let's go to its controller in controllers/chatRoom.js and define it:

postMessage: async (req, res) => {
  try {
    const { roomId } = req.params;
    const validation = makeValidation(types => ({
      payload: req.body,
      checks: {
        messageText: { type: types.string },
      }
    }));
    if (!validation.success) return res.status(400).json({ ...validation });

    const messagePayload = {
      messageText: req.body.messageText,
    };
    const currentLoggedUser = req.userId;
    const post = await ChatMessageModel.createPostInChatRoom(roomId, messagePayload, currentLoggedUser);
    global.io.sockets.in(roomId).emit('new message', { message: post });
    return res.status(200).json({ success: true, post });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Cool, let's discuss what we are doing here:

Operators discussed in this video are:

• $match
• $last
• $addToSet
• $lookup
• $unwind
• $group

See conversation for a chat room by it's id [Get request]

Now that we have

• Created a chat room
• Are able to add messages in that chat room

Let's see the entire conversation for that chat as well (with pagination).

For your route .get('/:roomId', chatRoom.getConversationByRoomId) in routes/chatRoom.js open its controller in the file controllers/chatRoom.js and add the following content to the chat room:

getConversationByRoomId: async (req, res) => {
  try {
    const { roomId } = req.params;
    const room = await ChatRoomModel.getChatRoomByRoomId(roomId)
    if (!room) {
      return res.status(400).json({
        success: false,
        message: 'No room exists for this id',
      })
    }
    const users = await UserModel.getUserByIds(room.userIds);
    const options = {
      page: parseInt(req.query.page) || 0,
      limit: parseInt(req.query.limit) || 10,
    };
    const conversation = await ChatMessageModel.getConversationByRoomId(roomId, options);
    return res.status(200).json({
      success: true,
      conversation,
      users,
    });
  } catch (error) {
    return res.status(500).json({ success: false, error });
  }
},

Next let's create a new static method in our ChatRoomModel file called getChatRoomByRoomId in models/ChatRoom.js:

chatRoomSchema.statics.getChatRoomByRoomId = async function (roomId) {
  try {
    const room = await this.findOne({ _id: roomId });
    return room;
  } catch (error) {
    throw error;
  }
}

Very straightforward – we are getting the room by roomId here.

Next in our UserModel, create a static method called getUserByIds in the file models/User.js:

userSchema.statics.getUserByIds = async function (ids) {
  try {
    const users = await this.find({ _id: { $in: ids } });
    return users;
  } catch (error) {
    throw error;
  }
}

The operator used here is $in – I'll talk about this in a bit.

And then at last, go to your ChatMessage model in models/ChatMessage.js and write a new static method called getConversationByRoomId:

chatMessageSchema.statics.getConversationByRoomId = async function (chatRoomId, options = {}) {
  try {
    return this.aggregate([
      { $match: { chatRoomId } },
      { $sort: { createdAt: -1 } },
      // do a join on another table called users, and 
      // get me a user whose _id = postedByUser
      {
        $lookup: {
          from: 'users',
          localField: 'postedByUser',
          foreignField: '_id',
          as: 'postedByUser',
        }
      },
      { $unwind: "$postedByUser" },
      // apply pagination
      { $skip: options.page * options.limit },
      { $limit: options.limit },
      { $sort: { createdAt: 1 } },
    ]);
  } catch (error) {
    throw error;
  }
}

Let's discuss all that we have done so far:

All the source code is available here.

Mark an entire conversation as read (feature similar to WhatsApp)

Once the other person is logged in and they view a conversation for a room id, we need to mark that conversation as read from their side.

To do this, in your routes/chatRoom.js for the route

put('/:roomId/mark-read', chatRoom.markConversationReadByRoomId)

go to its appropriate controller in controllers/chatRoom.js and add the following content in the markConversationReadByRoomId controller.

markConversationReadByRoomId: async (req, res) => {
  try {
    const { roomId } = req.params;
    const room = await ChatRoomModel.getChatRoomByRoomId(roomId)
    if (!room) {
      return res.status(400).json({
        success: false,
        message: 'No room exists for this id',
      })
    }

    const currentLoggedUser = req.userId;
    const result = await ChatMessageModel.markMessageRead(roomId, currentLoggedUser);
    return res.status(200).json({ success: true, data: result });
  } catch (error) {
    console.log(error);
    return res.status(500).json({ success: false, error });
  }
},

All we are doing here is first checking if the room exists or not. If it does, we proceed further. We take in the req.user.id as currentLoggedUser and pass it to the following function:

ChatMessageModel.markMessageRead(roomId, currentLoggedUser);

Which in our ChatMessage model is defined like this:

chatMessageSchema.statics.markMessageRead = async function (chatRoomId, currentUserOnlineId) {
  try {
    return this.updateMany(
      {
        chatRoomId,
        'readByRecipients.readByUserId': { $ne: currentUserOnlineId }
      },
      {
        $addToSet: {
          readByRecipients: { readByUserId: currentUserOnlineId }
        }
      },
      {
        multi: true
      }
    );
  } catch (error) {
    throw error;
  }
}

A possible use case is that the user might not have read the last 15 messages once they open up a specific room conversation. They should all be marked as read. So we're using the this.updateMany function by mongoose.

The query itself is defined in 2 steps:

• Find
• Update

And there can be multiple statements be updated.

To find a section, do this:

{
  chatRoomId,
  'readByRecipients.readByUserId': { $ne: currentUserOnlineId }
},

This says I want to find all the message posts in the chatmessages collection where chatRoomId matches and readByRecipients array does not. The userId that I am passing to this function is currentUserOnlineId.

Once it has all those documents where the criteria matches, it's then time to update them:

{
  $addToSet: {
    readByRecipients: { readByUserId: currentUserOnlineId }
  }
},

$addToSet will just push a new entry to the readByRecipients array. This is like Array.push but for mongo.

Next we want to tell mongoose to not just update the first record it finds, but also to update all the records where the condition matches. So doing this:

{
  multi: true
}

And that is all – we return the data as is.

Let's run this API.

Start up the server:

npm start;

Open your postman and create a new PUT request to test this route ocalhost:3000/room/<room=id-here>/mark-read:

Bonus Section

• How to delete a chat room and all its related messages
• How to delete a message by its message id

And we are done! Wow that was a lot of learning today.

You can find the source code of this tutorial here.

Reach out to me on twitter with your feedback – I would love to hear if you have any suggestions for improvements: twitter.com/adeelibr

If you liked to this article, please do give the github repository a star and subscribe to my youtube channel.

I'm a tabletop game developer, and am continually looking for ways to digitize game experiences. In this tutorial, we're going to build a multiplayer card game using Phaser 3, Express, and Socket.IO.

In terms of prerequisites, you'll want to make sure that you have Node/NPM and Git installed and configured on your machine. Some experience with JavaScript would be helpful, and you may want to run through the basic Phaser tutorial before tackling this one.

Major kudos to Scott Westover for his tutorial on the topic, Kal_Torak and the Phaser community for answering all my questions, and my good friend Mike for helping me conceptualize the architecture of this project.

Note: we'll be using assets and colors from my tabletop card game, Entromancy: Hacker Battles. If you prefer, you can use your own images (or even Phaser rectangles) and colors, and you can access the entire project code on GitHub.

If you'd prefer a more visual tutorial, you can also follow along with the companion video to this article:

Let's get started!

The Game

Our simple card game will feature a Phaser client that will handle most of the game logic and doing things like dealing cards, providing drag-and-drop functionality, and so on.

On the back end, we'll spin up an Express server that will utilize Socket.IO to communicate between clients and make it so that when one player plays a card, it shows up in another player's client, and vice-versa.

Our goal for this project is to create a basic framework for a multiplayer card game that you can build upon and adjust to suit your own game's logic.

First, let's tackle the client!

The Client

To scaffold our client, we're going to clone the semi-official Phaser 3 Webpack Project Template on GitHub.

Open your favorite command line interface and create a new folder:

mkdir multiplayer-card-project
cd multiplayer-card-project

Clone the git project:

git clone https://github.com/photonstorm/phaser3-project-template.git

This command will download the template in a folder called "phaser3-project-template" within /multiplayer-card-project. If you want to follow along with our tutorial's file structure, go ahead and change that template folder's name to "client."

Navigate into that new directory and install all dependencies:

cd client
npm install

Your project folder structure should look something like this:

Before we muck with the files, let's go back to our CLI and enter the following command in the /client folder:

npm start

Our Phaser template utilizes Webpack to spin up a local server that in turn serves up a simple game app in our browser (usually at http://localhost:8080). Neat!

Let's open our project in your favorite code editor and make some changes to fit our card game. Delete everything in /client/src/assets and replace them with the card images from GitHub.

In the /client/src directory, add a folder called "scenes" and another called "helpers."

In /client/src/scenes, add an empty file called "game.js".

In /client/src/helpers, add three empty files: "card.js", "dealer.js", and "zone.js".

Your project structure should now look like this:

Cool! Your client might be throwing you errors because we deleted some things, but not to worry. Open /src/index.js, which is the main entry point to our front end app. Enter the following code:

import Phaser from "phaser";
import Game from "./scenes/game";

const config = {
    type: Phaser.AUTO,
    parent: "phaser-example",
    width: 1280,
    height: 780,
    scene: [
        Game
    ]
};

const game = new Phaser.Game(config);

All we've done here is restructure the boilerplate to utilize Phaser's "scene" system so that we can separate our game scenes rather than try to cram everything in one file. Scenes can be useful if you're creating multiple game worlds, building things like instruction screens, or generally trying to keep things tidy.

Let's move to /src/scenes/game.js and write some code:

export default class Game extends Phaser.Scene {
    constructor() {
        super({
            key: 'Game'
        });
    }

    preload() {
        this.load.image('cyanCardFront', 'src/assets/CyanCardFront.png');
        this.load.image('cyanCardBack', 'src/assets/CyanCardBack.png');
        this.load.image('magentaCardFront', 'src/assets/MagentaCardFront.png');
        this.load.image('magentaCardBack', 'src/assets/MagentaCardBack.png');
    }

    create() {
        this.dealText = this.add.text(75, 350, ['DEAL CARDS']).setFontSize(18).setFontFamily('Trebuchet MS').setColor('#00ffff').setInteractive();
    }

    update() {

    }
}

We're taking advantage of ES6 classes to create a new Game scene, which incorporates preload(), create() and update() functions.

preload() is used to...well...preload any assets that we'll be using for our game.

create() is run when the game starts up, and where we'll be establishing much of our user interface and game logic.

update() is called once per frame, and we won't be making use of it in our tutorial (but it may be useful in your own game depending on its requirements).

Within the create() function, we've created a bit of text that says "DEAL CARDS" and set it to be interactive:

Very cool. Let's create a bit of placeholder code to understand how we want this whole thing to work once it's up and running. Add the following to your create() function:

        let self = this;

        this.card = this.add.image(300, 300, 'cyanCardFront').setScale(0.3, 0.3).setInteractive();
        this.input.setDraggable(this.card);

        this.dealCards = () => {

        }

        this.dealText.on('pointerdown', function () {
            self.dealCards();
        })

        this.dealText.on('pointerover', function () {
            self.dealText.setColor('#ff69b4');
        })

        this.dealText.on('pointerout', function () {
            self.dealText.setColor('#00ffff');
        })

        this.input.on('drag', function (pointer, gameObject, dragX, dragY) {
            gameObject.x = dragX;
            gameObject.y = dragY;
        })

We've added a lot of structure, but not much has happened. Now, when our mouse hovers over the "DEAL CARDS" text, it's highlighted in cyberpunk hot pink, and there's a random card on our screen:

We've placed the image at the (x, y) coordinates of (300, 300), set its scale to be a bit smaller, and made it interactive and draggable. We've also added a little bit of logic to determine what should happen when dragged: it should follow the (x, y) coordinates of our mouse.

We've also created an empty dealCards() function that will be called when we click on our "DEAL CARDS" text. Additionally, we've saved "this" - meaning the scene in which we're currently working - into a variable called "self" so that we can use it throughout our functions without worrying about scope.

Our Game scene is going to get messy fast if we don't start moving things around, so let's delete the code block that begins with "this.card" and move to /src/helpers/card.js to write:

export default class Card {
    constructor(scene) {
        this.render = (x, y, sprite) => {
            let card = scene.add.image(x, y, sprite).setScale(0.3, 0.3).setInteractive();
            scene.input.setDraggable(card);
            return card;
        }
    }
}

We've created a new class that accepts a scene as a parameter, and features a render() function that accepts (x, y) coordinates and a sprite. Now, we can call this function from elsewhere and pass it the necessary parameters to create cards.

Let's import the card at the top of our Game scene:

import Card from '../helpers/card';

And enter the following code within our empty dealCards() function:

        this.dealCards = () => {
            for (let i = 0; i < 5; i++) {
                let playerCard = new Card(this);
                playerCard.render(475 + (i * 100), 650, 'cyanCardFront');
            }
        }

When we click on the "DEAL CARDS" button, we now iterate through a for loop that creates cards and renders them sequentially on screen:

NICE. We can drag those cards around the screen, but it might be nice to limit where they can be dropped to support our game logic.

Let's move over to /src/helpers/zone.js and add a new class:

export default class Zone {
    constructor(scene) {
        this.renderZone = () => {
            let dropZone = scene.add.zone(700, 375, 900, 250).setRectangleDropZone(900, 250);
            dropZone.setData({ cards: 0 });
            return dropZone;
        };
        this.renderOutline = (dropZone) => {
            let dropZoneOutline = scene.add.graphics();
            dropZoneOutline.lineStyle(4, 0xff69b4);
            dropZoneOutline.strokeRect(dropZone.x - dropZone.input.hitArea.width / 2, dropZone.y - dropZone.input.hitArea.height / 2, dropZone.input.hitArea.width, dropZone.input.hitArea.height)
        }
    }
}

Phaser has built-in dropzones that allow us to dictate where game objects can be dropped, and we've set up one here and provided it with an outline. We've also added a tiny bit of data called "cards" to the dropzone that we'll use later.

Let's import our new zone into the Game scene:

import Zone from '../helpers/zone';

And call it in within the create() function:

        this.zone = new Zone(this);
        this.dropZone = this.zone.renderZone();
        this.outline = this.zone.renderOutline(this.dropZone);

Not too shabby!

We need to add a bit of logic to determine how cards should be dropped into the zone. Let's do that below the "this.input.on('drag')" function:

        this.input.on('dragstart', function (pointer, gameObject) {
            gameObject.setTint(0xff69b4);
            self.children.bringToTop(gameObject);
        })

        this.input.on('dragend', function (pointer, gameObject, dropped) {
            gameObject.setTint();
            if (!dropped) {
                gameObject.x = gameObject.input.dragStartX;
                gameObject.y = gameObject.input.dragStartY;
            }
        })

        this.input.on('drop', function (pointer, gameObject, dropZone) {
            dropZone.data.values.cards++;
            gameObject.x = (dropZone.x - 350) + (dropZone.data.values.cards * 50);
            gameObject.y = dropZone.y;
            gameObject.disableInteractive();
        })

Starting at the bottom of the code, when a card is dropped, we increment the "cards" data value on the dropzone, and assign the (x, y) coordinates of the card to the dropzone based on how many cards are already on it. We also disable interactivity on cards after they're dropped so that they can't be retracted:

We've also made it so that our cards have a different tint when dragged, and if they're not dropped over the dropzone, they'll return to their starting positions.

Although our client isn't quite complete, we've done as much as we can before implementing the back end. We can now deal cards, drag them around the screen, and drop them in a dropzone. But to move forward, we'll need to set up a server than can coordinate our multiplayer functionality.

The Server

Let's open up a new command line at our root directory (above /client) and type:

npm init
npm install --save express socket.io nodemon

We've initialized a new package.json and installed Express, Socket.IO, and Nodemon (which will watch our server and restart it upon changes).

In our code editor, let's change the "scripts" section of our package.json to say:

  "scripts": {
    "start": "nodemon server.js"
  },

Excellent. We're ready to put our server together! Create an empty file called "server.js" in our root directory and enter the following code:

const server = require('express')();
const http = require('http').createServer(server);
const io = require('socket.io')(http);

io.on('connection', function (socket) {
    console.log('A user connected: ' + socket.id);

    socket.on('disconnect', function () {
        console.log('A user disconnected: ' + socket.id);
    });
});

http.listen(3000, function () {
    console.log('Server started!');
});

We're importing Express and Socket.IO, asking for the server to listen on port 3000. When a client connects to or disconnects from that port, we'll log the event to the console with the client's socket id.

Open a new command line interface and start the server:

npm run start

Our server should now be running on localhost:3000, and Nodemon will watch our back end files for any changes. Not much else will happen except for the console log that the "Server started!"

In our other open command line interface, let's navigate back to our /client directory and install the client version of Socket.IO:

cd client
npm install --save socket.io-client

We can now import it in our Game scene:

import io from 'socket.io-client';

Great! We've just about wired up our front and back ends. All we need to do is write some code in the create() function:

        this.socket = io('http://localhost:3000');

        this.socket.on('connect', function () {
            console.log('Connected!');
        });

We're initializing a new "socket" variable that points to our local port 3000 and logs to the browser console upon connection.

Open and close a couple of browsers at http://localhost:8080 (where our Phaser client is being served) and you should see the following in your command line interface:

YAY. Let's start adding logic to our server.js file that will serve the needs of our card game. Replace the existing code with the following:

const server = require('express')();
const http = require('http').createServer(server);
const io = require('socket.io')(http);
let players = [];

io.on('connection', function (socket) {
    console.log('A user connected: ' + socket.id);

    players.push(socket.id);

    if (players.length === 1) {
        io.emit('isPlayerA');
    };

    socket.on('dealCards', function () {
        io.emit('dealCards');
    });

    socket.on('cardPlayed', function (gameObject, isPlayerA) {
        io.emit('cardPlayed', gameObject, isPlayerA);
    });

    socket.on('disconnect', function () {
        console.log('A user disconnected: ' + socket.id);
        players = players.filter(player => player !== socket.id);
    });
});

http.listen(3000, function () {
    console.log('Server started!');
});

We've initialized an empty array called "players" and add a socket id to it every time a client connects to the server, while also deleting the socket id upon disconnection.

If a client is the first to connect to the server, we ask Socket.IO to "emit" an event that they're going to be Player A. Subsequently, when the server receives an event called "dealCards" or "cardPlayed", it should emit back to the clients that they should update accordingly.

Believe it or not, that's all the code we need to get our server working! Let's turn our attention back to the Game scene. Right at the top of the create() function, type the following:

        this.isPlayerA = false;
        this.opponentCards = [];

Under the code block that starts with "this.socket.on(connect)", write:

        this.socket.on('isPlayerA', function () {
            self.isPlayerA = true;
        })

Now, if our client is the first to connect to the server, the server will emit an event that tells the client that it will be Player A. The client socket receives that event and turns our "isPlayerA" boolean from false to true.

Note: from this point forward, you may need to reload your browser page (set to http://localhost:8080), rather than having Webpack do it automatically for you, for the client to correctly disconnect from and reconnect to the server.

We need to reconfigure our dealCards() logic to support the multiplayer aspect of our game, given that we want the client to deal us a certain set of cards that may be different from our opponent's. Additionally, we want to render the backs of our opponent's cards on our screen, and vice versa.

We'll move to the empty /src/helpers/dealer.js file, import card.js, and create a new class:

import Card from './card';

export default class Dealer {
    constructor(scene) {
        this.dealCards = () => {
            let playerSprite;
            let opponentSprite;
            if (scene.isPlayerA) {
                playerSprite = 'cyanCardFront';
                opponentSprite = 'magentaCardBack';
            } else {
                playerSprite = 'magentaCardFront';
                opponentSprite = 'cyanCardBack';
            };
            for (let i = 0; i < 5; i++) {
                let playerCard = new Card(scene);
                playerCard.render(475 + (i * 100), 650, playerSprite);

                let opponentCard = new Card(scene);
                scene.opponentCards.push(opponentCard.render(475 + (i * 100), 125, opponentSprite).disableInteractive());
            }
        }
    }
}

With this new class, we're checking whether the client is Player A, and determining what sprites should be used in either case.

Then, we deal cards to our client, while rendering the backs of our opponent's cards at the top the screen and adding them to the opponentCards array that we initialized in our Game scene.

In /src/scenes/game.js, import the Dealer:

import Dealer from '../helpers/dealer';

Then replace our dealCards() function with:

        this.dealer = new Dealer(this);

Under code block that begins with "this.socket.on('isPlayerA')", add the following:

        this.socket.on('dealCards', function () {
            self.dealer.dealCards();
            self.dealText.disableInteractive();
        })

We also need to update our dealText function to match these changes:

        this.dealText.on('pointerdown', function () {
            self.socket.emit("dealCards");
        })

Phew! We've created a new Dealer class that will handle dealing cards to us and rendering our opponent's cards to the screen. When the client socket receives the "dealcards" event from the server, it will call the dealCards() function from this new class, and disable the dealText so that we can't just keep generating cards for no reason.

Finally, we've changed the dealText functionality so that when it's pressed, the client emits an event to the server that we want to deal cards, which ties everything together.

Fire up two separate browsers pointed to http://localhost:8080 and hit "DEAL CARDS" on one of them. You should see different sprites on either screen:

Note again that if you're having issues with this step, you may have to close one of your browsers and reload the first one to ensure that both clients have disconnected from the server, which should be logged to your command line console.

We still need to figure out how to render our dropped cards in our opponent's client, and vice-versa. We can do all of that in our game scene! Update the code block that begins with "this.input.on('drop')" with one line at the end:

        this.input.on('drop', function (pointer, gameObject, dropZone) {
            dropZone.data.values.cards++;
            gameObject.x = (dropZone.x - 350) + (dropZone.data.values.cards * 50);
            gameObject.y = dropZone.y;
            gameObject.disableInteractive();
            self.socket.emit('cardPlayed', gameObject, self.isPlayerA);
        })

When a card is dropped in our client, the socket will emit an event called "cardPlayed", passing the details of the game object and the client's isPlayerA boolean (which could be true or false, depending on whether the client was the first to connect to the server).

Recall that, in our server code, Socket.IO simply receives the "cardPlayed" event and emits the same event back up to all of the clients, passing the same information about the game object and isPlayerA from the client that initiated the event.

Let's write what should happen when a client receives a "cardPlayed" event from the server, below the "this.socket.on('dealCards')" code block:

         this.socket.on('cardPlayed', function (gameObject, isPlayerA) {
            if (isPlayerA !== self.isPlayerA) {
                let sprite = gameObject.textureKey;
                self.opponentCards.shift().destroy();
                self.dropZone.data.values.cards++;
                let card = new Card(self);
                card.render(((self.dropZone.x - 350) + (self.dropZone.data.values.cards * 50)), (self.dropZone.y), sprite).disableInteractive();
            }
        })

The code block first compares the isPlayerA boolean it receives from the server against the client's own isPlayerA, which is a check to determine whether the client that is receiving the event is the same one that generated it.

Let's think that through a bit further, as it exposes a key component to how our client - server relationship works, using Socket.IO as the connector.

Suppose that Client A connects to the server first, and is told through the "isPlayerA" event that it should change its isPlayerA boolean to true. That's going to determine what kind of cards it generates when a user clicks "DEAL CARDS" through that client.

If Client B connects to the server second, it's never told to alter its isPlayerA boolean, which stays false. That will also determine what kind of cards it generates.

When Client A drops a card, it emits a "cardPlayed" event to the server, passing information about the card that was dropped, and its isPlayerA boolean, which is true. The server then relays all that information back up to all clients with its own "cardPlayed" event.

Client A receives that event from the server, and notes that the isPlayerA boolean from the server is true, which means that the event was generated by Client A itself. Nothing special happens.

Client B receives the same event from the server, and notes that the isPlayerA boolean from the server is true, although Client B's own isPlayerA is false. Because of this difference, it executes the rest of the code block.

The ensuing code stores the "texturekey" - basically, the image - of the game object that it receives from the server into a variable called "sprite". It destroys one of the opponent card backs that are rendered at the top of the screen, and increments the "cards" data value in the dropzone so that we can keep placing cards from left to right.

The code then generates a new card in the dropzone that uses the sprite variable to create the same card that was dropped in the other client (if you had data attached to that game object, you could use a similar approach to attach it here as well).

Your final /src/scenes/game.js code should look like this:

import io from 'socket.io-client';
import Card from '../helpers/card';
import Dealer from "../helpers/dealer";
import Zone from '../helpers/zone';

export default class Game extends Phaser.Scene {
    constructor() {
        super({
            key: 'Game'
        });
    }

    preload() {
        this.load.image('cyanCardFront', 'src/assets/CyanCardFront.png');
        this.load.image('cyanCardBack', 'src/assets/CyanCardBack.png');
        this.load.image('magentaCardFront', 'src/assets/magentaCardFront.png');
        this.load.image('magentaCardBack', 'src/assets/magentaCardBack.png');
    }

    create() {
        this.isPlayerA = false;
        this.opponentCards = [];

        this.zone = new Zone(this);
        this.dropZone = this.zone.renderZone();
        this.outline = this.zone.renderOutline(this.dropZone);

        this.dealer = new Dealer(this);

        let self = this;

        this.socket = io('http://localhost:3000');

        this.socket.on('connect', function () {
            console.log('Connected!');
        });

        this.socket.on('isPlayerA', function () {
            self.isPlayerA = true;
        })

        this.socket.on('dealCards', function () {
            self.dealer.dealCards();
            self.dealText.disableInteractive();
        })

        this.socket.on('cardPlayed', function (gameObject, isPlayerA) {
            if (isPlayerA !== self.isPlayerA) {
                let sprite = gameObject.textureKey;
                self.opponentCards.shift().destroy();
                self.dropZone.data.values.cards++;
                let card = new Card(self);
                card.render(((self.dropZone.x - 350) + (self.dropZone.data.values.cards * 50)), (self.dropZone.y), sprite).disableInteractive();
            }
        })

        this.dealText = this.add.text(75, 350, ['DEAL CARDS']).setFontSize(18).setFontFamily('Trebuchet MS').setColor('#00ffff').setInteractive();

        this.dealText.on('pointerdown', function () {
            self.socket.emit("dealCards");
        })

        this.dealText.on('pointerover', function () {
            self.dealText.setColor('#ff69b4');
        })

        this.dealText.on('pointerout', function () {
            self.dealText.setColor('#00ffff');
        })

        this.input.on('drag', function (pointer, gameObject, dragX, dragY) {
            gameObject.x = dragX;
            gameObject.y = dragY;
        })

        this.input.on('dragstart', function (pointer, gameObject) {
            gameObject.setTint(0xff69b4);
            self.children.bringToTop(gameObject);
        })

        this.input.on('dragend', function (pointer, gameObject, dropped) {
            gameObject.setTint();
            if (!dropped) {
                gameObject.x = gameObject.input.dragStartX;
                gameObject.y = gameObject.input.dragStartY;
            }
        })

        this.input.on('drop', function (pointer, gameObject, dropZone) {
            dropZone.data.values.cards++;
            gameObject.x = (dropZone.x - 350) + (dropZone.data.values.cards * 50);
            gameObject.y = dropZone.y;
            gameObject.disableInteractive();
            self.socket.emit('cardPlayed', gameObject, self.isPlayerA);
        })
    }

    update() {

    }
}

Save everything, open two browsers, and hit "DEAL CARDS". When you drag and drop a card in one client, it should appear in the dropzone of the other, while also deleting a card back, signifying that a card has been played:

That's it! You should now have a functional template for your multiplayer card game, which you can use to add your own cards, art, and game logic.

One first step could be to add to your Dealer class by making it shuffle an array of cards and return a random one (hint: check out Phaser.Math.RND.shuffle([array])).

Happy coding!

If you enjoyed this article, please consider checking out my games and books, subscribing to my YouTube channel, or joining the Entromancy Discord.

M. S. Farzan, Ph.D. has written and worked for high-profile video game companies and editorial websites such as Electronic Arts, Perfect World Entertainment, Modus Games, and MMORPG.com, and has served as the Community Manager for games like Dungeons & Dragons Neverwinter and Mass Effect: Andromeda. He is the Creative Director and Lead Game Designer of Entromancy: A Cyberpunk Fantasy RPG and author of The Nightpath Trilogy. Find M. S. Farzan on Twitter @sominator.

I'm a tabletop game developer, and enjoy making apps that have the potential to perform some service related to gaming. In this article, we'll walk through the steps to create a roleplaying game character generator using MongoDB, Express, Vue, and Node (also known as the "MEVN" stack).

Prerequisites: this tutorial presumes that you have Node/NPM and MongoDB installed and configured, with a code editor and CLI (or IDE) ready to go.

If you'd prefer to follow along with a visual tutorial, you can check out the companion video to this article below:

I should also mention that this tutorial would not have been possible without Bennett Dungan's article on building a REST API, Aneeta Sharma's tutorial on full stack MEVN web apps, and Matt Maribojoc's article on the same topic.

I used each of these articles in addition to official documentation (for Vue, Express, and a whole lot more) in learning to create my own MEVN apps (you can read more about my journey with web APIs here).

You can access the entire repository for this tutorial on GitHub.

The Front End

Our app is going to allow us to create new roleplaying game characters and view them altogether, with the following stack:

• Vue Client
• Node/Express Server
• MongoDB Database

The Vue Client will make HTTP requests to the Node/Express Server (or "API"), which will in turn communicate with our MongoDB Database to send data back up the stack.

We'll begin by opening a command line, creating a new directory for our project, and navigating into that directory:

mkdir mevn-character-generator
cd mevn-character-generator

We'll then install the Vue CLI globally to help us scaffold a basic app:

npm install -g @vue/cli

Next, we'll use the Vue CLI to create a new app called "Client" within our mevn-character-generator directory:

vue create client

You can just hit "enter" at the prompt to keep going.

We can run our app by first navigating into the /client folder:

cd client
npm run serve

When the script has completed running, we can now open a browser page and navigate to the URL indicated by our terminal (usually http://localhost:8080 or 8081). We should see something like this:

Nice! The Vue CLI has scaffolded a basic app for us, and is rendering it right into the browser. It'll also reload the page automatically upon file changes, and throw errors if something in the code looks amiss.

Let's open the project directory in our code editor to take a look at the file structure, which should look like this:

If you're OCD like I am, you can go ahead and delete the "favicon.ico" file and "/assets" folder as we won't need them for this project.

Diving into /src/main.js, we see:

import Vue from 'vue'
import App from './App.vue'

Vue.config.productionTip = false

new Vue({
  render: h => h(App),
}).$mount('#app')

This file is the main entry point for our client. It tells the browser to mount our App.vue file to the div with id "#app" in /public/index.html.

Let's look at /src/App.vue (I've omitted some code for readability):

<template>
  <div id="app">
    <img alt="Vue logo" src="./assets/logo.png">
    <HelloWorld msg="Welcome to Your Vue.js App"/>
  </div>
</template>

<script>
import HelloWorld from './components/HelloWorld.vue'

export default {
  name: 'App',
  components: {
    HelloWorld
  }
}
</script>

<style>
#app {
...
}
</style>

App.vue is a typical Vue component, with ,

Is your phone locked? Do you have a pin-code, password, fingerprint, or FaceID? I am 99 percent sure that you do. And it is clear why – you care about your safety. Nowadays, keeping your phone protected is as important as brushing your teeth in the morning.

For diligent and mindful software developers, keeping their app secure is equally important to protecting their phones. If you are a developer and you choose to neglect it – please, reconsider your approach. If you are a project owner and your development team says that data safety can wait, please, reconsider your team.

In this article, I want to talk about how to make sure that your Express.js project is safe and invincible to malicious attacks.

There are 7 simple and not very simple measures to take for the purpose of data security:

1. Use reliable versions of Express.js
2. Secure the connection and data
3. Protect your cookies
4. Secure your dependencies
5. Validate the input of your users
6. Protect your system against brute force
7. Control user access

Let’s have a closer look at each.

1. Use reliable versions of Express.js

Deprecated or outdated versions of Express.js are a no go. The 2nd and 3rd versions of Express are no longer supported. In these, safety or performance issues are not fixed anymore.

As a developer, you absolutely have to migrate to Express 4. This version is a revolution! It is quite different in terms of the routing system, middleware, and other minor aspects.

2. Secure the connection and data

To secure HTTP headers, you can make use of Helmet.js – a helpful Node.js module. It is a collection of 13 middleware functions for setting HTTP response headers. In particular, there are functions for setting Content Security Policy, handling Certificate Transparency, preventing clickjacking, disabling client-side caching, or adding some small XSS protections.

npm install helmet --save

Even if you do not want to use all the functions of Helmet, the absolute minimum that you must do is to disable X-Powered-By header:

app.disable('x-powered-by')

This header can be used to detect that the application is powered by Express, which lets hackers conduct a precise attack. Surely, X-Powered-By header is not the only way to identify an Express-run application, but it is probably the most common and simple one.

To protect your system from HTTP parameter pollution attacks, you can use HPP. This middleware puts aside such parameters as req.query and req.body and selects the latest parameter value instead. The installation command looks as follows:

npm install hpp --save

To encrypt data which is being sent from the client to the server, use Transport Layer Security (TLS). TLS is a cryptographic protocol for securing the computer network, the descendant of the Secure Socket Layer (SSL) encryption. TLS can be handled with Nginx – a free but effective HTTP server – and Let’s Encrypt – a free TLS certificate.

3. Protect your cookies

In Express.js 4, there are two cookie session modules:

• express-session (in Express.js 3, it was express.session)
• cookie-session (in Express.js 3, it was express.cookieSession)

The express-session module stores session ID in the cookie and session data on the server. The cookie-session stores all the session data to the cookie.

In general, cookie-session is more efficient. Yet, if the session data you need to store is complex and likely to exceed 4096 bytes per cookie, use express-session. Another reason to use express-session is when you need to keep the cookie data invisible to the client.

Besides, you should set cookie security options, namely:

• secure
• httpOnly
• domain
• path
• expires

If “secure” is set to “true”, the browser will send cookies only via HTTPS. If “httpOnly” is set to “true”, the cookie will be sent not via client JS but via HTTP(S). The value of “domain” indicates the domain of the cookie. If the cookie domain matches the server domain, “path” is used to indicate the cookie path. If the cookie path matches the request path, the cookie will be sent in the request. Finally, as the name itself suggests, the value of “expires” stands for the time when the cookies will expire.

Another important recommendation is not to use the default session cookie name. It may enable hackers to detect the server and to run a targeted attack. Instead, use generic cookie names.

4. Secure your dependencies

No doubt, npm is a powerful web development tool. However, to ensure the highest level of security, consider using only the 6th version of it – npm@6. The older ones may contain some serious dependency safety vulnerabilities, which will endanger your entire app. Also, to analyze the tree of dependencies, use the following command:

npm audit

npm audit can help to fix real problems in your project. It checks all your dependencies in dependencies, devDependencies, bundledDependencies, and optionalDependencies, but not your peerDependencies. Here you can read about all current vulnerabilities in any npm packages.

Another tool to ensure dependency safety is Snyk. Snyk runs the application check to identify whether it contains any vulnerability listed in Snyk’s open-source database. To conduct the check, run three simple steps.

Step 1. Install Snyk

npm install -g snyk
cd your-app

Step 2. Run a test

snyk test

Step 3. Learn how to fix the issue

snyk wizard

Wizard is a Snyk method, which explains the nature of the dependency vulnerability and offers ways of fixing it.

5. Validate the input of your users

Controlling user input is an extremely important part for server-side development. This is a no less important problem than unauthorized requests, which will be described in the seventh part of this article.

First of all, wrong user input can break your server when some values are undefined and you do not have error handling for a specific endpoint. However, different ORM systems can have unpredictable behavior when you try to set undefined, null, or other data types in the database.

For example, destroyAll method in Loopback.js ORM (Node.js framework) can destroy all data in a table of the database: when it does not match any records it deletes everything as described here. Imagine that you can lose all data in a production table just because you have ignored input validation.

Use body/object validation for intermediate inspections

To start with, you can use body/object validation for intermediate inspections. For example, we use ajv validator which is the fastest JSON Schema validator for Node.js.

const Ajv = require('ajv'); 
const ajv = new Ajv({allErrors: true}); 
const speaker = { 
  'type': 'object', 
  'required': [
    'id', 
    'name'
  ],
  'properties': { 
    'id': {
      'type': 'integer', 
    }, 
    'name': { 
      'type': 'string',
    }, 
  }, 
};
const conversation = { 
  type: 'object', 
  required: [
    'duration', 
    'monologues'
  ], 
  properties: { 
    duration: { 
      type: 'integer',
    }, 
    monologues: { 
      type: 'array', 
      items: monolog, 
    }, 
  }, 
};
const body = { 
  type: 'object', 
  required: [
    'speakers', 
    'conversations'
  ], 
  properties: { 
    speakers: { 
      type: 'array', 
      items: speaker, 
    }, 
    conversations: { 
      type: 'array', 
      items: conversation, 
    }, 
  }, 
}; 
const validate = ajv.compile(body); 
const isValidTranscriptBody = transcriptBody => { 
  const isValid = validate(transcriptBody);
  if (!isValid) { 
    console.error(validate.errors); 
  } 
  return isValid; 
};

Handle errors

Now, imagine that you forgot to check a certain object and you do some operations with the undefined property. Or you use a certain library and you get an error. It can break your instance, and the server will crash. Then, the attacker can ping a specific endpoint where there is this vulnerability and can stop your server for a long time.

The simplest way to do an error handling is to use try-catch construction:

try { 
  const data = body;
  if (data.length === 0) throw new Error('Client Error'); 
  const beacons = await  this.beaconLogService.filterBeacon(data); 
  if (beacons.length > 0) { 
    const max = beacons.reduce((prev, current) => (prev.rssi > current.rssi) ? prev : current); 
    await this.beaconLogService.save({ 
      ...max,
      userId: headers['x-uuid'] 
    }); 
    return { 
      data: { 
        status: 'Saved', 
        position: max 
      }, 
    }; 
  } 
  return { 
    data: { 
      status: 'Not valid object, 
    }, 
  }; 
} 
catch(err) { 
  this.logger.error(err.message, err.stack); 
  throw new HttpException('Server Error',     HttpStatus.INTERNAL_SERVER_ERROR); 
}

Feel free to use a new Error(‘message’) constructor for error handling or even extend this class for your own purpose!

Use JOI

The main lesson here is that you should always validate user input so you don't fall victim to man-in-the-middle attacks. Another way to do it is with the help of @hapi/joi – a part of the hapi ecosystem and a powerful JS data validation library.

Pay attention here that the module joi has been deprecated. For this reason, the following command is a no go:

npm install joi

Instead, use this one:

npm install @hapi/joi

Use express-validator

One more way to validate user input is to use express-validator – a set of express.js middlewares, which comprises validator.js and function sanitizer. To install it, run the following command:

npm install --save express-validator

Sanitize user input

Also, an important measure to take is to sanitize user input to protect the system from a MongoDB operator injection. For this, you should install and use express-mongo-sanitize:

npm install express-mongo-sanitize

Protect your app against CSRF

Besides, you should protect your app against cross-site request forgery (CSRF). CSRF is when unauthorized commands are sent from a trusted user. You can do this with the help of csurf. Prior to that, you need to make sure that session middleware for cookies is configured as described earlier in this article. To install this Node.js module, run the command:

npm install csurf

6. Protect your system against brute force

A brute force attack is the simplest and most common way to get access to a website or a server. The hacker (in most cases automatically, rarely manually) tries various usernames and passwords repeatedly to break into the system.

These attacks can be prevented with the help of rate-limiter-flexible package. This package is fast, flexible, and suitable for any Node framework.

To install, run the following command:

npm i --save rate-limiter-flexible
yarn add rate-limiter-flexible

This method has a simpler but more primitive alternative: express-rate-limit. The only thing it does is limiting repeated requests to public APIs or to password reset.

npm install --save express-rate-limit

7. Control user access

Among the authentication methods, there are tokens, Auth0, and JTW. Let’s focus on the third one! JTW (JSON Web Tokens) are used to transfer authentication data in client-server applications. Tokens are created by the server, signed with a secret key, and transferred to a client. Then, the client uses these tokens to confirm identity.

Express-jwt-permissions is a tool used together with express-jwt to check permissions of a certain token. These permissions are an array of strings inside the token:

"permissions": [
  "status",
  "user:read",
  "user:write"
]

To install the tool, run the following command:

npm install express-jwt-permissions --save

To Wrap Up

Here, I have listed the essential Express.js security best practices and some tools that can be used along the way.

Just to review:

I strongly recommend that you make sure that your application is resistant to malicious attacks. Otherwise, your business and your users may suffer significant losses.

Do you have an idea for Express.js project?

My company KeenEthics is experienced in express js development. In case you need a free estimate of a similar project, feel free to get in touch.

If you have enjoyed the article, you should definitely continue with a piece on data safety in outsourcing to Ukraine: KeenEthics Team on Guard: Your Data is Safe in Ukraine. The original article posted on KeenEthics blog can be found here: express js security tips.

P.S.

A huge shout-out to Volodia Andrushchak, Full-Stack Software Developer @KeenEthics for helping me with the article.

The original article posted on KeenEthics blog can be found here: Express.js Security Tips: Save Your App!

#

Setting up the test was the hard part. People who write about tests don't actually teach you how they set it up. I could not find any useful information about this, and I had to try and figure it out.

So today, I want to share the setup I created for myself. Hopefully, this can help you when you create your own tests.

Table of Contents

1. Setting up Jest and Supertest
2. Connecting Jest and Mongoose
3. Seeding a database

Setting up Jest and Supertest

First, let's talk about the stack.

The Stack

• I created my app with Express.
• I used Mongoose to connect to MongoDB
• I used Jest as my test framework.

You might have expected Express and Mongoose because everyone else seems to use those two frameworks. I used them too.

But why Jest and not other test frameworks?

Why Jest

I don't like Facebook, so I didn't want to try anything that was created by Facebook's team. I know it sounds silly, but that was the truth.

Before Jest, I tried out all sorts of test frameworks. I tried Tap, Tape, Mocha, Jasmine, and AVA. Each test framework has its own pros and cons. I almost ended up with AVA, but I didn't go with AVA because I found it hard to set up. Eventually, I tried Jest out because Kent C. Dodds recommended it.

I fell in love with Jest after trying it out. I love it because:

1. It's easy to setup
2. The watch-mode is amazing
3. When you console.log something, it actually shows up without any difficulty (this was a bitch with AVA).

Setting up Jest

First, you need to install Jest.

npm install jest --save-dev

Next, you want to add tests scripts to your package.json file. It helps to add the test and test:watch scripts (for one-off testing and watch-mode respectively).

"scripts": {
  "test": "jest",
  "test:watch": "jest --watch"
},

You can choose to write your test files in one of the following formats. Jest picks them up for you automatically.

1. js files in the __tests__ folder
2. files named with test.js (like user.test.js)
3. files named with spec.js (like user.spec.js)

You can place your files however you like. When I tested endpoints, I put the test files together with my endpoints. I found this easier to manage.

- routes
  |- users/
    |- index.js
    |- users.test.js

Writing your first test

Jest includes describe, it and expect for you in every test file. You don't have to require them.

• describe lets you wrap many tests together under one umbrella. (It is used for organizing your tests).
• it lets you run a test.
• expect lets you perform assertions. The test passes if all assertions passes.

Here's an example of a test that fails. In this example, I expect that 1 should be strictly equal to 2. Since 1 !== 2, the test fails.

// This test fails because 1 !== 2
it("Testing to see if Jest works", () => {
  expect(1).toBe(2);
});

You'll see a failing message from Jest if you run Jest.

npm run test:watch

You can make the test pass by expecting 1 === 1.

// This passes because 1 === 1
it("Testing to see if Jest works", () => {
  expect(1).toBe(1);
});

This is the most basic of tests. It's not useful at all because we haven't testing anything real yet.

Asynchronous tests

You need to send a request to test an endpoint. Requests are asynchronous, which means you must be able to conduct asynchronous tests.

This is easy with Jest. There are two steps:

1. Add the async keyword
2. Call done when you're done with your tests

Here's what it can look like:

it("Async test", async done => {
  // Do your async tests here

  done();
});

Note: Here's an article on Async/await in JavaScript if you don't know how to use it.

Testing Endpoints

You can use Supertest to test endpoints. First, you need to install Supertest.

npm install supertest --save-dev

Before you can test endpoints, you need to setup the server so Supertest can use it in your tests.

Most tutorials teach you to listen to the Express app in the server file, like this:

const express = require("express");
const app = express();

// Middlewares...
// Routes...

app.listen(3000);

This doesn't work because it starts listening to one port. If you try to write many test files, you'll get an error that says "port in use".

You want to allow each test file to start a server on their own. To do this, you need to export app without listening to it.

// server.js
const express = require("express");
const app = express();

// Middlewares...
// Routes...

module.exports = app;

For development or production purposes, you can listen to your app like normal in a different file like start.js.

// start.js
const app = require("./server.js");
app.listen(3000);

Using Supertest

To use Supertest, you require your app and supertest in the test file.

const app = require("./server"); // Link to your server file
const supertest = require("supertest");
const request = supertest(app);

Once you do this, you get the ability to send GET, POST, PUT, PATCH and DELETE requests. Before we send a request, we need to have an endpoint. Let's say we have a /test endpoint.

app.get("/test", async (req, res) => {
  res.json({ message: "pass!" });
});

To send a GET request to /test, you use the .get method from Supertest.

it("Gets the test endpoint", async done => {
  // Sends GET Request to /test endpoint
  const res = await request.get("/test");

  // ...
  done();
});

Supertest gives you a response from the endpoint. You can test both HTTP status and the body (whatever you send through res.json) like this:

it("gets the test endpoint", async done => {
  const response = await request.get("/test");

  expect(response.status).toBe(200);
  expect(response.body.message).toBe("pass!");
  done();
});

Connecting Jest and Mongoose

The hard part about testing a backend application is setting up a test database. It can be complicated.

Today, I want to share how I setup Jest and Mongoose.

Setting up Mongoose with Jest

Jest gives you a warning if you try to use Mongoose with Jest.

If you don't want to see this error, you need to set testEnvironment to node in your package.json file.

"jest": {
  "testEnvironment": "node"
}

Setting up Mongoose in a test file

You want to connect to a database before you begin any tests. You can use the beforeAll hook to do so.

beforeAll(async () => {
  // Connect to a Mongo DB
});

To connect to a MongoDB, you can use Mongoose's connect command.

const mongoose = require("mongoose");
const databaseName = "test";

beforeAll(async () => {
  const url = `mongodb://127.0.0.1/${databaseName}`;
  await mongoose.connect(url, { useNewUrlParser: true });
});

This creates a connection to the database named test. You can name your database anything. You'll learn how to clean them up later.

Note: Make sure you have an active local MongoDB Connection before you test. Your tests will fail if you don't have an active local MongoDB Connection. Read this to learn how to create a local MongoDB connection.

Creating databases for each test file

When you test, you want to connect to a different database for each test file, because:

1. Jest runs each test file asynchronously. You won't know which file comes first.
2. You don't want tests to share the same database. You don't want data from one test file to spill over to the next test file.

To connect to a different database, you change the name of the database.

// Connects to database called avengers
beforeAll(async () => {
  const url = `mongodb://127.0.0.1/avengers`;
  await mongoose.connect(url, { useNewUrlParser: true });
});

// Connects to database power-rangers
beforeAll(async () => {
  const url = `mongodb://127.0.0.1/power-rangers`;
  await mongoose.connect(url, { useNewUrlParser: true });
});

Sending a POST request

Let's say you want to create a user for your app. The user has a name and an email address. Your Mongoose Schema might look like this:

const mongoose = require("mongoose");
const Schema = mongoose.Schema;

const userSchema = new Schema({
  name: String,
  email: {
    type: String,
    require: true,
    unique: true
  }
});

module.exports = mongoose.model("User", userSchema);

To create a user, you need to save the name and email into MongoDB. Your route and controller might look like this:

const User = require("../model/User"); // Link to your user model

app.post("/signup", async (req, res) => {
  const { name, email } = req.body;
  const user = new User({ name, email });
  const ret = await user.save();
  res.json(ret);
});

To save the user into the database, you can send a POST request to signup. To send a post request, you use the post method. To send data along with the POST request, you use the send method. In your tests, it'll look like this.

it("Should save user to database", async done => {
  const res = await request.post("/signup").send({
    name: "Zell",
    email: "testing@gmail.com"
  });
  done();
});

Note: If you run this code two times, you'll get an E1100 duplicate key error. This error occurred because:

1. We said the email should be unique in the Schema above.
2. We tried to create another user with testing@gmail.com. even though one already exists in the database. (The first one was created when you sent the first request).

Cleaning up the database between tests

You want to remove entries from the database between each test. This ensures you always start with an empty database.

You can do this with the afterEach hook.

// Cleans up database between each test
afterEach(async () => {
  await User.deleteMany();
});

In this code above, we only cleared the User collection in the database. In a real scenario, you want to clear all collections. You can use the following code to do so:

async function removeAllCollections() {
  const collections = Object.keys(mongoose.connection.collections);
  for (const collectionName of collections) {
    const collection = mongoose.connection.collections[collectionName];
    await collection.deleteMany();
  }
}

afterEach(async () => {
  await removeAllCollections();
});

Testing the Endpoint

Let's begin our tests. In this test, we will send a POST request to the /signup endpoint. We want to make sure:

1. The user gets saved to the database
2. The returned object contains information about the user

Checking if the user was saved to the database

To check whether the user gets saved into the database, you search the database for the user.

const User = require("../model/User"); // Link to your user model

it("Should save user to database", async done => {
  const res = await request.post("/signup").send({
    name: "Zell",
    email: "testing@gmail.com"
  });

  // Searches the user in the database
  const user = await User.findOne({ email: "testing@gmail.com" });

  done();
});

If you console.log user, you should see something like this:

This means our user got saved to the database. If we want to confirm the user has a name and an email, we can do expect them to be true.

it("Should save user to database", async done => {
  // Sends request...

  // Searches the user in the database
  const user = await User.findOne({ email: "testing@gmail.com" });
  expect(user.name).toBeTruthy();
  expect(user.email).toBeTruthy();

  done();
});

Checking if the returned object contains the information about the user

We want to make sure the returned object contains the user's name and email address. To do this, we check the response from the post request.

it("Should save user to database", async done => {
  // Sends request...

  // Searches the user in the database...

  // Ensures response contains name and email
  expect(res.body.name).toBeTruthy();
  expect(res.body.email).toBeTruthy();
  done();
});

We're done with our tests now. We want to delete the database from MongoDB.

Deleting the database

To delete the database, you need to ensure there are 0 collections in the database. We can do this by dropping each collection we used.

We'll do after all our tests have run, in the afterAll hook.

afterAll(async () => {
  // Removes the User collection
  await User.drop();
});

To drop all your collections you can use this:

async function dropAllCollections() {
  const collections = Object.keys(mongoose.connection.collections);
  for (const collectionName of collections) {
    const collection = mongoose.connection.collections[collectionName];
    try {
      await collection.drop();
    } catch (error) {
      // This error happens when you try to drop a collection that's already dropped. Happens infrequently.
      // Safe to ignore.
      if (error.message === "ns not found") return;

      // This error happens when you use it.todo.
      // Safe to ignore.
      if (error.message.includes("a background operation is currently running"))
        return;

      console.log(error.message);
    }
  }
}

// Disconnect Mongoose
afterAll(async () => {
  await dropAllCollections();
});

Finally, you want to close the Mongoose connection to end the test. Here's how you can do it:

afterAll(async () => {
  await dropAllCollections();
  // Closes the Mongoose connection
  await mongoose.connection.close();
});

That's everything you need to do to setup Mongoose with Jest!

Refactoring

There's a lot of code that goes into beforeEach, afterEach, and afterAll hooks. We will be using them for every test file. It makes sense to create a setup file for these hooks.

// test-setup.js
const mongoose = require("mongoose");
mongoose.set("useCreateIndex", true);
mongoose.promise = global.Promise;

async function removeAllCollections() {
  const collections = Object.keys(mongoose.connection.collections);
  for (const collectionName of collections) {
    const collection = mongoose.connection.collections[collectionName];
    await collection.deleteMany();
  }
}

async function dropAllCollections() {
  const collections = Object.keys(mongoose.connection.collections);
  for (const collectionName of collections) {
    const collection = mongoose.connection.collections[collectionName];
    try {
      await collection.drop();
    } catch (error) {
      // Sometimes this error happens, but you can safely ignore it
      if (error.message === "ns not found") return;
      // This error occurs when you use it.todo. You can
      // safely ignore this error too
      if (error.message.includes("a background operation is currently running"))
        return;
      console.log(error.message);
    }
  }
}

module.exports = {
  setupDB(databaseName) {
    // Connect to Mongoose
    beforeAll(async () => {
      const url = `mongodb://127.0.0.1/${databaseName}`;
      await mongoose.connect(url, { useNewUrlParser: true });
    });

    // Cleans up database between each test
    afterEach(async () => {
      await removeAllCollections();
    });

    // Disconnect Mongoose
    afterAll(async () => {
      await dropAllCollections();
      await mongoose.connection.close();
    });
  }
};

You can import the setup file for each test like this:

const { setupDB } = require("../test-setup");

// Setup a Test Database
setupDB("endpoint-testing");

// Continue with your tests...

There's one more thing I want to show you.

When you create tests, you want to seed the database with fake data.

Seeding a database

When you write tests for the backend, you need to test for four different kinds of operations:

1. Create (for adding things to the database)
2. Read (for getting things from the database)
3. Update (for changing the database)
4. Delete (for deleting things from the database)

The easiest type to test for is create operations. You put something into the database and test whether it's there.

For the other three types of operations, you need to put something into the database before you write the test.

Putting things into the database

The process where you add things to a database is called seeding a database.

Let's say you want to add three users to the database. These users contain a name and an email address.

const users = [
  {
    name: "Zell",
    email: "testing1@gmail.com"
  },
  {
    name: "Vincy",
    email: "testing2@gmail.com"
  },
  {
    name: "Shion",
    email: "testing3@gmail.com"
  }
];

You can use your models to seed the database at the start of the test.

const User = require("../model/User"); // Link to User model

it("does something", async done => {
  // Add users to the database
  for (const u of users) {
    const user = new User(u);
    await user.save();
  }

  // Create the rest of your test here
});

If you need these users for every test, the best way is to add them through the beforeEach hook. The beforeEach hook runs before every it declaration.

// Seed the database with users
beforeEach(async () => {
  for (u of users) {
    const user = new User(u);
    await user.save();
  }
});

You can also use Mongoose's create function to do the same thing. It runs new Model() and save(), so the code below and the one above does the same thing.

// Seed the database with users
beforeEach(async () => {
  await User.create(users);
});

create vs insertMany

Mongoose has a second method to help you seed the database. This method is called insertMany. insertMany is faster than create, because:

• insertMany sends one operation to the server
• create sends one operation for each document

However, insertMany does not run the save middleware.

Is triggering the save middleware important?

This depends on your seed data. If your seed data needs to go through the save middleware, you need to use create. For example, let's say you want to save a user's password into the database. You have this data:

const users = [
  {
    name: "Zell",
    email: "testing1@gmail.com",
    password: "12345678"
  },
  {
    name: "Vincy",
    email: "testing2@gmail.com",
    password: "12345678"
  },
  {
    name: "Shion",
    email: "testing3@gmail.com",
    password: "12345678"
  }
];

When we save a user's password into the database, we want to hash the password for security reasons. We usually hash the password through the save middleware.

// Hashes password automatically
userSchema.pre("save", async function(next) {
  if (!this.isModified("password")) return next();
  const salt = bcrypt.genSaltSync(10);
  const hashedPassword = bcrypt.hashSync(password, salt);
  this.password = hashedPassword;
});

If you use create, you'll get users with hashed passwords:

If you use insertMany, you'll get users without hashed passwords:

When to use create, when to use insertMany

Since insertMany is faster than create, you want to use insertMany whenever you can.

Here's how I do it:

1. If seed data does not require the save middleware, use insertMany.
2. If seed data requires save middleware, use create. Then, overwrite seed data so it no longer requires the save middleware.

For the password example above, I would run create first. Then, I copy-paste the hashed password seed data. Then, I'll run insertMany from this point onwards.

If you want to overwrite complicated seed data, you might want to get JSON straight from MongoDB. To do this, you can use mongoexport:

mongoexport --db <databaseName> --collection <collectionName> --jsonArray --pretty --out output.json

This says:

1. Export <collection> from <databaseName>
2. Creates output as a JSON Array, prettified, in a file called output.json. This file will be placed in the folder where you run the command.

Seeding multiple test files and collections

You want a place to store your seed data so you can use them across all your tests and collections. Here's a system I use:

1. I name my seed files according to their models. I seed a User model with the user.seed.js file.
2. I put my seed files in the seeds folder
3. I loop through each seed file to seed the database.

To loop through each seed file, you need to use the fs module. fs stands for filesystem.

The easiest way to loop through the files is to create an index.js file in the same seeds folder. Once you have the index.js file, you can use the following code to look for all files with *.seed.js

const fs = require("fs");
const util = require("util");

// fs.readdir is written with callbacks.
// This line converts fs.readdir into a promise
const readDir = util.promisify(fs.readdir);

async function seedDatabase() {
  // Gets list of files in the directory
  // `__dirname` points to the `seeds/` folder
  const dir = await readDir(__dirname);

  // Gets a list of files that matches *.seed.js
  const seedFiles = dir.filter(f => f.endsWith(".seed.js"));
}

Once you have a list of seed files, you can loop through each seed file to seed the database. Here, I use a for...of loop to keep things simple.

async function seedDatabase() {
  for (const file of seedFiles) {
    // Seed the database
  }
}

To seed the database, we need to find the correct Mongoose model from the name of the seed file. A file called user.seed.js should seed the User model. This means:

1. We must find user from user.seed.js
2. We must capitalize user into User

Here's a crude version that does what's required. (If you want to, you can make the code more robust with regex instead of split).

for (const file of seedFiles) {
  const fileName = file.split(".seed.js")[0];
  const modelName = toTitleCase(fileName);
  const model = mongoose.models[modelName];
}

Next, we want to make sure each file has a Model that corresponds to it. If the model cannot be found, we want to throw an error.

for (const file of seedFiles) {
  //...
  if (!model) throw new Error(`Cannot find Model '${modelName}'`);
}

If there's a corresponding model, we want to seed the database with the contents in the seed file. To do this, we need to read the seed file first. Here, since I used the .js extension, I can simply require the file.

for (const file of seedFiles) {
  //...
  const fileContents = require(path.join(__dirname, file));
}

For this to work, my seed files must export an array of data.

module.exports = [
  {
    name: "Zell",
    email: "testing1@gmail.com",
    password: "12345678"
  },
  {
    name: "Vincy",
    email: "testing2@gmail.com",
    password: "12345678"
  },
  {
    name: "Shion",
    email: "testing3@gmail.com",
    password: "12345678"
  }
];

Once I have the contents of the seed file, I can run create or insertMany.

async function seedDatabase(runSaveMiddleware = false) {
  // ...
  for (const file of seedFiles) {
    // ...

    runSaveMiddleware
      ? model.create(fileContents)
      : model.insertMany(fileContents);
  }
}

Here's the whole seedDatabase code:

const fs = require("fs");
const util = require("util");
const readDir = util.promisify(fs.readdir).bind(fs);
const path = require("path");
const mongoose = require("mongoose");

function toTitleCase(str) {
  return str.replace(/\w\S*/g, txt => {
    return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
  });
}

async function seedDatabase(runSaveMiddleware = false) {
  const dir = await readDir(__dirname);
  const seedFiles = dir.filter(f => f.endsWith(".seed.js"));

  for (const file of seedFiles) {
    const fileName = file.split(".seed.js")[0];
    const modelName = toTitleCase(fileName);
    const model = mongoose.models[modelName];

    if (!model) throw new Error(`Cannot find Model '${modelName}'`);
    const fileContents = require(path.join(__dirname, file));

    runSaveMiddleware
      ? await model.create(fileContents)
      : await model.insertMany(fileContents);
  }
}

Why JS, not JSON?

It's the industry norm to use JSON to store data. In this case, I find it easier to use JavaScript objects because:

1. I don't have to write opening and closing double-quotes for each property.
2. I don't have to use double-quotes at all! (It's easier to write single-quotes because there's no need to press the shift key).

// Which is easier to write. JavaScript objects or JSON?

// JavaScript objects
module.exports = [
  {
    objectName: "property"
  }
][
  // JSON
  {
    objectName: "property"
  }
];

If you want to use JSON, make sure you change seedDatabase to work with JSON. (I'll let you work through the code yourself).

Adjusting the setupDB function

Earlier, I created a setupDB function to help set up databases for my tests. seedDatabase goes into the setupDB function since seeding is part of the setting up process.

async function seedDatabase(runSaveMiddleware = false) {
  // ...
}

module.exports = {
  setupDB(databaseName, runSaveMiddleware = false) {
    // Connect to Mongoose
    beforeAll(/*...*/);

    // Seed Data
    beforeEach(async () => {
      await seedDatabase(runSaveMiddleware);
    });

    // Cleans up database between each test
    afterEach(/*...*/);

    // Disconnect Mongoose
    afterAll(/*...*/);
  }
};

A Github Repository

I created a Github repository to go with this article. I hope this demo code helps you start testing your applications.

Thanks for reading. This article was originally posted on my blog. Sign up for my newsletter if you want more articles to help you become a better frontend developer.

Imagine you have an e-commerce website and you’re allowing users to create accounts using their name and email. You want to make sure they sign up with real names, not something like cool_dud3.

That's where we use validation to validate inputs and make sure input data follows certain rules.

In the market, we already have a bunch of validation libraries, but I will compare two important validation libraries: Joi and express-validator for express.js based applications.

This comparison is useful when you have decided to use external input validation library for your application built on expressjs and are somewhat not sure which one to use.

Who is what?

Joi

Joi allows you to create blueprints or schemas for JavaScript objects (an object that stores information) to ensure validation of key information.

Express-validator

express-validator is a set of express.js middlewares that wraps validator.js validator and sanitizer functions.

So by definition, we can say that:

• Joi can be used for creating schemas (just like we use mongoose for creating NoSQL schemas) and you can use it with plain Javascript objects. It's like a plug n play library and is easy to use.
• On the other hand, express-validator uses validator.js to validate expressjs routes, and it's mainly built for express.js applications. This makes this library more niche and provides out of box custom validation and sanitization. Also, I find it easy to understand personally :)

Too many methods and API's for doing certain validation in Joi might make you feel overwhelmed so you might end up closing the tab.

But I may be wrong — so let’s keep opinions aside and compare both libraries.

Instantiation

Joi

In Joi, you need to use **Joi.object()** to instantiate a Joi schema object to work with.

All schemas require Joi.object()to process validation and other Joi features.

You need to separately read req.body , req.params , req.query to request body, params, and query.

const Joi = require('joi');

const schema = Joi.object().keys({
   // validate fields here
})

Express-validator

You can just require express-validator and start using its methods. You don't need to read values from req.body , req.params , and req.query separately.

You just need to use the param, query, body methods below to validate inputs respectively as you can see here:

const {
  param, query, cookies, header 
  body, validationResult } = require('express-validator/check')

app.post('/user', [   

// validate fields here

], (req, res) => {
const errors = validationResult(req);

  if (!errors.isEmpty()) {     
    return res.status(422).json({ errors: errors.array() });   
  }
}

Field is required

Let’s take a very basic example where we want to make sure that a username should be required string and is alphaNumeric with min and max characters.

• Joi:

const Joi = require('joi');
const schema = Joi.object().keys({
    username: Joi.string().alphanum().min(3).max(30).required()
})

app.post('/user', (req, res, next) => {   
  const result = Joi.validate(req.body, schema)
  if (result.error) {
    return res.status(400).json({ error: result.error });
  }
});

• Express-validator

const { body, validationResult } = require('express-validator/check')

app.post('/user', [   
 body('username')
  .isString()
  .isAlphanumeric()
  .isLength({min: 3, max: 30})
  .exists(), 
], (req, res) => {
  const errors = validationResult(req);

  if (!errors.isEmpty()) {     
    return res.status(422).json({ errors: errors.array() });   
  }
}

Sanitization

Sanitization is basically checking input to make sure it's free of noise, for example, we all have used .trim() on string to remove spaces.

Or if you have faced a situation where a number is coming in as "1" so in those cases, we want to sanitize and convert the type during runtime.

Sadly, Joi doesn’t provide sanitization out of the box but express-validator does.

Example: converting to MongoDB’s ObjectID

const { sanitizeParam } = require('express-validator/filter');  

app.post('/object/:id',  
   sanitizeParam('id')
  .customSanitizer(value => {
     return ObjectId(value); 
}), (req, res) => {   // Handle the request });

Custom Validation

Joi: .extend(extension)

This creates a new Joi instance customized with the extension(s) you provide included.

The extension makes use of some common structures that need to be described first:

• value - the value being processed by Joi.
• state - an object containing the current context of validation.
• key - the key of the current value.
• path - the full path of the current value.
• parent - the potential parent of the current value.
• options - options object provided through [any().options()](https://github.com/hapijs/joi/blob/master/API.md#anyoptionsoptions) or [Joi.validate()](https://github.com/hapijs/joi/blob/master/API.md#validatevalue-schema-options-callback).

Extension

extension can be:

• a single extension object
• a factory function generating an extension object
• or an array of those

Extension objects use the following parameters:

• name - name of the new type you are defining, this can be an existing type. Required.
• base - an existing Joi schema to base your type on. Defaults to Joi.any().
• coerce - an optional function that runs before the base, usually serves when you want to coerce values of a different type than your base. It takes 3 arguments value, state and options.
• pre - an optional function that runs first in the validation chain, usually serves when you need to cast values. It takes 3 arguments value, state and options.
• language - an optional object to add error definitions. Every key will be prefixed by the type name.
• describe - an optional function taking the fully formed description to post-process it.
• rules - an optional array of rules to add.
• name - name of the new rule. Required.
• params - an optional object containing Joi schemas of each parameter ordered. You can also pass a single Joi schema as long as it is a Joi.object(). Of course some methods such as pattern or rename won't be useful or won't work at all in this given context.
• setup - an optional function that takes an object with the provided parameters to allow for internal manipulation of the schema when a rule is set. You can optionally return a new Joi schema that will be taken as the new schema instance. At least one of either setup or validate must be provided.
• validate - an optional function to validate values that takes 4 parameters params, value, state and options. At least one of setup or validate must be provided.
• description - an optional string or function taking the parameters as an argument to describe what the rule is doing.

Example:

joi.extend((joi) => ({
    base: joi.object().keys({
        name: joi.string(),
        age: joi.number(),
        adult: joi.bool().optional(),
    }),
    name: 'person',
    language: {
        adult: 'needs to be an adult',
    },
rules: [
        {
            name: 'adult',
            validate(params, value, state, options) {

                if (!value.adult) {
                    // Generate an error, state and options need to be passed
                    return this.createError('person.adult', {}, state, options);
                }

                return value; // Everything is OK
            }
        }
    ]
})

Express-validator

A custom validator may be implemented by using the chain method [.custom()](https://express-validator.github.io/docs/validation-chain-api.html#customvalidator). It takes a validator function.

Custom validators may return Promises to indicate an async validation (which will be awaited upon), or throw any value/reject a promise to use a custom error message.

const {
  param, query, cookies, header 
  body, validationResult } = require('express-validator/check')

app.get('/user/:userId', [   
 param('userId')
  .exists()
  .isMongoId()
  .custom(val => UserSchema.isValidUser(val)), 
], (req, res) => {

const errors = validationResult(req);

  if (!errors.isEmpty()) {     
    return res.status(422).json({ errors: errors.array() });   
  }
}

Conditional Validation

express-validator does not support conditional validation as of now, but there is a PR for that already you can check https://github.com/express-validator/express-validator/pull/658

Let’s see how it works in Joi:

any.when(condition, options)

**any:** Generates a schema object that matches any data type.

const schema = Joi.object({
    a: Joi.any().valid('x'),
    b: Joi.any()
}).when(
    Joi.object({ b: Joi.exist() })
    .unknown(), {
    then: Joi.object({
        a: Joi.valid('y')
    }),
    otherwise: Joi.object({
        a: Joi.valid('z')
    })
});

alternatives.when(condition, options)

Adds a conditional alternative schema type, either based on another key (not the same as any.when()) value, or a schema peeking into the current value, where:

• condition - the key name or reference, or a schema.
• options - an object with:
• is - the required condition joi type. Forbidden when condition is a schema.
• then - the alternative schema type to try if the condition is true. Required if otherwise is missing.
• otherwise - the alternative schema type to try if the condition is false. Required if then is missing.

const schema = Joi
     .alternatives()
     .when(Joi.object({ b: 5 }).unknown(), {
        then: Joi.object({
           a: Joi.string(),
           b: Joi.any()
      }),
      otherwise: Joi.object({
        a: Joi.number(),
        b: Joi.any()
      })
});

Nested Validation

When you want to validate an array of objects/items or just object keys

Both libraries support nested validation

Now what about express-validator?

Wildcards

Wildcards allow you to iterate over an array of items or object keys and validate each item or its properties.

The * character is also known as a wildcard.

const express = require('express'); 
const { check } = require('express-validator/check'); 
const { sanitize } = require('express-validator/filter');  
const app = express(); 

app.use(express.json());  
app.post('/addresses', [   
    check('addresses.*.postalCode').isPostalCode(),
    sanitize('addresses.*.number').toInt() 
], 
(req, res) => {   // Handle the request });

Joi

const schema = Joi.object().keys({
    addresses: Joi.array().items(
        Joi.object().keys({
            postalCode: Joi.string().required(),
        }),
    )
});

Custom Error Messages

Joi

any.error(err, [options])

Overrides the default joi error with a custom error

let schema = Joi.string().error(new Error('Was REALLY expecting a string'));

Express-validator

const { check } = require('express-validator/check'); 

app.post('/user', [   
   // ...some other validations...   
   check('password')     
   .isLength({ min: 5 }).withMessage('must be at 5 chars long')
   .matches(/\d/).withMessage('must contain a number') 
], 
(req, res) => {   // Handle the request somehow });

Conclusion

I covered the most important parts of both libraries and you can decide yourself which one you want to use. Please let me know in the comments below if I left out anything important in the comparison.

I hope you find it helpful when deciding the next input validation module for your express.js application.

I wrote an in-depth article on it here: how to validate inputs. Do check it out.

Don’t hesitate to clap if you considered this a worthwhile read!

Originally published at 101node.io on March 31, 2019.

This course is the perfect way to become familiar with Express.js and how to use it with Node.js. Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.

The code from this project is a good starting place for you to develop your own project.

You can watch the full video on the freeCodeCamp.org YouTube channel (2.5 hour watch).

Hello, world! Recently I had to deploy a website to Heroku for one of the pieces of freelance work I was doing. I think this process may be somewhat difficult, and a detailed tutorial or article on how to do this should help. So this one is going to be very simple and hopefully very short.

We will start by creating an Express app, which will act as our server. Once the server is done, we will create a simple create-react-app application, connect the server with the frontend and, finally, deploy the whole thing to a hosting platform such as Heroku.

Before we go any further, I want you to understand that in the world of web development almost everything is up to one’s preference. Some of you may disagree on certain things, you may continue the way you want to do things, and that’s totally fine. Up to the point when we’re breaking the application I consider everything to be fine.

Let’s get started.

Create a Node/Express app

Start by creating a folder for the overall project. This folder will contain the client side application — our React app in this case. Navigate to the directory in your terminal and type the commands below.

$ touch server.js$ npm init

The last command from the above snippet will take you through some of the steps and will initialise your project with a package.json file. If you are totally new to this, you can consider this file to be a ledger where you keep the record of all the dependencies you’ll be using across the build process of your application.

Moving on, now that we have the package.json file ready, we can start by adding our dependency for the project.

Adding Express:

$ npm i -g express --save

This will add Express as a dependency to your package.json. Now that we have this, all we need is one more dependency and that is for hot reloading of the app whenever we make some change to the code:

$ npm i -g nodemon --save --dev

This will add nodemon to the app. If you would like to read more about nodemon, you can check this link for more information.

Now that we have these added, we are all set to create our most basic server in Express. Let’s see how that’s done.

const express = require('express');const app = express();const port = process.env.PORT || 5000;

//Route setupapp.get('/', (req, res) => {    res.send('root route');

})

//Start serverapp.listen(port, (req, res) => {

console.log(`server listening on port: ${port}`)

 });

That’s it. Just navigate to the terminal, make sure you are in the root directory of the project, and type:

$ nodemon <name-of-the-file> (index.js/server.js)

In our case since we named it server.js it would be nodemon server.js . This will start the server on port 5000 of your computer locally. If you go visit the browser and open https://localhost:5000/ you will see the text “root route”, which means the server has started. In case you face any issues, feel free to add them in the comments below.

Now that the server is set up and is working, let’s proceed towards getting the React app setup.

React app

$ npm i -g create-react-app$ create-react-app <name-of-the-app>

For the purpose of this tutorial we will name the app “client,” so our command will look like this create-react-app client.

Once this is done, the setup will take some time and will create a nice little folder inside your main application with the name “client”.

We will not make any major changes in the overall React application for now — that is outside the scope of this tutorial.

Now the challenge is that we need to call and connect with our server running on the localhost. To do that we need to make an API call.

Adding a proxy

React gives us the ability to do so by adding a proxy value to our package.json file. Navigate to the package.json file in your directory and add the piece of code below.

"proxy": "http://localhost:5000",

In our case, the server is running at port 5000, hence the 5000 in the proxy value. The value may vary if you are using a different port altogether.

Now we need to call the Express-defined endpoints, or API endpoints, from our React components. What that really means is that now we can simply call “api/users/all” from our client side, which will then proxy our request and it will look like this “https://localhost:5000/api/users/all". This saves us from making a cross origin request, which most of the modern browsers do not allow for security reasons.

Next we will make some changes to the src/app.js file.

import React, { Component } from 'react';import './App.css';import Navbar from './Components/Layout/Navbar';import { BrowserRouter as Router, Route } from 'react-router-dom';import Footer from './Components/Layout/Footer';import Home from './Components/Layout/Home';import Social from './Components/social/Social';

class App extends Component {  constructor(props) {    super(props);    this.state = {}    this.connecToServer = this.connecToServer.bind(this);  }

  connecToServer() {    fetch('/');  }

  componentDidMount() {    this.connecToServer();  }

  render() {    return (      <Router>      <div className="container">         <Navbar />         <Route exact path="/" component={Home} />         <Route exact path="/social" component={Social} />         <Footer />      </div>      </Router>    );  }}

export default App;

What we did was to create a constructor, and bind the value of this in our function which will make the fetch API call. Then we call the function as soon as the component is mounted. Next we have the render function which has the overall markup for the app. So that was the last change we will do in React or our frontend application.

Your package.json file should look like the code snippet below.

{  "name": "project-name",  "version": "0.1.0",  "private": true,  "dependencies": {    "react": "^16.6.3",    "react-dom": "^16.6.3",    "react-scripts": "2.1.1",    "react-router-dom": "^4.3.1"  },

  "scripts": {    "start": "react-scripts start",    "build": "react-scripts build",    "test": "react-scripts test",    "eject": "react-scripts eject"  },

  "eslintConfig": {    "extends": "react-app"  },

  "proxy": "http://localhost:5000",

  "browserslist": [    ">0.2%",    "not dead",    "not ie <= 11",    "not op_mini all"  ]}

Now let’s pause for a minute and think about what we need to do next…. any thoughts?

Some of you are right by thinking we need to make sure our React files are being served by our Express server. Let’s make some modifications to the server.js file to make sure that the React files get served by the Express server.

Server file change

const express = require('express');const app = express();const path = require('path');const port = process.env.PORT || 5000;

//Static file declarationapp.use(express.static(path.join(__dirname, 'client/build')));

//production modeif(process.env.NODE_ENV === 'production') {  app.use(express.static(path.join(__dirname, 'client/build')));  //  app.get('*', (req, res) => {    res.sendfile(path.join(__dirname = 'client/build/index.html'));  })}

//build modeapp.get('*', (req, res) => {  res.sendFile(path.join(__dirname+'/client/public/index.html'));})

//start serverapp.listen(port, (req, res) => {  console.log( `server listening on port: ${port}`);})

In the above code snippet, first you need to use the inbuilt path module in node and declare the static folder you would like to use in this Express server.

Then you check if the process is production, which it will be once the app is deployed to Heroku. Under this condition you would like to serve the index.html file from the build folder and not the public folder.

If it’s not the production mode, and you are testing some feature and your server is running on the localhost, you would like the index.html from the public folder to be served.

Now we need to make sure that first we start our Express server and then go about starting our React server. Now there are a lot of ways to do this, and for the simplicity of this tutorial we will be using a module called concurrently to run both the servers with one command.

Make sure you are in the root directory, and then run the command below from your terminal.

npm i concurrently --save

After doing this, let’s make some changes to the scripts we have in our Express server package.json files.

"scripts": {    "client-install": "npm install --prefix client",    "start": "node index.js",    "server": "nodemon index.js",    "client": "npm start --prefix client",    "dev": "concurrently \"npm run server\" \"npm run client\"",

}

• npm run client-install will install all the dependencies for the React application
• npm start will start the server and not reload after detecting any change
• npm run server will start the server, listen for any changes in the code, and hot reload the page on browser to reflect the change.
• npm run client will run the React application without starting the server
• npm run dev will concurrently run the server and then run the client on your browser

Heroku setup

Make sure you have an account on Heroku. If you don’t, you can make one using your GitHub credentials very quickly.

Next we will install the Heroku CLI , which will help us deploy the application right from our terminal. Click here to get install instructions on both macOS and Windows.

$ heroku login

Enter the login credentials for Herkou and then:

$ heroku create

This will create an application for you. If you visit the Heroku dashboard now, it will have the application there.

Now we need to make sure we have a build folder in our project before we push the project to the Heroku repository. Add the script below into your package.json file.

"scripts": {    "client-install": "npm install --prefix client",

    "start": "node server.js",

    "server": "nodemon server.js",

    "client": "npm start --prefix client",

    "dev": "concurrently \"npm run server\" \"npm run client\"",

    "heroku-postbuild":      "NPM_CONFIG_PRODUCTION=false npm install --prefix client        && npm run build --prefix client"  },

After doing this, save the file and push the entire project repository to your Heroku application branch.

//add remote

$ heroku git:remote -a application-name

$ git add .

$ git commit -am 'prepare to deploy'

$ git push heroku master

And that should be it.

Once this is all done, you will get a URL for your live hosted project. Share and showcase what you can build using these technologies.

If you have any questions or comments feel free to add your comment or connect directly.

Github: https://github.com/ashishcodes4

Twitter: https://twitter.com/ashishnandansin

LinkedIn: https://www.linkedin.com/in/ashish-nandan-singh-490987130/

Writing backend logic for any project these days is pretty easy, thanks to full stack JavaScript. This is especially so with the introduction of dozens of frameworks for both client side and server side implementation.

One of the most popular Node.js frameworks is Express.js. It offers an easy approach to building applications at different scales. However, as a project grows, it becomes hard to scale at some point.

Many developers tend to keep adding new route files and models for new services and API end points. This approach works, but it really makes it hard for future engineers to scale and add new services.

In this blog I am going to build a login and registration system that uses JWT authentication with scalable architecture. For those who prefer to get right into the code, go ahead and clone this repository.

There will be four parts in this blog.

1. Basic Architecture Setup
2. Registration
3. Login
4. Dashboard

This blog assumes that you already installed Node.js in your system. Let’s get into the first step — basic architecture setup.

Basic architecture setup

First things first, make a new directory on your file system and call it auth (or anything you like).

mkdir auth

Now cd into that directory and create a package.json file. Add the lines below into it.

{    "name": "auth",    "version": "0.0.0",    "private": true,    "main": "index.js",    "scripts": {      "start": "node index.js"    },    "dependencies": {      "bcrypt": "latest",      "body-parser": "^1.18.2",      "cookie-parser": "~1.4.3",      "express": "~4.15.5",      "jsonwebtoken": "^8.1.1",      "mongoose": "^5.0.3",      "lodash": "^4.17.11",      "morgan": "^1.9.0",      "passport": "^0.4.0",      "passport-jwt": "^3.0.1",      "serve-favicon": "~2.4.5"    }  }

The most important part of the file above is the dependencies property. These are the dependencies required for the project. They will be used as middleware later in this blog.

Now go ahead and run the command below to install all of these dependencies. You may need to wait a few seconds.

npm install

Once it installs all of above dependencies, go ahead and create an index.js file in your root folder, like below:

touch index.js

This particular file is only responsible to start the server. To do that, add below code into it:

'use strict';

const server = require('./server')();const config = require('./configs');const db = require('./configs/db');

server.create(config, db);server.start();

As you can see, this file requires three files:

1. server
2. config
3. db

We will create these next.

The code above then calls the create method on the server module. Finally, it calls the start method, which starts the server.

1. Create the server folder

mkdir server

Once done, cd into that folder and create another index.js file.

touch index.js

Now add the code below into this file:

'use strict';

const express = require('express');const bodyParser = require('body-parser');const logger = require('morgan');const mongoose = require('mongoose');const passport = require('passport');const cookieParser = require('cookie-parser');

module.exports = function() {  let server = express(),      create,      start;

   create = function(config, db) {      let routes = require('./routes');

       // Server settings       server.set('env', config.env);       server.set('port', config.port);       server.set('hostname', config.hostname);

       // Returns middleware that parses json       server.use(bodyParser.json());       server.use(bodyParser.urlencoded({ extended: false }));       server.use(cookieParser());       server.use(logger('dev'));       server.use(passport.initialize());       mongoose.connect(db.database);       require('../configs/passport')(passport);

       // Set up routes       routes.init(server);   };

   start = function() {       let hostname = server.get('hostname'),       port = server.get('port');

       server.listen(port, function () {          console.log('Express server listening on - http://' + hostname + ':' + port);        });    };

    return {       create: create,       start: start    };};

In this file, we first require all of the dependencies needed for this project. Note that more dependencies can be added into this file whenever required.

Then we export an anonymous function from this module using module.exports. Inside that function, create three variables: server, create and start.

The server variable is for the Express.js server. So, call the express() function and assign that to server. We will assign anonymous functions to the create and start variables.

Now, it is time to write a create function with two parameters: config and db.

Then set a few server settings using the server.use() function i.e. env, port and hostname. Then use cookieParser, bodyParser, logger and passport middlewares. Then connect to mongoose database and finally require passport’s configuration file and call it with the required passport.

Passport middleware is used for authentication, which we will use later in this blog. To learn more about it click here.

Now it’s time for API end points i.e. routes. Simply call the init function on routes and pass server into this.

Next, write the start function. Set hostname and port and start the server with the listen command inside this function.

Then, return both create and start functions to make them available for other modules to use.

2. Create the config folder

At the root level, create a configs folder:

mkdir configs

cd into that folder and create an index.js file:

touch index.js

Add the below code to the index.js file:

'use strict';

const _ = require('lodash');const env = process.env.NODE_ENV || 'local';const envConfig = require('./' + env);

let defaultConfig = {  env: env};

module.exports = _.merge(defaultConfig, envConfig);

Now create a local.js file:

touch local.js

Open it, and add the code below:

'use strict';

let localConfig = {  hostname: 'localhost',  port: 3000};

module.exports = localConfig;

This one’s simple too. We are creating a localConfig object and adding a few properties such as hostname and port. Then export it to use it like we are doing in the ./index.js file.

3. Now create a database

touch db.js

Open db.js in your favourite editor and paste the below code into it.

module.exports = {   'secret': 'putsomethingsecretehere',  'database': 'mongodb://127.0.0.1:27017/formediumblog'};

We are exporting a JavaScript object with properties secret and database. These are used to connect with a MongoDB database using middleware called mongoose.

Building the app

Now we are done with basic setup of our project, time for the fun stuff!

cd into the server folder and create the following folders:

mkdir controllers models routes services

First, we will cover the routes folder. This folder is used to add all the end points that are available for client side use. First of all go ahead and create the index.js file first inside the routes folder.

touch index.js

And put the below code into this file:

'use strict';

const apiRoute = require('./apis');

function init(server) {  server.get('*', function (req, res, next) {    console.log('Request was made to: ' + req.originalUrl);    return next();  });

  server.use('/api', apiRoute);}

module.exports = {  init: init};

First, require the apiRoute folder which we are going to create next. This folder will contain another folder with the version number of the API i.e. v1 .

Second create an init function. We are calling this function from the server/index.js file inside the create function at the bottom and passing server as a parameter. It simply gets all the routes and returns the next callback function.

Then use the apiRoute that we are requiring above. Finally, export the init function to make this function available in the rest of the project.

Now go ahead create an apis folder. Inside that folder create a file index.js .

mkdir apistouch index.js

Paste the below code into the index.js file.

'use strict';

const express = require('express');const v1ApiController = require('./v1');

let router = express.Router();

router.use('/v1', v1ApiController);

module.exports = router;

This file requires express and the api version folder i.e. v1. Then create the router and make /v1 end point using router.use() method. Finally export the router.

It’s time to create apis/v1.js file. Paste the below code inside the v1.js file:

'use strict';

const registerController = require('../../controllers/apis/register');const express = require('express');

let router = express.Router();

router.use('/register', registerController);

module.exports = router;

We need to register the controller and express.js and create a router. Then we need to expose register API endpoints for client side use. Finally, we must export the router from this module.

This is the file that we are going to keep modifying. We will require more controllers here when we create them.

Now we are done with the routes folder, and it is time for the controllers folder. Go ahead and CD into that folder and create a folder apis .

mkdir apis

Now that we have the apis folder inside controllers, we are going to create the following three controllers and their respective services.

1. Basic Architecture Setup
2. Registration
3. Login
4. Dashboard

First up is the registerController. Go ahead and create the below file.

touch register.js

Open this file in your favourite editor and paste the below code into it:

'use strict';

const express = require('express');const registerService = require('../../services/authentication/register');

let router = express.Router();

router.post('/', registerService.registerUser);

module.exports = router;

First it is requiring express.js and the register service (which we are going to write later). Then create a router using the express.Router() method and make a post request to the '/' path. Then call the registerUser method on registerService (which we are going to write later). Finally, export the router from this module.

Now we need to require this controller inside the routes/apis/v1.js file which we already did.

Now registering the controller is done. It is time to get to the services folder. CD into that folder and create an authentication folder. First things first, cd into authentication and create a register.js file.

touch register.js

Then open the register.js file and paste the below code into it:

'use strict';

const express = require('express');const User = require('../../models/User');

const httpMessages = {  onValidationError: {    success: false,    message: 'Please enter email and password.'  },  onUserSaveError: {    success: false,    message: 'That email address already exists.'  },  onUserSaveSuccess: {    success: true,    message: 'Successfully created new user.'  }}

// Register new usersfunction registerUser(request, response) {  let { email, password } = request.body;

  if (!email || !password) {    response.json(httpMessages.onValidationError);  } else {    let newUser = new User({      email: email,      password: password    });

    // Attempt to save the user    newUser.save(error => {      if (error) {        return response.json(httpMessages.onUserSaveError);      }      response.json(httpMessages.onUserSaveSuccess);    });  }}

module.exports = {  registerUser: registerUser};

In the register service, first we are requiring expressjs and User model. Then we are creating a JavaScript object i.e. httpMessages which is basically a list of all the messages we are going to send to clients via the api when the client sends the request.

Then the function registerUser which actually performs the registration process. Before saving the user there is a check if the user provided their email and password. If they did then create a newUser using the new keyword with the provided email and password.

Then simply call the save function on newUser to save that user in the database and send the appropriate response using response.json.

Finally export this function using module.exports to make use of it in the rest of the project. We are using this inside the controllers/register.js file.

Before testing this to see if it works, first we need to create a User model. Go ahead create a file User.js inside the models folder.

touch User.js

And paste this code into the above file:

const mongoose = require('mongoose');const bcrypt = require('bcrypt');

const UserSchema = new mongoose.Schema({  email: {    type: String,    lowercase: true,    unique: true,    required: true  },  password: {    type: String,    required: true  },  role: {    type: String,    enum: ['Client', 'Manager', 'Admin'],    default: 'Client'  }});

UserSchema.pre('save', function(next) {  let user = this;

   if (this.isModified('password') || this.isNew) {      bcrypt.genSalt(10, (err, salt) => {        if (err) {          console.log(err);          return next(err);        }

        bcrypt.hash(user.password, salt, (err, hash) => {          if (err) {            console.log(err);            return next(err);          }

          user.password = hash;          next();        });      });  } else {    return next();  }});

// Create method to compare password input to password saved in databaseUserSchema.methods.comparePassword = function(pw, cb) {  bcrypt.compare(pw, this.password, function(err, isMatch) {    if (err) {      return cb(err);    }

    cb(null, isMatch);  });};

module.exports = mongoose.model('User', UserSchema);

First of all require the mongoose and bcrypt modules. Mongoose is used to create mongodb schema whereas bcrypt is used to encrypt passwords before storing them into the database.

Create UserSchema with email, password and role properties. Then before saving the user, perform some checks before hashing the password.

The final function is to compare the passwords. It compares the user’s password with the hashed password in the database.

Now in order to test this code, open postman (if you haven’t installed postman go ahead install it from here). Open postman and enter the below url:

http://localhost:3000/api/v1/register

Select POST as the request, choose the body tab and form-urlencoded and enter the email and password. Press the send button and you should see the below success message.

Now the register part is done.

1. Basic Architecture Setup
2. Register
3. Login
4. Dashboard

It is time to focus on login. Create a login.js file inside the controllers folder.

touch login.js

Now open it and paste the below code:

'use strict';

const express = require('express');const loginService = require('../../services/authentication/login');

let router = express.Router();

router.post('/', loginService.loginUser);

module.exports = router;

Again it’s simple and the same as the register module: after importing express.js and loginService we are creating the router and make a post request to the root path '/' with the loginUser callback function on loginService . Finally export the router.

It’s time to require loginController in the routes/apis/v1.js file. Your v1.js file should look like the below now.

'use strict';

const registerController = require('../../controllers/apis/register');const loginController = require('../../controllers/apis/login');

const express = require('express');

let router = express.Router();

router.use('/register', registerController);router.use('/login', loginController);

module.exports = router;

Now for the login service, create a login.js file inside services/authentication/:

touch login.js

And paste the below code into this file:

'use strict';

const express = require('express');const apiRoutes = express.Router();

const jwt = require('jsonwebtoken');const passport = require('passport');const db = require('../../../configs/db');

const User = require('../../models/User');

const httpResponse = {  onUserNotFound: {    success: false,    message: 'User not found.'  },  onAuthenticationFail: {    success: false,    message: 'Passwords did not match.'  }}

function loginUser(request, response) {   let { email, password } = request.body;

User.findOne({    email: email  }, function(error, user) {    if (error) throw error;

    if (!user) {      return response.send(httpResponse.onUserNotFound);    }

    // Check if password matches    user.comparePassword(password, function(error, isMatch) {      if (isMatch && !error) {        var token = jwt.sign(user.toJSON(), db.secret, {           expiresIn: 10080        });

        return response.json({           success: true, token: 'JWT ' + token        });      }

      response.send(httpResponse.onAuthenticationFail);    });  });};

module.exports = {  loginUser: loginUser};

First require some necessary modules such as: express.js, jsonwebtoken, passport, db and User model. Create a JavaScript object that has a list of messages to be sent to the client side when the http request is made to this service.

Create a loginUser function, and inside that create a couple of variables i.e. email and password, and assign the email and password sent by the user to these variables which are in request.body.

Then use the findOne() method on the User model to find a use based on the email sent from the client by the user. The callback function of findOne() accepts 2 parameters, error and user. First check if the above findOne() method throws any error — if it does then throw an error.

Then perform a check: if no user is found, then send the proper response with a message from the list of messages that we declared above in this module.

Then compare the password that the user sent with the one in the database using the compare function we wrote in the User model earlier in this blog.

If the password matches and it does not return an error, then we create a token using the jsonwebtoken module and return that token using json.response() to the client. Otherwise we send an authenticationFail message.

Finally export the loginUser function with exports.module so that we can use it in our controllers and anywhere else.

It’s time to test login functionality. Go back to postman and this time replace register with login as the api end point in the url. Enter the email and password and press the send button. You should be able to receive a token. Go ahead and copy that to the clipboard because you will use it later to access the dashboard.

1. Basic Architecture Setup
2. Register
3. Login
4. Dashboard

Now it’s time for the dashboard.js file. Create dashboard.js file insidecontrollers folder.

touch dashboard.js

And open it and paste the below code:

'use strict';

const passport = require('passport');const express = require('express');const dashboardService = require('../../services/dashboard/dashboard');

let router = express.Router();

router.get('/', passport.authenticate('jwt', { session: false }), dashboardService.getDashboard);

module.exports = router;

This controller is different in the sense that it requires authenticated access. That is, only a logged-in user can access the dashboard service and make different http requests.

For that reason we are also importing passport, and for the get request we are using the passport.authenticate() function to getDashboard service.

Again we need to require dashboardController in the routes/apis/v1.js file. Your v1.js file should look like the below:

'use strict';

const registerController = require('../../controllers/apis/register');const loginController = require('../../controllers/apis/login');const dashboardController = require('../../controllers/apis/dashboard');

const express = require('express');

let router = express.Router();

router.use('/register', registerController);router.use('/login', loginController);router.use('/dashboard', dashboardController);

module.exports = router;

Now that dashboardController is available to be used for client side requests, it’s time to create its respective service. Go to the services folder and create a dashboard folder inside it. Create a dashboard.js file and put the below code inside this file.

'use strict';

function getDashboard(request, response) {  response.json('This is from dashboard');}

module.exports = {  getDashboard: getDashboard}

No fancy stuff going on. For demonstration purposes, I am simply responding with a text message This is from dashboard. Then export this method to be used in its respective controller which we already accomplished.

Now it’s testing time. Open postman and change the end point of the url to the dashboard. Click the headers tab and add Authorization and paste the JTW copied in the previous step when you logged in.

You should see the message This is from dashboard as a response.

As you can see, when we make a new service we need one controller for it and we can keep adding new services into the architecture. If you would like to change the version of the API and also keep the current one, simply add a new v2.js file and redirect all requests to that end point. That is one simple example.

I hope you liked this blog and see you next time.

UPDATE: If you would like to implement its client side then please click here where I used react.js to authenticate with this server.

I first came across this idea on this website. Since the tutorial there is a few years old, I wrote a new tutorial with updated code.

If you’re in a hurry, you can find the completed code here.

We all know about them or use them — the shiny badges at the top of nearly every ReadMe that say things like “build: passing”, or “size: 10KB”. This guide will teach you how to generate your own badges, with nothing but Node.js, ExpressJS, and Squirrelly.

Now for the tutorial

Prerequisites

This tutorial assumes you have Node.js and npm (or yarn) already installed. If you don’t, go to the Node site here (it installs with npm by default).

Setup

First, create a new directory and cd into it:

mkdir badge-generator && cd badge-generator

Next, install the necessary dependencies, Express and Squirrelly.

With npm:

npm install express squirrelly

Or for those who use yarn:

yarn add express squirrelly

Creating the server

We’ll only need two files for our program, index.js and template.svg (which we’ll create next).

Create a file named index.js and paste the following code:

This opens a server on port 8080, and listens to requests. By the end of this tutorial, you’ll be able to make a request to http://localhost:8080/left-text/right-text/color and have an awesome-looking SVG badge returned! Yay! But what’s the part of the code with Sqrl about?

var badge = Sqrl.renderFile(path.join(__dirname, 'template.svg'), req.params)

This is where Squirrelly comes in. We want to serve an SVG image file, but the content (width, length, and text) of the image will be different each time. Squirrelly is a template engine, a program that takes a file or string called a template and inserts the data. It also does some other fancy stuff, like handling caching, but we won’t need to worry about that.

The code above reads the file named template.svg , then uses req.params (an object that contains the paths) to fill the template. In this case, req.params will look like:

{  left: "first-part-of-the-url-path",  right: "second-part-of-the-url-path",  color: "third-part"}

Creating a template

Create a new file called template.svg, and paste the following code:

You can read the full Squirrelly docs here, but essentially, anything between double brackets: {{ and }} will be replaced by its actual value.

But wait: we only passed in left , right , and color — where did we get leftWidth and rightWidth from? That’s what the code below (at the top of the template) does; using the js helper (which lets you write JS inside of a template), it defines a new variable, called leftWidth .

{{js(options.leftWidth = options.left.length * 10)/}}

There’s one more thing to do. Notice that line 18 looks like this:

<rect ...stuff... fill="{{returnColor(options.color)/}}"/>

With SVG images, the fill attribute must either contain one of a few predefined colors that don’t look that great, or an RGB or hex color. We want to use hex codes, but there’s a catch: you’ll notice that if you enter http://localhost:8080/some/text/#fff into a browser, it thinks the hex code at the end is the hash at the end of the url, and Express doesn’t recognize it.

What we’re going to do is create a helper (called returnColor) that will translate color words, like ‘brightgreen’, ‘green’, and ‘red’, into hex color codes. Paste the following anywhere into index.js:

See if it works

Type node index.js into your terminal, then go to http://localhost:8080/test/badge/brightgreen. If all went well, you should see a badge!

If anything throws an error, compare your code to the working code here.

You can find more information about Squirrelly below.

Squirrelly Documentation
_Squirrelly is only 2KB gzipped, has 0 dependencies, and is blazing fast._squirrelly.js.org

Thanks for reading this guide. I hope it was helpful!

Project Structuring

When I started building Node & Express applications, I didn’t know how important it was to structure your application. Express doesn’t come with strict rules or guidelines for maintaining the project structure.

You are free to use any structure you want. When your codebase grows you end up having long route handlers. This makes your code hard to understand and it contains potential bugs.

If you’re working for a startup, most of the time you won’t have time to refractor your project or modularize it. You can end up with an endless loop of bug fixing and patching.

Over time, while working with both small teams and large teams, I realized what kind of structure can grow with your project and still be easy to maintain.

Model View Controller

The MVC pattern helps in rapid and parallel development. For example, one developer can work on the view, while another one can work on creating the business logic in the controller.

Let’s take a look at an example of a simple user CRUD application.

project/
  controllers/
    users.js
  util/
    plugin.js
  middlewares/
    auth.js
  models/
    user.js
  routes/
    user.js
    router.js
  public/
    js/
    css/
    img/
  views/
    users/
      index.jade
  tests/
    users/
      create-user-test.js 
      update-user-test.js
      get-user-test.js
  .gitignore
  app.js
  package.json

• controllers: Define your app route handlers and business logic
• util: Writes utility/helper functions here which can be used by any controllers. For example, you can write a function like mergeTwoArrays(arr1, arr2).
• middlewares: You can write middlewares to interpret all incoming requests before moving to the route handler. For example,
  router.post('/login', auth, controller.login) where auth is a middleware function defined in middlewares/auth.js.
• models: also a kind of middleware between your controller and the database. You can define a schema and do some validation before writing to the database. For example, you can use an ORM like Mongoose which comes with great features and methods to use in the schema itself
• routes: Define your app routes, with HTTP methods. For example, you can define everything related to the user.

router.post('/users/create', controller.create)
router.put('/users/:userId', controller.update)
router.get('/users', controller.getAll)

• public: Store static images in/img, custom JavaScript files, and CSS /css
• views: Contains templates to be rendered by the server.
• tests: Here you can write all the unit tests or acceptance tests for the API server.
• app.js: Acts as the main file of the project where you initialize the app and other elements of the project.
• package.json: Takes care of the dependencies, the scripts to run with the npm command, and the version of your project.

Exceptions and Error Handling

This is one of the most important aspects to think about when creating any project with any language. Let’s see how to handle errors and exceptions gracefully in an Express app.

Using promises

One of the advantages of using promises over callbacks is they can handle implicit or explicit exceptions/errors in asynchronous code blocks as well as for synchronous code defined in .then(), a promise callback

Just add .catch(next) at the end of the promise chain. For example:

router.post('/create', (req, res, next) => {

   User.create(req.body)    // function to store user data in db
   .then(result => {

     // do something with result

     return result 
   })
   .then(user => res.json(user))
   .catch(next)
})

Using try-catch

Try-catch is a traditional way of catching exceptions in asynchronous code.

Let’s take a look at an example with a possibility of getting an exception:

router.get('/search', (req, res) => {

  setImmediate(() => {
    const jsonStr = req.query.params
    try {
      const jsonObj = JSON.parse(jsonStr)

      res.send('Success')
    } catch (e) {
      res.status(400).send('Invalid JSON string')
    }
  })
})

Avoid using synchronous code

Synchronous code also known as blocking code, because it blocks the execution until they are executed.

So avoid using synchronous functions or methods that might take milliseconds or microseconds. For a high traffic website it will compound and may lead to high latency or response time of the API requests.

Don’t use them in production especially :)

Many Node.js modules come with both .sync and .async methods, so use async in production.

But, if you still want to use a synchronous API use --trace-sync-io command-line flag. It will print a warning and a stack trace whenever your application uses a synchronous API.

For more on the fundamentals of error handling, see:

• Error Handling in Node.js
• Building Robust Node Applications: Error Handling (StrongLoop blog)

_What you should not do is to listen for the uncaughtException event, emitted when an exception bubbles all the way back to the event loop. Using it is generally not preferred._

Logging properly

Logging is essential for debugging and app activity. It is used mainly for development purposes. We use console.log and console.error but these are synchronous functions.

For Debugging purposes

You can use a module like debug. This module enables you to use the DEBUG environment variable to control what debug messages are sent to console.err(), if any.

For app activity

One way is to write them to the database.

Check out How I used mongoose plugins to do auditing of my application .

Another way is to write to a file OR use a logging library like Winston or Bunyan. For a detailed comparison of these two libraries, see the StrongLoop blog post Comparing Winston and Bunyan Node.js Logging.

require(“./../../../../../../”) mess

There are different workarounds for this problem.

If you find any module getting popular and if it has logical independence from the application, you can convert it to private npm module and use it like any other module in package.json.

OR

const path  = require('path');
const HOMEDIR  = path.join(__dirname,'..','..');

where __dirname is the built-in variable that names the directory that contains the current file, and .. ,..is the requisite number of steps up the directory tree to reach the root of the project.

From there it is simply:

const foo = require(path.join(HOMEDIR,'lib','foo'));
const bar = require(path.join(HOMEDIR,'lib','foo','bar'));

to load an arbitrary file within the project.

Let me know in the comment below if you have better ideas :)

Set NODE_ENV to “production”

The NODE_ENV environment variable specifies the environment in which an application is running (usually, development or production). One of the simplest things you can do to improve performance is to set **NODE_ENV**to “production.”

Setting NODE_ENV to “production” makes Express:

• Cache view templates.
• Cache CSS files generated from CSS extensions.
• Generate less verbose error messages.

Tests indicate that just doing this can improve app performance by a factor of three!

Using Process Manager

For production, you should not simply use node app.j — if your app crashes, it will be offline until you restart it.

The most popular process managers for Node are:

• StrongLoop Process Manager
• PM2
• Forever

I personally use PM2.

For a feature-by-feature comparison of the three process managers, see http://strong-pm.io/compare/. For a more detailed introduction to all three, see Process managers for Express apps.

Run your app in a cluster

In a multi-core system, you can increase the performance of a Node app by many times by launching a cluster of processes.

A cluster runs multiple instances of the app, ideally one instance on each CPU core. This distributes the load and tasks among the instances.

Using Node’s cluster module

Clustering is made possible with Node’s cluster module. This enables a master process to spawn worker processes. It distributes incoming connections among the workers.

However, rather than using this module directly, it’s far better to use one of the many tools out there that do it for you automatically. For example node-pm or cluster-service.

Using PM2

For pm2 you can use cluster directly through a command. For example,

# Start 4 worker processes
pm2 start app.js -i 4

# Auto-detect number of available CPUs and start that many worker processes
pm2 start app.js -i max

If you encounter any problems, feel free to get in touch or comment below.
I would be happy to help :)

Don’t hesitate to clap if you considered this a worthwhile read!

References: https://expressjs.com/en/advanced/best-practice-performance.html

Originally published at 101node.io on September 30, 2018.

This tutorial requires prior knowledge of using the expressjs framework

Why do we need server-side validation?

• Your client side validation is not enough and it may be subverted
• More prone to Man in middle attacks, and the server should never trust the client-side
• A user can turn off client-side JavaScript validation and manipulate the data

If you have been building web applications using an Express framework or any other Node.js framework, validation plays a crucial role in any web app which requires you to validate the request body param query.

Writing your own middleware function can be cumbersome if

• you want to move fast while maintaining the quality of code or
• you want to avoid using **if** (**req**.body.head) or **if** (**req**.params.isCool) in your main controller function where you define business logic

In this tutorial, you’ll learn how to validate input in an Express.js app using an open source and popular module called express-validator.

Introduction to express-validator

The definition on Github says:

express-validator is a set of express.js middlewares that wraps validator.js validator and sanitizer functions.

The module implements five important API’s:

• Check API
• Filter API
• Sanitization chain API
• Validation chain API
• Validation Result API

Let's take a look at a basic user route without any validation module to create a user: /route/user.js

/**
* @api {post} /api/user Create user
* @apiName Create new user
* @apiPermission admin
* @apiGroup User
*
* @apiParam  {String} [userName] username
* @apiParam  {String} [email] Email
* @apiParam  {String} [phone] Phone number
* @apiParam  {String} [status] Status
*
* @apiSuccess (200) {Object} mixed `User` object
*/

router.post('/', userController.createUser)

Now in user controller /controllers/user.js

const User = require('./models/user')

exports.createUser = (req, res, next) => {
  /** Here you need to validate user input. 
   Let's say only Name and email are required field
 */

  const { userName, email, phone, status } = req.body
  if (userName && email &&  isValidEmail(email)) { 

    // isValidEmail is some custom email function to validate email which you might need write on your own or use npm module
    User.create({
      userName,
      email,
      phone,
      status,   
    })
    .then(user => res.json(user))
    .catch(next)
  }
}

The above code is just a basic example of validating fields on your own.

You can handle some validations in your user model using Mongoose. For best practices, we want to make sure validation happens before business logic.

express-validator will take care of all these validations and the sanitization of inputs as well.

Installation

npm install --save express-validator

Include module in your main server.js file:

const express = require('express')
const bodyParser = require('body-parser')
const expressValidator = require('express-validator')
const app = express()
const router = express.Router()

app.use(bodyParser.json())

app.use(expressValidator())

app.use('/api', router)

Now using express-validator, your /routes/user.js will be like this:

router.post(
  '/', 
  userController.validate('createUser'), 
  userController.createUser,
)

Here userController.validate is a middleware function which is explained below. It accepts the method name for which the validation will be used.

Let’s create a middleware function validate()in our/controllers/user.js:

const { body } = require('express-validator/check')

exports.validate = (method) => {
  switch (method) {
    case 'createUser': {
     return [ 
        body('userName', 'userName doesn't exists').exists(),
        body('email', 'Invalid email').exists().isEmail(),
        body('phone').optional().isInt(),
        body('status').optional().isIn(['enabled', 'disabled'])
       ]   
    }
  }
}

Please refer to this article to know more about function definition and its use.

The body function will only validate req.body and takes two arguments. First is the property name. Second is your custom message that will be shown if validation fails. If you don’t provide a custom message, then the default message will be used.

As you can see, for a required field we are using the .exists() method. We are using .optional()for an optional field. Similarly isEmail() isInt() is used to validate email and integer.

If you want an input field to include only certain values, then you can use .isIn([]). This takes an array of values, and if you receive values other than the above, then an error will be thrown.

For example, the status field in the above code snippet can only have an enabled or disabled value. If you provide any value other than that, an error will be thrown.

In /controllers/user.js let’s write a**createUser** function where you can write business logic. It will be called after **validate()** with the result of the validations.

const { validationResult } = require('express-validator/check');

exports.createUser = async (req, res, next) => {
   try {
      const errors = validationResult(req); // Finds the validation errors in this request and wraps them in an object with handy functions

      if (!errors.isEmpty()) {
        res.status(422).json({ errors: errors.array() });
        return;
      }

      const { userName, email, phone, status } = req.body

      const user = await User.create({

        userName,

        email,

        phone,

        status,   
      })

      res.json(user)
   } catch(err) {
     return next(err)
   }
}

If you are wondering what is validationResult(req)?

This function finds the validation errors in this request and wraps them in an object with handy functions

Now whenever request includes invalid body params or userName field is missing in req.body, your server will respond like this:

{
  "errors": [{
    "location": "body",
    "msg": "userName is required",
    "param": "userName"
  }]
}

So if userName or email failed to satisfy the validation then each error returned by .array()method has the following format by default:

{   
  "msg": "The error message",

  "param": "param name", 

  "value": "param value",   
  // Location of the param that generated this error.   
  // It's either body, query, params, cookies or headers.   
  "location": "body",    

  // nestedErrors only exist when using the oneOf function
  "nestedErrors": [{ ... }] 
}

As you can see, this module really helps us take care of most of the validations on its own. It maintains code quality as well, and focuses mainly on business logic.

This was the introduction to input validation using the express-validator module and check out how to validate an array of the item and make your own custom validation in Part 2 of this series.

I have tried my best and hope I covered enough to explain it in detail so that you can get started.

If you encounter any problems, feel free to get in touch or comment below.
I would be happy to help :)

Follow Shailesh Shekhawat to get notified whenever I publish a new post.

Don’t hesitate to clap if you considered this a worthwhile read!

Originally published at 101node.io on September 2, 2018.