Conventional database queries often aren't totally reliable in getting accurate search results fast. But fortunately, Elasticsearch comes to the rescue.

In this article, I'll walk you through how to use Elasticsearch to enhance database searches and analytics while still maintaining efficiency.

Here are the prerequisites for this tutorial:

• A Node.js environment

• Basic backend knowledge

With that, let's get started. But first of all, what is Elasticsearch?

Table of Content

• What is Elasticsearch?

• Elasticsearch Key Terms

• How to Set Up Elasticsearch

• How to Set Up the Demo Project

• How to Set Up Elasticsearch in Your Project

• How to Work with Indexes in Elasticsearch

• Search Implementation

• Full Code

• Wrapping Up

• Conclusion

What is Elasticsearch?

Elasticsearch is a search engine built by Apache that can index words and phrases, providing advanced text and vector search capabilities. It also has other useful features such as search analytics and an auto-complete feature.

Note that Elasticsearch isn't a database, even though it does provide indexing features (which popular databases also do).

Other popular alternatives to this tool used in production environments include Algolia, OpenSearch and MeiliSearch.

Elasticsearch Key Terms

in this section, we'll go over some important terminology used in Elasticsearch. To ease your understanding, I'll make references to common database terminologies.

• Index: This serves as a storage location for the data you're going to explore. It's like the database for Elasticsearch. It also shares other properties that DBs possess like uniqueness.

• Document: This is the smallest unit of information stored within the index. It's structurally identical to the MongoDB-based document and is also similar to rows in SQL-based databases.

• Mapping: Mapping refers to sets of rules or instructions that define how documents and fields are stored in the Elasticsearch index.

• Score: This is generated by Elasticsearch to show the degree of relevance of the search query to the stored index.

• Analyzer: When data is sent to the Elasticsearch engine for indexing, it initially passes through an analyzer which processes the text before indexing. This is achieved via Filters and Tokenizers.

• Tokenizers: This tool converts the gross unstructured data sent to the Elasticsearch engine into structured data tokens for further processing and storage.

• Aggregator: This search tool performs detailed analysis on the tokens stored in the index to generate actionable data insights. It's an advantage of the Elasticsearch engine. Mongo DB’s aggregator offer similar functions.

• Filter: A set of instructions which modifies tokens generated during the process of analysis. This could entail removal of fillers, capitalization rules, and so on.

• Bulk index: This refers to indexing more than one document at once. You typically do this when indexing a database with pre-existing content.

How to Set Up Elasticsearch

For the purpose of this tutorial, we'll use Elasticsearch's installable software on our local machine. Online hosted versions of Elasticsearch also exist which work hitch-free as well.

Here is a link detailing how to setup Elasticsearch on Windows. For non-Windows users, you can also install Elasticsearch on Linux/Mac OS or use Docker.

Note that for Windows users, make sure you run Elasticsearch as an Administrator to avoid installation errors.

After successful installation, you can test if it's functioning by navigating to localhost:9200 which serves as the default local endpoint for Elasticsearch. There you'll see a success message on the screen similar to the image below:

With that , we'll move on to setting up our project and integrating ElasticSearch into our demo project.

How to Set Up the Demo Project

For the sake of this tutorial, we will be utilizing a ready-built forum-based backend application built in Node Express JS.  Here is the link to the project.

to get the project up and running, clone this package and run

npm start

MySQL will serve as the default database for this tutorial.  Let's now proceed to the next section.

How to Set Up Elasticsearch in Your Project

The existing demo project is a backend implementation of a forum site which allows users to post text content and facilitate discussions through category-based threads.

Elasticsearch is great for ensuring that users can sift through these posts and threads to accurately locate key content using distinct keywords. This is more effective than using traditional database search queries which can be cumbersome.

To set up Elasticsearch, start by installing the Elasticsearch npm package. To do this, run the command below in your project directory:

npm install @elastic/elasticsearch

After successful installation, create a config.js file where you'll setup your driver to connect to your Elasticsearch application.

const { Client } = require('@elastic/elasticsearch');

const esClient = new Client({
  node: 'http://localhost:9200',
  auth: {
    username: process.env.ELASTICSEARCH_USERNAME,
    password: process.env.ELASTICSEARCH_PASSWORD
  },
  maxRetries: 5,
  requestTimeout: 60000,
  tls: {
    rejectUnauthorized: process.env.NODE_ENV !== 'development'
  }
});

module.exports = esClient;

To access and use Elasticsearch's capabilities within your backend application, you'll need to setup and configure your Elasticsearch driver. The details are specified in the config file code above.

As mentioned earlier, Elasticsearch runs on the localhost:9200 port. So your Elasticsearch node will be directed to the localhost port. Online hosted Elasticsearch nodes will also work in similar scenarios.

Next in the config file, you'll provide the authentication credentials required to access Elasticsearch. The requested username and password will be supplied within the Auth object. If you're running Elasticsearch locally, authentication may not be required unless security is enabled.

In this scenario, MaxRetries refers to the number of maximum unsuccessful attempts to access Elasticsearch. In this case, we've pegged it at 5 attempts. requestTimeout is the time in milliseconds after which the request will automatically terminate if it's not processed.

Once you've completion the Config file, you'll import this config and initialize the Elasticsearch client when your backend starts.

How to Work with Indexes in Elasticsearch

Before we start harnessing the full power of Elasticsearch, we need to customize its search capabilities within the backend of the project. This involves setting up an index within the Elasticsearch Engine that indexes all posts made to the backend application.

const esClient = require('./config');

const setupIndex = async () => {
  try {
    const indexExists = await esClient.indices.exists({
      index: INDEX_NAME
    });

    if (indexExists) {
      console.log(`Index "${INDEX_NAME}" already exists`);
      return;
    }

    await esClient.indices.create({
      index: INDEX_NAME,
      ...indexMapping
    });

    console.log(`Index "${INDEX_NAME}" created`);
  } catch (err) {
    console.error(err);
    throw err;
  }
};

The code above highlights creating a new index. First, you need to invoke the setupIndex() function. Within this function, you're providing the preferred name for your index.  Elasticsearch then checks if the name already exists within its indexes.

The function terminates if the index name already exists (to prevent duplication). But if it doesn't exist, it proceeds to create an index with that unique name alongside the index Mapping rules (which we'll discuss further shortly).

After creating the index, you'll see a success message in your application console.

How to Delete an Index

After a while, an index may no longer serve its purpose and you may need to remove it from Elasticsearch.

You can do this by executing the esClient.indices.delete() command as shown below:

const deleteIndex = async () => {
  try {
    await esClient.indices.delete({ index: INDEX_NAME });
    console.log(`${INDEX_NAME} deleted`);
  } catch (err) {
    console.error("Error deleting index:", err);
  }
};

How to Delete a Post within an Index

Sometimes, posts get deleted and modified. Also, users may get banned, after which you'd want to remove their content from the stored database .

In these cases, you'll want to ensure true deletion – that is, both from the database and from Elasticsearch indexed storage.

To do this, you'll call the esClient.delete() function, passing the Elasticsearch Client ID and the post's unique ID that you want to delete as callback arguments to your esClient.delete function.

const deletePost = async (postId) => {
  try {
    await esClient.delete({
      index: INDEX_NAME,
      id: postId.toString(),
    });

    console.log("Post successfully deleted");
    return { success: true, postId };
  } catch (err) {
    console.error(err);
    throw err;
  }
};

How to Index a Post

After setting up the Elasticsearch Index, you'll want to automatically index posts made to the database into the Elasticsearch index.

To do this, you'll need to make sure that the post is compatible with your index schema via the transformPostTOESRepo function. This function extracts and formats the post data so it matches the Elasticsearch document structure.

const transformPostToESDoc = (post) => {
  return {
  id: post.id,
  title: post.title,
  content: post.body,
  author: post.author,
  category: post.category,
  tags: post.tags,
  views: post.views || 0,
  published_at: post.created_at
};

const indexPost = async (postId) => {
  try {
    const postRepo = await getPostRepo();
    const post = await postRepo.findOne({ where: { id: postId } });

    if (!post) {
      throw new Error("Post not available");
    }

    const esDocument = transformPostToESDoc(post);

    await esClient.index({
      index: INDEX_NAME,
      id: post.id.toString(),
      document: esDocument
    });

    console.log("Post successfully indexed");
    return { success: true, postId };
  } catch (err) {
    console.error(err);
    throw err;
  }
};

The post to be indexed must have a unique ID. For ease of use, we used the unique post ID constraint that comes by default in regular databases. Optionally, you can also use UUID libraries to generate unique post IDs.

The Post information is then attached to the esClient.index() function as the document to be indexed. We also put appropriate error handling measures in place to prevent the app from crashing if the process is unsuccessful.

How to Define Elastic Search Mapping Rules

Elasticsearch mappings define how your data is stored and indexed. They specify the data type of each field and how text is analyzed for search.

In the example below, we'll define an index configuration that includes custom analyzers for autocomplete and mappings for each post field (like title, content, and author).

const indexMapping = {
  settings: {
    analysis: {
      analyzer: {
        autocomplete: {
          type: 'custom',
          tokenizer: 'standard',
          filter: ['lowercase', 'autocomplete_filter']
        },
        autocomplete_search: {
          type: 'custom',
          tokenizer: 'standard',
          filter: ['lowercase']
        }
      },
      filter: {
        autocomplete_filter: {
          type: 'edge_ngram',
          min_gram: 2,
          max_gram: 10
        }
      }
    }
  },
  mappings: {
    properties: {
      id: { type: 'integer' },
      title: {
        type: 'text',
        analyzer: 'autocomplete',
        search_analyzer: 'autocomplete_search',
        fields: {
          keyword: { type: 'keyword' },
          standard: { type: 'text' }
        }
      },
      content: {
        type: 'text',
        analyzer: 'standard'
      },
      category: {
        type: 'keyword'
      },
      tags: { type: 'keyword' },
      author: {
        type: 'text',
        fields: {
          keyword: { type: 'keyword' }
        }
      },
      views: { type: 'integer' },
      published_at: { type: 'date' }
    }
  }
};

The indexMapping object defines how Elasticsearch should store and process your data. It consists of two main parts: settings and mappings.

The mappings section defines the structure of your documents. Each field (like title, content, or author) has a type such as text, keyword, integer, or date. This tells Elasticsearch how to store and search that field.

For text fields, we can also define analyzers. Analyzers control how text is broken into smaller pieces (tokens) during indexing and search.

In the settings section, we defined a custom analyzer for autocomplete. This uses an edge_ngram filter to generate partial word matches, so users can find results as they type. We also defined a separate search_analyzer to ensure that search queries are processed correctly.

Together, these settings allow you to support features like autocomplete while keeping search results accurate and efficient.

Search Implementation

In order to implement your search functionality, you'll need to build out the API. This involves building the business logic service and the API route. You'll also use GET requests and attach your search term as a query. The result it generates will be received as a JSON document.

Then you'll implement the search post service function. In this scenario, you'll be using the search engine capabilities to search for phrases within the index. In line with best practices, you'll use a pagination technique to minimize receiving unwanted information.

The search query will consist of the index name, pagination parameters (from and size) to control which results are returned, and the expected maximum size of the result.  You'll also attach a query object specifying the modality of the search that the Elasticsearch engine should use.

const searchElastic = async (query, page = 1, size = 10) => {
  const searchQuery = {
    index: INDEX_NAME,
    from: (page - 1) * size,
    size,
    query: {
      bool: {
        must: [
          {
            multi_match: {
              query,
              fields: ["title^3", "content"],
              type: "best_fields",
              fuzziness: "AUTO"
            }
          }
        ]
      }
    }
  };

  const result = await esClient.search(searchQuery);
  return result.hits.hits;
};

In the code above, the function is named searchElastic. The function contains three variables which must be passed in order to execute it: size, page and query.

The size variable specifies the maximum number of documents per search query to be returned. The default count could be any integer.

The query uses a multi_match clause to search across multiple fields, such as title and content. The title^3 syntax boosts matches in the title, making them more relevant than matches in other fields.

We also included a must clause which defines conditions that documents must match to be included in the results.

The search results are usually ranked based on their degree of relevance to the search query.

Full Code

With this, you've completed this tutorial and have configured Elasticsearch to index posts made to your database. Here's the full code:

1. Elasticsearch Client (config.js):

const { Client } = require('@elastic/elasticsearch');

const esClient = new Client({
  node: 'http://localhost:9200',
  auth: {
    username: process.env.ELASTICSEARCH_USERNAME,
    password: process.env.ELASTICSEARCH_PASSWORD
  },
  maxRetries: 5,
  requestTimeout: 60000,
  tls: {
    rejectUnauthorized: process.env.NODE_ENV !== 'development'
  }
});

module.exports = esClient;

1. Index mapping:

const indexMapping = {
  settings: {
    analysis: {
      analyzer: {
        autocomplete: {
          type: 'custom',
          tokenizer: 'standard',
          filter: ['lowercase', 'autocomplete_filter']
        },
        autocomplete_search: {
          type: 'custom',
          tokenizer: 'standard',
          filter: ['lowercase']
        }
      },
      filter: {
        autocomplete_filter: {
          type: 'edge_ngram',
          min_gram: 2,
          max_gram: 10
        }
      }
    }
  },
  mappings: {
    properties: {
      id: { type: 'integer' },
      title: {
        type: 'text',
        analyzer: 'autocomplete',
        search_analyzer: 'autocomplete_search',
        fields: {
          keyword: { type: 'keyword' },
          standard: { type: 'text' }
        }
      },
      content: {
        type: 'text',
        analyzer: 'standard'
      },
      category: {
        type: 'keyword'
      },
      tags: { type: 'keyword' },
      author: {
        type: 'text',
        fields: {
          keyword: { type: 'keyword' }
        }
      },
      views: { type: 'integer' },
      published_at: { type: 'date' }
    }
  }
};

1. Create index:

const setupIndex = async () => {
  try {
    const indexExists = await esClient.indices.exists({
      index: INDEX_NAME
    });

    if (indexExists) {
      console.log(`Index "${INDEX_NAME}" already exists`);
      return;
    }

    await esClient.indices.create({
      index: INDEX_NAME,
      ...indexMapping
    });

    console.log(`Index "${INDEX_NAME}" created`);
  } catch (err) {
    console.error(err);
    throw err;
  }
};

1. Delete index:

const deleteIndex = async () => {
  try {
    await esClient.indices.delete({ index: INDEX_NAME });
    console.log(`${INDEX_NAME} deleted`);
  } catch (err) {
    console.error("Error deleting index:", err);
  }
};

1. Delete document (post):

const deletePost = async (postId) => {
  try {
    await esClient.delete({
      index: INDEX_NAME,
      id: postId.toString()
    });

    console.log("Post successfully deleted");
    return { success: true, postId };
  } catch (err) {
    console.error(err);
    throw err;
  }
};

1. Transform and index post:

const transformPostToESDoc = (post) => {
  return {
  id: post.id,
  title: post.title,
  content: post.body,
  author: post.author,
  category: post.category,
  tags: post.tags,
  views: post.views || 0,
  published_at: post.created_at
};

const indexPost = async (postId) => {
  try {
    const postRepo = await getPostRepo();
    const post = await postRepo.findOne({ where: { id: postId } });

    if (!post) {
      throw new Error("Post not available");
    }

    const esDocument = transformPostToESDoc(post);

    await esClient.index({
      index: INDEX_NAME,
      id: post.id.toString(),
      document: esDocument
    });

    console.log("Post successfully indexed");
    return { success: true, postId };
  } catch (err) {
    console.error(err);
    throw err;
  }
};

1. Search function:

const searchElastic = async (query, page = 1, size = 10) => {
  const searchQuery = {
    index: INDEX_NAME,
    from: (page - 1) * size,
    size,
    query: {
      bool: {
        must: [
          {
            multi_match: {
              query,
              fields: ["title^3", "content"],
              type: "best_fields",
              fuzziness: "AUTO"
            }
          }
        ]
      }
    }
  };

  const result = await esClient.search(searchQuery);
  return result.hits.hits;
};

Wrapping Up

Now you know how to use Elasticsearch to improve user search in your web applications. Elasticsearch is agnostic which allows you to use it across programming languages and frameworks. Its large community base also provides helpful user guides to make onboarding easier.

To further harness Elasticsearch's power, you can explore other tools within the ELK stack (Elasticsearch, Log Stash, and Kibana ) that'll help you generate high quality data visualizations for your data, especially for enterprise applications.

Conclusion

A fast and reliable search engine isn’t negotiable in your web applications these days. Elasticsearch is your go-to for getting this done.

If you would like to read other articles that will enhance your tech journey, feel free to check out my website here . Stay active!

Integrating multifactor authentication capabilities into applications is crucial for strengthening their integrity. In social apps, authentication mechanisms eliminate unwanted access to personal information between two parties. Facial authentication is not entirely new, as most devices have it built-in as security measure. It offers stronger protection compared to many traditional methods, especially against risks like phishing, brute-force attacks, and account hacking.

Outline

• What to expect

• Prerequisites

• A Brief Intro to the Face API tool

• Project Setup

• Demo Project: Integrating Facial Recognition and Authentication

• Additional Information and Tips

• Conclusion

What to Expect

In this article, I’ll walk you through creating a multi-factor authentication system for a chat application powered by Stream.io, and ensuring efficient user face ID authentication to allow only authorized access to your app. I will illustrate all these with relevant code examples.

Prerequisites

Here are the necessary prerequisites to follow along with this tutorial:

• Intermediate knowledge of Node.js/Express for the backend aspect

• Knowledge of React for the frontend aspect

• Stream.io API key

Before we get started, we’ll briefly highlight the facial authentication tool of choice: Face-Api.js.

A Brief Intro to the Face API tool

Face-Api.js is a facial recognition package designed for integration with JavaScript-powered applications. It was built on top of the Tensor flow library and provides extensive facial detection based on machine learning models and abstract calculations.

In addition to all these features, it's friendly to use and can also be used locally with its predefined models. Here is a link to its documentation page, which provides relevant code examples.

It provides features such as face detection, face capture, and face match, which use the Euclidean algorithm to make precise distinctions. We'll now set it up alongside our chat application project in the next section.

Project Setup

As mentioned earlier, this is a full-stack project containing both the frontend and the backend aspects. In this section, we’ll set up both code bases before proceeding to the demo project section.

Frontend

We will power the application using the Vite framework for the frontend.

npm create vite@latest

After creating the React application, install face-api.js with this command:

npm i face-api.js

This will install the face package and the required dependencies. You can then install Stream’s powered chat SDK, which will form the main crux of the project.

npm i stream-chat stream-chat-react

After successful completion, we are finally done with the project structure scaffold. To aid ease of local testing of our frontend application, we will have to host the face models needed by the Face package locally. Here is a link to the models. Kindly copy the model's folder and paste it into the public folder in the code project. Next, we’ll set up our backend project.

Backend

The backend is built to store user details and ensure user authentication before accessing the chat application. MongoDB will be the database of choice, and we will use the Express.js library as the backend API development environment of choice. For the ease of setup, kindly clone this code-base and install it on the local PC. It comes preloaded with the necessary installation files. To further enjoy a seamless backend experience, you can utilize the MongoDB Atlas option as the database for storing user details. With that, we will now begin the code project in the next section.

Demo Project: Integrating Facial Recognition and Authentication

In this section, we will walk through setting up an authentication page on the frontend where a user can register their details, username, email, and password on the registration page. They are also obliged to take a snapshot of their face, and the face API will be called to detect a face in the image. They won't be allowed to proceed beyond this until it is successful.

Thereafter, the image faceDescriptor function is called, which generates a unique face description vector value of the user’s face based on the machine learning models provided. These values are securely stored in the MongoDB database via the Express.js backend after successfully registering. The application is coupled to a multifactor authentication system, which has both the password based authentication and the facial authentication mechanisms.

When the first hurdle (password authentication) is completed, the user is then required to take a face match, comparing it with the user's face descriptor stored from the registration page. The comparison is achieved using the Euclidean algorithmic comparison based on the threshold we provide. If it meets the threshold, the face is said to be matched, and the user gets access to the chat application; else, the user is denied access to the Stream.io-powered chat application. Relevant source code snippets highlighting these steps will be provided concurrently with images.

We’ll begin by building a defunct registration page for our chat application using React, of course. We will begin by importing and initializing the necessary packages.

import React, {useState, useRef, useEffect} from 'react';
import * as faceapi from 'face-api.js'
import {useNavigate} from 'react-router-dom'
import axios from 'axios';

const Register =()=> {

    const navigate= useNavigate("/")
    const userRef = useRef();
    const passwordRef= useRef();
    const emailRef = useRef();
    const FullRef = useRef()

In the code snippet above, we imported useful React hooks and initialized our installed Face-api.js tool. Axios will serve as our API request tool of choice for this project. The useRef hook will be used to track the user inputs. We then defined the register function and initialized the various useRef hooks for the various input fields to be inputted.

    useEffect(()=> {
const loadModels =async() => {
await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
await faceapi.nets.faceExpressionNet.loadFromUri('/models');
await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
setModelIsLoaded(true);
                startVideo();
}
  loadModels()  }, [])

In the code above, the useEffect hook is called to ensure that the various locally stored face-api models are initialized and active in our application. The models are stored in the models sub-folder within the public folder. Going forward, after initializing our models, we will now set up our camcorder feature on our webpage.

  const [faceDetected, setFaceDetected] = useState(false);


        // Start video feed
        const startVideo = () => {
            navigator.mediaDevices
                .getUserMedia({ video: true })
                .then((stream) => {
                    videoRef.current.srcObject = stream;
                })
                .catch((err) => console.error("Error accessing webcam: ", err));
        };
        const captureSnapshot = async () => {
            const canvas = snapshotRef.current;
            const context = canvas.getContext('2d');
            context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);
            const dataUrl = canvas.toDataURL('image/jpeg');
            setSnapshot(dataUrl);

            // Generate the face descriptor (128-dimensional vector)
            const detection = await faceapi
                .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
                .withFaceLandmarks()
                .withFaceDescriptor();

            if (detection) {
                const newDescriptor = detection.descriptor;
                setDescriptionValue(newDescriptor)
                console.log( newDescriptor);
               setSubmitDisabled(false)
                stopVid()
            } else {
                console.error("No face detected in snapshot");
            }
        };
    const stopVid =() => {

        navigator.mediaDevices
                .getUserMedia({ video: false })
                const stream = videoRef?.current?.srcObject;
        if (stream) {
            stream.getTracks().forEach(track => {track.stop()})
            videoRef.current.srcObject = null;
            setCameraActive(false)
        }
    }
        // Detect face in the video stream
        const handleVideoPlay = async () => {
            const video = videoRef.current;
            const canvas = canvasRef.current;

            const displaySize = { width: video.width, height: video.height };
            faceapi.matchDimensions(canvas, displaySize);

            setInterval(async () => {
                if (!cameraActive) return ;
                const detections = await faceapi.detectAllFaces(
                    video,
                    new faceapi.TinyFaceDetectorOptions()
                );

                const resizedDetections = faceapi.resizeResults(detections, displaySize);

                canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
                faceapi.draw.drawDetections(canvas, resizedDetections);
                    const detected = detections.length > 0;
                 if (detected && !faceDetected) {
                captureSnapshot();  // Capture the snapshot as soon as a face is detected
            }

                setFaceDetected(detections.length > 0);
            }, 100);
        };

In the code above, we begin by defining a useState array when the user’s face is detected during the sign-up process. Thereafter, the function to trigger the browser camcorder is then activated. With this on, we can then trigger the handlePlayFunction in the code. This function monitors facial detection as highlighted by the face models already initialized. The stopVid function is also triggered when the user’s facial detection has been successfully completed.

In this section, we also activated the browser camcorder tool in our application to provide us with real time video. The CaptureSnapshot function helps to obtain a snapshot from the current video being showcased.

const RegSubmit = async (e) => {
  e.preventDefault();
  console.log("hello");

  try {
    const res = await axios.post(BACKEND_URL, {
      username: userRef.current.value,
      email: emailRef.current.value,
      FullName: FullRef.current.value,
      password: passwordRef.current.value,
      faceDescriptor: descriptionValue,
    });

    console.log(res.data);
    setError(false);
    navigate("/login");
    console.log("help");
  } catch (err) {
    console.error(err);
    setError(true);
  }
};

With all the values obtained, the regSubmit function is then defined. When executed, it stores the provided user details with the face description object on our backend server which can then be accessed in the next section for authentication.

Below is the full registration code.

import React, { useState, useRef, useEffect } from 'react';
import * as faceapi from 'face-api.js';
import { useNavigate } from 'react-router-dom';
import axios from 'axios';

const Register = () => {
  const navigate = useNavigate("/");

  const userRef = useRef();
  const passwordRef = useRef();
  const emailRef = useRef();
  const FullRef = useRef();
  const snapshotRef = useRef(null);
  const videoRef = useRef(null);
  const canvasRef = useRef(null);

  const [modelIsLoaded, setModelIsLoaded] = useState(false);
  const [detections, setDetections] = useState([]);
  const [error, setError] = useState(false);
  const [snapshot, setSnapshot] = useState(null);
  const [cameraActive, setCameraActive] = useState(true);
  const [submitDisabled, setSubmitDisabled] = useState(true);
  const [descriptionValue, setDescriptionValue] = useState(null);
  const [faceDetected, setFaceDetected] = useState(false);

  useEffect(() => {
    const loadModels = async () => {
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
      await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
      await faceapi.nets.faceExpressionNet.loadFromUri('/models');
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      setModelIsLoaded(true);
      startVideo();
    };

    loadModels();
  }, []);

  const RegSubmit = async (e) => {
    e.preventDefault();
    console.log("hello");

    try {
      const res = await axios.post('http://localhost:5000/v1/users', {
        username: userRef.current.value,
        email: emailRef.current.value,
        FullName: FullRef.current.value,
        password: passwordRef.current.value,
        faceDescriptor: descriptionValue
      });

      console.log(res.data);
      setError(false);
      navigate("/login");
      console.log("help");
    } catch (err) {
      console.log(err);
      setError(true);
    }
  };

  const startVideo = () => {
    navigator.mediaDevices
      .getUserMedia({ video: true })
      .then((stream) => {
        videoRef.current.srcObject = stream;
      })
      .catch((err) => console.error("Error accessing webcam: ", err));
  };

  const stopVid = () => {
    navigator.mediaDevices.getUserMedia({ video: false });
    const stream = videoRef?.current?.srcObject;
    if (stream) {
      stream.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
      setCameraActive(false);
    }
  };

  const captureSnapshot = async () => {
    const canvas = snapshotRef.current;
    const context = canvas.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);
    const dataUrl = canvas.toDataURL('image/jpeg');
    setSnapshot(dataUrl);

    const detection = await faceapi
      .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor();

    if (detection) {
      const newDescriptor = detection.descriptor;
      setDescriptionValue(newDescriptor);
      console.log(newDescriptor);
      setSubmitDisabled(false);
      stopVid();

      if (storedDescriptor && isMatchingFace(storedDescriptor, newDescriptor)) {
        setInterval(alert("face matched"), 100);
      } else {
        alert("No Match Found!");
      }
    } else {
      console.error("No face detected in snapshot");
    }
  };

  const handleVideoPlay = async () => {
    const video = videoRef.current;
    const canvas = canvasRef.current;
    const displaySize = { width: video.width, height: video.height };
    faceapi.matchDimensions(canvas, displaySize);

    setInterval(async () => {
      if (!cameraActive) return;

      const detections = await faceapi.detectAllFaces(
        video,
        new faceapi.TinyFaceDetectorOptions()
      );

      const resizedDetections = faceapi.resizeResults(detections, displaySize);
      canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
      faceapi.draw.drawDetections(canvas, resizedDetections);

      const detected = detections.length > 0;
      if (detected && !faceDetected) {
        captureSnapshot();
      }

      setFaceDetected(detected);
    }, 100);
  };

  return (
    <div className="flex flex-col w-full h-screen justify-center">
      <div className="flex flex-col">
        <form className="flex flex-col mb-2 w-full" onSubmit={RegSubmit}>
          <h3 className="flex flex-col mx-auto mb-5">Registration Page</h3>

          <div className="flex flex-col mb-2 w-[50%] mx-auto items-center">
            <input
              type="text"
              placeholder="Email"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={emailRef}
            />
            <input
              type="text"
              placeholder="Username"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={userRef}
            />
            <input
              type="text"
              placeholder="Full Name"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={FullRef}
            />
            <input
              type="password"
              placeholder="Password"
              className="w-full rounded-2xl h-[50px] border-2 p-2 mb-2 border-gray-900"
              required
              ref={passwordRef}
            />

            <div>
              {!modelIsLoaded && cameraActive && !descriptionValue ? (
                <p>Loading</p>
              ) : (
                <>
                  {!descriptionValue && (
                    <>
                      <video
                        ref={videoRef}
                        width="200"
                        height="160"
                        onPlay={handleVideoPlay}
                        autoPlay
                        muted
                      />
                      <canvas
                        ref={canvasRef}
                        width="200"
                        height="160"
                        style={{ position: 'absolute', top: 0, left: 0 }}
                      />
                      <p>
                        {faceDetected ? (
                          <span style={{ color: 'green' }}>Face Detected</span>
                        ) : (
                          <span style={{ color: 'red' }}>No Face Detected</span>
                        )}
                      </p>
                      <canvas
                        ref={snapshotRef}
                        width="480"
                        height="360"
                        style={{ display: 'none' }}
                      />
                    </>
                  )}
                </>
              )}

              {snapshot && (
                <div style={{ marginTop: '20px' }}>
                  <h4>Face Snapshot:</h4>
                  <img
                    src={snapshot}
                    alt="Face Snapshot"
                    width="200"
                    height="160"
                  />
                </div>
              )}
            </div>

            <div className="mt-2">
              <button type="button" onClick={stopVid}>
                Stop Video
              </button>
            </div>

            <button
              disabled={submitDisabled}
              className="mx-auto mt-4 rounded-2xl cursor-pointer text-white bg-primary w-[80%] lg:w-[50%] h-[40px] text-center items-center justify-center"
              type="submit"
            >
              Register
            </button>
          </div>

          <div className="flex flex-col mt-1 w-full">
            <p className="flex justify-center">
              Registered previously?&nbsp;
              <a href="/login" className="text-blue-600 underline">
                Login
              </a>
            </p>
          </div>

          {error && (
            <p className="text-red-600 text-center mt-2">
              Error while registering, try again
            </p>
          )}
        </form>
      </div>
    </div>
  );
};

export default Register;

Going forward, we will be working on our multifactor authentication system. In the code below, we will be highlighting the loginSubmit function which will be triggered when the user email and password credentials are provided for logging in to our chat application. The useRef hook is initialized which ensures that the values passed in the input boxes are parsed to the backend via the Axios request tool.

import React, { useState, useRef, useEffect } from 'react';
import { Link, useNavigate } from 'react-router-dom';
import axios from 'axios';

function Login() {
  const navigate = useNavigate();
  const userRef = useRef();
  const passwordRef = useRef();

  const [error, setError] = useState(false);

  const LoginSubmit = async (e) => {
    e.preventDefault();
    try {
      const res = await axios.post(
        'http://localhost:5000/v1/auth/login',
        {
          email: userRef.current.value,
          password: passwordRef.current.value,
        },
        { withCredentials: true }
      );

      console.log(res?.data);
      setError(false);
      navigate('/confirm-auth');
      console.log(res);
    } catch (err) {
      setError(true);
      console.log(err);
    }
  };
}

The full login page code example will be provided here. After successfully confirming their identity via the use of the password authentication feature, we can then go on to confirm the user’s identity via the use of the face recognition system.

import axios from 'axios';
import React, { useRef, useEffect, useState } from 'react';
import * as faceapi from 'face-api.js';
import { useNavigate } from 'react-router-dom';

First of all, we will set up the app by importing the necessary dependencies as highlighted in the code snippet above.

  useEffect(() => {
    const loadModels = async () => {
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
      await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
      await faceapi.nets.faceExpressionNet.loadFromUri('/models');
    };

    loadModels();
  }, []);

  const handleVideoPlay = async () => {
    const video = videoRef.current;
    const canvas = canvasRef.current;

    const displaySize = { width: video.width, height: video.height };
    faceapi.matchDimensions(canvas, displaySize);

    setInterval(async () => {
      if (!cameraActive) return;

      const detections = await faceapi.detectAllFaces(
        video,
        new faceapi.TinyFaceDetectorOptions()
      );

      const resizedDetections = faceapi.resizeResults(detections, displaySize);
      canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
      faceapi.draw.drawDetections(canvas, resizedDetections);

      const detected = detections.length > 0;
      if (detected && !faceDetected) {
        captureSnapshot();
      }

      setFaceDetected(detected);
    }, 100);
  };

  const startVideo = () => {
    navigator.mediaDevices
      .getUserMedia({ video: true })
      .then((stream) => {
        videoRef.current.srcObject = stream;
      })
      .catch((err) => console.error("Error accessing webcam: ", err));
  };

  const stopVid = () => {
    const stream = videoRef.current.srcObject;
    if (stream) {
      stream.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
      setCameraActive(false);
    }
  };

  const deleteImage = () => {
    setSnapshot(null);
    setDescriptionValue(null);
    setFaceDetected(false);
    setCameraActive(true);
    startVideo();
  };

  const captureSnapshot = async () => {
    const canvas = snapshotRef.current;
    const context = canvas.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);

    const dataUrl = canvas.toDataURL('image/jpeg');
    setSnapshot(dataUrl);
    stopVid();

    const detection = await faceapi
      .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor();

    if (detection) {
      const newDescriptor = detection.descriptor;
      setDescriptionValue(newDescriptor);
      console.log(newDescriptor);
    }
  };

After initializing all the necessary dependencies, we also imported our models as we did in the registration page to detect the user’s face and then generate a face description. We also allowed for the user to delete the snapshot and retake the image as many times as possible.

  const FaceAuthenticate = async (e) => {
    e.preventDefault();

    try {
      const res = await axios.post(
        'http://localhost:5000/v1/auth/face-auth',
        { faceDescriptor: descriptionValue },
        { withCredentials: true }
      );

      console.log(res?.data);
      navigate('/chat');
    } catch (err) {
      console.log(err);
    }
  };

After the face descriptor object gets generated, we then sent it to the backend to compare it with the stored face descriptor obtained at the point of registration. If they match, we get redirected to the chat application. Otherwise, an appropriate error message denying us access to the chat application is displayed.

Here is the code to the FaceAuth page:

import axios from 'axios';
import React, { useRef, useEffect, useState } from 'react';
import * as faceapi from 'face-api.js';
import { useNavigate } from 'react-router-dom';

const FaceAuth = () => {
  const navigate = useNavigate("/");

  const videoRef = useRef(null);
  const canvasRef = useRef(null);
  const snapshotRef = useRef(null);

  const [cameraActive, setCameraActive] = useState(true);
  const [snapshot, setSnapshot] = useState(null);
  const [descriptionValue, setDescriptionValue] = useState(null);
  const [faceDetected, setFaceDetected] = useState(false);

  useEffect(() => {
    const loadModels = async () => {
      await faceapi.nets.tinyFaceDetector.loadFromUri('/models');
      await faceapi.nets.faceLandmark68Net.loadFromUri('/models');
      await faceapi.nets.faceRecognitionNet.loadFromUri('/models');
      await faceapi.nets.faceExpressionNet.loadFromUri('/models');
    };

    loadModels();
  }, []);

  const handleVideoPlay = async () => {
    const video = videoRef.current;
    const canvas = canvasRef.current;

    const displaySize = { width: video.width, height: video.height };
    faceapi.matchDimensions(canvas, displaySize);

    setInterval(async () => {
      if (!cameraActive) return;

      const detections = await faceapi.detectAllFaces(
        video,
        new faceapi.TinyFaceDetectorOptions()
      );

      const resizedDetections = faceapi.resizeResults(detections, displaySize);
      canvas.getContext('2d').clearRect(0, 0, canvas.width, canvas.height);
      faceapi.draw.drawDetections(canvas, resizedDetections);

      const detected = detections.length > 0;
      if (detected && !faceDetected) {
        captureSnapshot();
      }

      setFaceDetected(detected);
    }, 100);
  };

  const startVideo = () => {
    navigator.mediaDevices
      .getUserMedia({ video: true })
      .then((stream) => {
        videoRef.current.srcObject = stream;
      })
      .catch((err) => console.error("Error accessing webcam: ", err));
  };

  const stopVid = () => {
    const stream = videoRef.current.srcObject;
    if (stream) {
      stream.getTracks().forEach((track) => track.stop());
      videoRef.current.srcObject = null;
      setCameraActive(false);
    }
  };

  const deleteImage = () => {
    setSnapshot(null);
    setDescriptionValue(null);
    setFaceDetected(false);
    setCameraActive(true);
    startVideo();
  };

  const captureSnapshot = async () => {
    const canvas = snapshotRef.current;
    const context = canvas.getContext('2d');
    context.drawImage(videoRef.current, 0, 0, canvas.width, canvas.height);

    const dataUrl = canvas.toDataURL('image/jpeg');
    setSnapshot(dataUrl);
    stopVid();

    const detection = await faceapi
      .detectSingleFace(canvas, new faceapi.TinyFaceDetectorOptions())
      .withFaceLandmarks()
      .withFaceDescriptor();

    if (detection) {
      const newDescriptor = detection.descriptor;
      setDescriptionValue(newDescriptor);
      console.log(newDescriptor);
    }
  };

  const FaceAuthenticate = async (e) => {
    e.preventDefault();

    try {
      const res = await axios.post(
        'http://localhost:5000/v1/auth/face-auth',
        { faceDescriptor: descriptionValue },
        { withCredentials: true }
      );

      console.log(res?.data);
      navigate('/chat');
    } catch (err) {
      console.log(err);
    }
  };

  return (
    <>
      <div className="flex w-full h-screen flex-col justify-center">
        <p className="flex flex-col mx-auto items-center text-lg font-semibold mb-3">
          Take a snapshot to confirm your identity
        </p>
        <p className="text-center mb-4">Ensure that the picture is taken in a bright area</p>

        <button
          onClick={startVideo}
          className="flex w-[30%] mx-auto text-center items-center justify-center mb-5 h-[40px] bg-blue-600 rounded-md text-white"
        >
          Turn on Webcam
        </button>

        {!snapshot ? (
          <>
            <video
              className="flex mx-auto items-center rounded-md"
              ref={videoRef}
              width="240"
              height="180"
              onPlay={handleVideoPlay}
              autoPlay
              muted
            />
            <canvas
              ref={snapshotRef}
              width="240"
              height="180"
              style={{ position: 'absolute', top: 0, left: 0 }}
            />
            <button onClick={captureSnapshot} className="mt-4 mx-auto block text-sm text-blue-600 underline">
              Take a snapshot
            </button>
          </>
        ) : (
          <div className="flex w-full justify-center">
            <img
              src={snapshot}
              className="rounded-lg"
              width="240"
              height="180"
              alt="Face Snapshot"
            />
          </div>
        )}

        <div className="flex flex-row w-full justify-evenly mt-5">
          <button
            onClick={deleteImage}
            className="bg-purple-500 text-white p-2 h-[35px] rounded-lg"
          >
            Delete Image
          </button>
          <button
            onClick={FaceAuthenticate}
            className="bg-purple-500 text-white p-2 h-[35px] rounded-lg"
          >
            Upload Image
          </button>
        </div>
      </div>
    </>
  );
};

export default FaceAuth;

Displayed below is how the face authentication page should look like.

Having set up the frontend, let's head to the backend and configure the registration and login endpoint for our project. The entire code to the backend project can be gotten here. We will only be highlighting the faceAuth backend function in this article.

To verify authentication, we will be using the sessions option instead of the JWT option. Important user information will be stored and accessed in the session cookies attached to the requests and responses to the frontend. Here is the faceAuth function:

const faceAuth = async (req, res) => {
  try {
    console.log(req.session);

    const id = req.session.passport?.user;
    console.log(id);


    const user = await User.findById(id);
    console.log(user);

    if (user == null) {
      return res.status(400).json({ err: "User not found" });
    }


  } catch (err) {
    console.error(err);
    res.status(500).json({ err: "Internal Server Error" });
  }
};

First, we defined an asynchronous function named faceAuth. We then obtained the unique ID of the user who had successfully scaled over the initial login process from the request session.

To confirm the similarity of the user's stored face descriptor and the picture sent from the frontend, we utilized the matching face function based on the Euclidean algorithm to confirm the user's identity as done below.

const isMatchingFace = (descriptor1, descriptor2, threshold = 0.6) => {
  // Convert the stored descriptors to Float32Array if they aren't already
  if (!(descriptor1 instanceof Float32Array)) {
    descriptor1 = new Float32Array(Object.values(descriptor1));
  }

  if (!(descriptor2 instanceof Float32Array)) {
    descriptor2 = new Float32Array(Object.values(descriptor2));
  }

  const distance = faceapi.euclideanDistance(descriptor1, descriptor2);
  console.log("Euclidean Distance:", distance);

  return distance < threshold;
};

As stated in the code above, the threshold of similarity of comparison used was 0.6. This is flexible and can be modified to suit the user's preference, as a higher threshold will provide better accuracy overall.
If the function returns true, then the user has been successfully authenticated and can then have access to our chat application. Here is the full code snippet.

const faceAuth = async (req, res) => {
  try {
    console.log(req.session);

    const id = req.session.passport?.user;
    console.log(id);

    const { faceDescriptor } = req.body;
    const user = await User.findById(id);
    console.log(user);

    if (user == null) {
      return res.status(400).json({ err: "User not found" });
    }

    const isMatchingFace = (descriptor1, descriptor2, threshold = 0.6) => {
      // Convert the stored descriptor (object) to a Float32Array
      if (!(descriptor1 instanceof Float32Array)) {
        descriptor1 = new Float32Array(Object.values(descriptor1));
      }

      if (!(descriptor2 instanceof Float32Array)) {
        descriptor2 = new Float32Array(Object.values(descriptor2));
      }

      const distance = faceapi.euclideanDistance(descriptor1, descriptor2);
      console.log("Euclidean Distance:", distance);

      return distance < threshold;
    };

    if (isMatchingFace(faceDescriptor, user.faceDescriptor)) {
      console.log("Face match successful");
      req.session.mfa = true;

      return res.status(200).json({
        msg: "User authentication was successful. Proceed to the chat app.",
      });
    } else {
      return res.status(401).json({ msg: "Face does not match. Access denied." });
    }
  } catch (err) {
    console.log(err);
    res.status(500).json({
      err: "User face couldn't be authenticated. Please try again later",
    });
  }
};

With the main hurdle completed, we can then navigate to our application and have a seamless chat experience.

Additionally, as a safety measure, a rate limiter is also in place to minimize the use of brute-force techniques by malicious individuals to gain access to the chat application.

Additional Information and Tips

The overall aim of these efforts is to achieve a more scalable and secure method of user validation. The threshold can easily be modified and tweaked to improve application accuracy. Alternatively, the AWS Rekognition tool can sufficiently replace the Face API tool with efficient cloud-powered models. The limitations of facial recognition can also be overcome by exploring biometric authentication, as it’s a known fact that each individual's fingerprint is unique, greatly reducing the risk of user compromise.

Conclusion

So far, we have walked through the process of creating an efficient multi-factor facial authentication-based tool to prevent intruder access to our chat application, ensuring and prioritizing the highest level of user privacy. Need an SDK that assures you of a seamless and secure chat experience? Try Stream.io today.

But before we get started, here are some prerequisites for the tutorial:

• An active AWS account

• Basic beginner to intermediate knowledge of Node.js and Express

• Knowledge of Linux commands

With that, let's get started.

Expected takeaways

At the end of this guide, you should be equipped with basic knowledge of:

• AWS EC2 and other hosting mediums

• Linux

• Networking

Table Of Contents

1. Introduction

2. How to Set Up AWS

3. How to Deploy the Node.js Application

4. What is Caddy?

5. Setting Up Caddy

6. Additional Information

7. Conclusion

Introduction

Deploying backend applications on the cloud is pretty straightforward thanks to the advent of public cloud service providers like AWS.

You can host these applications on the cloud using several methods, depending on your application’s complexity and use cases.

For the AWS platform, you could deploy on:

• AWS EC2 (Elastic Cloud Compute): This option lets you deploy a virtual operating system server which serves as the backbone on which the application is hosted. It's suitable for complex monolithic backend applications.

• AWS Lambda: This is popularly termed AWS serverless which allows speedy execution of functions when needed. It doesn’t require consistent uptime hours. This is more suited for simple backend functions with specific use cases.

For the purpose of this tutorial, we will be exploring the AWS EC2 option for deploying a Node.js application. We will also cover using a dedicated reverse proxy to give you easy access to your deployed application.

How to Set Up AWS

Firstly, I assume that you have an AWS account (minimum free tier). If you don't, kindly navigate here and set up your account.

After you have logged in successfully, you will have access to all the various AWS products available. Search for AWS EC2 on the search bar. Once you click that, you will see the EC2 dashboard.

We’ll now create a new EC2 instance.

For cost-effectiveness and simplicity, we’ll use the free tier options made available by AWS. First, you’ll begin by creating an instance name. You can set this according to your preference.

Next, you’ll choose a specific operating system to serve as the operating system for your virtual sever. In this tutorial, I’ll be opting for the Ubuntu Linux distro.

Next you’ll choose a compatible compute instance type. AWS provides free access to t2.micro and t3.micro which will suffice for hosting your application.

This part allows you to configure ssh to enable remote access to your server. In my case, I opted out since I intend to use the EC2 connect console option.

After completing the steps above you’ll configure the network settings. This is where your knowledge of Networking will come in handy.

Automatically, AWS creates and assigns a virtual private cloud network to your soon-to-be-launched EC2 instance. You can customize it to suit your purposes, but for this tutorial, we’ll go with the default VPC assigned. Also, we’ll leave the subnet and auto-assign IP the way it is.

For the firewall settings, it comes with two options: create a security group or use an existing security group. For ease of use with the EC2 instance you’ll create, you can configure the Firewall setting by creating a new security group, specifying the network types, and restricting the IP address of users trying to access the EC2 instance you’ll create. Further, you still might want to use any of your preformed firewall configurations in the “use existing security group” option.

You can leave other sections as they are, and then go ahead and create your EC2 instance.

Once you’ve successfully created the EC2 instance, you can connect to the instance via the options provided.

For the purpose of this tutorial, you’ll use the EC2 instance connect option to access your EC2 instance. Once you’ve clicked on the button, you will then be brought into the Linux console.

To begin, you’ll immediately update your operating system packages using the sudo command.

The sudo apt update command will get this done. With your packages up to date, now you can install the relevant packages needed to get your Node.js application powered on: the Node.js installer package and npm (node package manager) tools, respectively.

Then install them using the sudo apt install nodejs npm command.

With this done, you’ve gotten everything ready to run your application.

How to Deploy the Node.js Application

We’ll use a simple Node.js/Express application in this tutorial. You can find the source code here. Inside your EC2 console, clone the application source code using the git clone https://github.com/rat9615/simple-nodejs-app command.

After successfully cloning the code project, execute the command cd simple-nodejs-app to navigate into the code folder directory. Now, install the various necessary application dependencies that are included in the code project’s package.json file by executing npm install:

After completing the above steps, the npm start command will bring your application to life. You should then see a success message on your screen.

But if you navigate to the EC2 instance public IP address with the Instance IP address sub-port 3000, you’ll still see an error – so what haven’t we completed?

Here is where your knowledge of Networking comes in again. Now, navigate back to your cloud EC2 compute dashboard and go to the security groups. These contain and enforce your inbound and outbound rules.

But before we go on, what are inbound and outbound rules?

• Inbound rules allow you to easily configure access to the cloud resource via any route. They ensure that only those who are expressly authorized are granted access. These routes include, but are not limited to, SSH, HTTP, HTTPS, and TCP.

• Outbound rules allow you to easily configure the flow of cloud resources information to the external world. Only authorized Internet protocol addresses will get access to the information requested.

So now, you’ll be creating a new inbound rule which will give you access to the backend API endpoint at port 3000:

Clicking on the Add rule button allows you to add new network access routes on your EC2 machine. Already setup are the HTTP and TCP traffic routes.

The image above highlights all the included inbound rules in a demo application. Now let’s return to configuring our Networking settings.

Within the security groups in your dashboard, click on the Add rule button at the end of the page.

Select the TCP network type, set the host to 0.0.0.0.0/0, and set the port to port 3000. Lastly, to complete the IP address request setup, just click on the “allow from anywhere” option. Save the rule and refresh your tab. Within a few minutes, the endpoint will be made available on port 3000.

To see the application live in action, click on the external IP address of the EC2 instance and add a suffix port 3000 to view the backend application interface. And violà, your application should now be live.

But that’s not all – it's inconvenient to include the sub-port number while browsing the IP address. So you’ll need a compatible effective reverse proxy tool which routes any information from port 3000 to port 80 (which is a general port). A reverse proxy in this scenario routes and forwards requests from clients to a server serving as a protective intermediary. To do this, you can use a tool called Caddy.

What is Caddy?

Caddy is a flexible, beginner-friendly, open-source tool which offers reverse proxy, load balancing, and caching features in addition to its role as a web server. It has an extensive user guide and documentation which you can lookup here.

Now, let’s set up Caddy on your EC2 instance.

Setting up Caddy

Install Caddy

First, you’ll need to install Caddy on your Linux OS. You can do this by executing sudo apt install caddy.

After completing the installation process, the above image should be what you should see on your screen. Once you’ve done this, make sure that Caddy is enabled and powered on by running this command:

caddy run

After running the command above, the Caddy tool should be enabled and active. Then you should see the below image on your screen:

Next, navigate to the external IP address available on your EC2 instance. There, displayed on the screen, should be the Caddy home page.

Configure Caddy

Going back to your console, you’ll need to tweak the Caddy configuration file to enable the reverse proxy of port 3000 to the default port 80.

To access the config file, navigate to the file from the route directory via the cd /etc/caddy/Caddyfile command.

You can then use the sudo nano command to open the config file for editing. The sudo prefix gives you admin privileges that allow you to make necessary changes on the config file.

Within the port 80 section in the config file, add this line of code:

reverse_proxy :3000

Save the file – and now you’re done. You can restart the Caddy tool to reflect the new settings by running sudo systemctl restart caddy. Refreshing the IP address will display your powered-on Node.js application. With this, you’ve successfully hosted your Node.js application on EC2 and utilized a reverse proxy tool to bring it to life.

Additional Information

Going forward, it's good practice to enable DNS features with domain name customization, as users may not always remember the IP address. You can also further increase your expertise in deploying backend applications by experimenting with powering backend functions with AWS lambda.

Conclusion

With this, we have come to the end of the tutorial. I hope you’ve learned about AWS operations and setting up a backend application on the cloud. You can also reach out to me on my blog and check out my other articles here. Till next time, keep on coding!

Fortunately, Docker comes to the rescue. But you might be wondering – what is Docker, and how does it help? You’ll learn all this and more in this tutorial.

But before we start, here are some prerequisites:

• Knowledge of Linux commands

• Knowledge of terminal usage

• Knowledge of Node.js and Express.js

Table Of Contents

1. What is Docker?

2. How to Install Docker

3. Demo Project: How to Containerize a Node.js Application

4. Wrapping Up

What is Docker?

Docker is an open-source tool that makes it easy to run software in a consistent way, no matter where you are. It does this by putting your application and everything it needs (like libraries and settings) into a container (which I’ll discuss more in a moment).

Think of a container like a box: it holds your app and all its parts, so it works exactly the same on your laptop, a server, or in the cloud. Docker helps developers avoid the "it works on my machine" problem by ensuring everything is packaged together in a reliable and portable way.

Docker was created by Solomon Hykes in 2013. Over the years, it has evolved to cover a wide range of tools. It’s become a go-to tool for improving the application deployment and networking processes.

Before we proceed, here are some key terms you will come across as we go through this tutorial:

Docker Engine

The Docker engine, as its name implies, is the powerhouse for Docker applications. It has a client and a server component. The Docker client, in our case, is the command-line interface tool or Docker terminal we’ll be using to send relevant commands for project execution. The Docker server, popularly known as the daemon, is the server that handles running the various Docker images and containers.

Docker Image

Docker images are premade templates of executable software and systems. Docker offers a wide range of images ranging from operating system templates to server templates, software templates, and so on. You can find all these on the Docker hub registry where these images are stored.

You can also build a specific image and host it either publicly on the Docker hub or in a private registry.

Docker Containers

Docker containers are executable compact instances built on the template generated which is the Docker image. They’re lightweight, portable packages that include everything needed to run a piece of software—code, runtime, libraries, and system tools. A container ensures the application runs consistently regardless of the environment.

Benefits of Using Docker

Here are some of the benefits of using Docker as a backend developer:

• Docker is a great tool for creating a solid DevOps culture for application development, as it clarifies the functions of the development and operations teams.

• It’s also quite flexible, allowing for easy deployment of microservices and distributed monolithic backend applications.

• It also minimizes errors from dependency misconfigurations during installations as it ports the app with its necessary dependencies all at once.

Moving on, we will be diving into how to Dockerize a Node.JS Express application. But before that, you’ll need to install Docker on your computer. You can skip this if you already have it installed.

How to Install Docker

Docker is a cross-platform tool which can be installed across all popular operating systems (Windows, Mac OS, and Linux distros). For this tutorial, I’ll only be highlighting how to set up Docker on Windows.

If you’re currently using any OS other than Windows, you can easily set Docker up by following the steps in the Docker documentation here.

For windows users, it is essential that your PC meets the minimum specifications – otherwise the installation won't be successful. The minimum requirements are the following:

• A Windows OS version not less than Windows 10 home

• A PC with WSL-2 installed or Hypervisor enabled.

With that, let's move on to downloading the Docker installer executable. You can download the latest Docker installer from here. After you do that, run the software and accept the terms and conditions. On successful completion, launch the application. This is what you should see:

To confirm that you’ve successfully installed the application, navigate to the command prompt terminal and run Docker –-version. You should see the exact version of the Docker engine tool you’ve installed if it was successful.

We’ll now move on to the project proper.

Demo Project: How to Containerize a Node.js Application

In this section, we will be containerizing a simple Node.js-based backend service with minimal dependencies. This will show you how to containerize and port an application using a Docker application containerization technique known as the Dockerfile. Keep in mind that if you have a more complex application, it may be better to use the Docker compose YAML tool.

To begin with, we will set up the sample Node.js application. I’ll provide the entire code setup in this article, below. But first, let’s understand what a dockerfile is.

What is a Dockerfile?

Basically, a Dockerfile is a template system which allows the user to input commands which, when executed, can produce a functional image of the application. This image can then be converted into a container.

Here are some commands included in the basic structure of a Dockerfile:

• CMD: sets the default command to run if no command is specified when the container starts. It can be overridden by providing a command when running the container (docker run ...).

• ENTRYPOINT: Specifies the main command that always runs when the container starts. It’s not easily overridden, but arguments can be appended.
  Note that CMD and ENTRYPOINT both specify what command or process the container should run when it starts. But they’re used differently and have distinct purposes. Use CMD for default behavior that can be overridden. Use ENTRYPOINT for a fixed command that defines the container's primary purpose.

• FROM: This is usually the opening statement in a Dockerfile. This command fetches a base image which forms the foundation for building the image of the application in question. For instance, in our application, the base image for a Node.js application is to have the baseline Node.js engine installed.

• WORKDIR: This syntax defines the active working directory where the application files will live within the defined container. An automatic folder will be created if it’s not already available.

• COPY: This syntax is used to ensure that the files necessary for creating the Docker image from the code base project file are copied into the newly created Docker container. The directories of these files are carefully highlighted.

• RUN: This syntax specifies the script that you want to be run before completing the application’s containerization.

• ENV: This syntax is used to highlight environmental variables and secrets which will be invoked during the process of running the application.

• EXPOSE: This syntax maps out the browsing port where the application is used to communicate with the external internet. For example EXPOSE: 3000 maps out the application web interface to localhost:3000.

Diving deeper into Docker, let’s quickly go over some key Docker commands we’ll be using throughout this tutorial:

• Docker ps: This command lists all the running containers on your Docker terminal.

• Docker run: This command executes a Docker image to trigger an instance of a container.

• Docker build: This command works based on the Docker file to generate an image of a service or application.

• Docker rm: this command can be used to delete an image using the image identification details.

How to Containerize the App

Now we can start containerizing our simple Node/Express application. To follow along with the tutorial, you can get the base code from here.

On testing it locally, it returns a CRUD API where you can create, fetch, update, and delete products when executed. We’ll package the application for easy deployment on the cloud using our Docker engine. We’ll be able to do this using the Dockerfile tool we discussed above.

Step 1: Create the dockerfile

In your project folder, create a file named Dockerfile. Make sure the name is exactly "Dockerfile" (no extension, and case-sensitive in some systems – so make sure it’s capitalized).

If you're using a code editor, simply create a new file named Dockerfile. If you're using a basic text editor, save the file with the name Dockerfile and ensure it doesn’t accidentally save with an extension like .txt.

Then enter the first line:

FROM Node:18-alpine

This command fetches the base image we’ll use to power our Express application which is the Node engine itself.

You might be wondering what the alpine is for. Alpine is a lightweight, much more compressed version of a Docker image. It excludes incorporating additional packages not directly essential to the base operating system. It's advocated as a standard good code practice to use lightweight distros for faster execution and easy use.

Step 2: Set the working directory

WORKDIR /app

This sets the working directory of the image to the /app folder of the container. It makes sure that all file actions occur here and all files are copied into this directory.

Step 3: Copy the necessary files

COPY package.json

This command copies the package.json files which has a list of dependences and packages to be installed to power our application.

Step 4: Execute a setup script

RUN npm install

This command ensures that all the necessary dependencies to power our Node.js applications are installed on the container.

Step 5: Copy the code files

COPY . .

This command ensures that all the files within the local directory get copied into the container file system within the established working directory.

Step 6: Expose the server port

EXPOSE 3000

This command exposes the server port that we intend to use to access the container. In this case it's port 3000.

Step 7: Include the command to bring the container to life

CMD ["npm", "run", "dev"]4

This command is executed a the end in order to power on the Node.js application. It simply runs the npm run dev command which is what you’d use for a development environment. To run it in a production environment, you’d use the npm start command instead.

Having completed this process, here is how the final Dockerfile structure should look:

FROM Node:18-alpine
WORKDIR /app

COPY package.json

RUN npm install

COPY . .

CMD ["npm", "run", "dev"]

Testing the Docker container

To round it up, we will be creating a Docker image of our Node.js application. To do this, execute the command docker build -t nodeapp . . The docker build command builds the image, while the -t allows for specifying the image tag’s details.

In our case, we’re assigning the name nodeapp to the image we will be creating and the image will be created within the working directory.

Congratulations! You have successfully built your first Docker image. To see all the images on your local repo, execute the command docker images.

In order to create a working instance of your image for testing, execute the command docker run nodeapp.

We’re using Mongo DB as our database for this tutorial, so we’ll need to pass the MongoDB URL as an environment variable to the Docker container. Environment variables help you safeguard certain key variables which shouldn’t be exposed to the public. Other variables which can be passed as environment variables include API keys and encryption codes.

To pass the MongoDB URL to the Docker container, we use the -e tag to ensure that Docker recognizes the corresponding value inputted as an environment variable.

docker run -e JWT_SECRETS={enter the value of your choice} -e MONGO_URL={The mongo url of your choice} nodeapp.

To also use the container in the background, just attach the -d tag which represents the detach option. This option allows the container to run in the background despite exiting the command line terminal.

In the event of no errors, navigating to localhost:5000 should also produce something similar to the image below.

Wrapping Up

In this article, you learned about what Docker is and how it works, along with its common commands and how to use it to containerize a backend application. Moving on from the basics, you can also explore other uses of Docker in continuous integration and development. To learn more about Docker, you can check out its documentation here.

I would also recommend using your new knowledge to deploy projects with real-life use cases, as well as exploring networking in Docker applications. To make your app live, you can easily deploy the Docker image you created to any of the popular cloud service providers like AWS, GCP, Azure, and so on.

Feel free to ask me any questions! You can also check out my other articles here. Till next time, keep on coding!

You might be wondering: what is Infrastructure-as-Code? How does it improve the development process and experience, and where does Terraform come into the picture? Well, we’ll explore all this and more in this guide. But before we start, here are some pre-requisites:

• Basic knowledge of the cloud and cloud terminologies

• Access to a PC to implement code examples

• A GCP account

With this, let's get started.

Here’s what we’ll cover:

1. Overview of Infrastructure as Code

2. What is Terraform?

3. Benefits of Terraform

4. Common Terms Used in Terraform

5. Demo Project: How to Write a Terraform Configuration

6. Conclusion

Overview of Infrastructure as Code (IaC)

Infrastructure as code refers to generating cloud infrastructure tools and applications with a code-based configuration document. This process, when running, automates the sequence and process of creating databases, virtual machines, and servers. This improves the user experience by reducing the frequency of manual cloud service deployments, especially for multiple identical services.

There are two distinct approaches to infrastructure as code: the Imperative approach and the Declarative approach.

When you’re using the Declarative approach to infrastructure generation, you simply detail your expected/desired outputs for the Infrastructure to be generated, and then the IaC tool you’re using figures out how to produce that output.

On the other hand, the Imperative approach involves specifying the exact steps needed to achieve the desired infrastructure state. While the Imperative approach seems more suited for complex infrastructure setups, the Declarative approach can work just as well.

Some tools are capable of both approaches while others are only suited to one or the other. Examples of some of the popular IaC tools used globally include Terraform IaC, AWS Cloud Formation, Ansible, and Pulumi, Chef, among others.

Like the name implies – infrastructure as code – the code creating the infrastructure is written in various template languages within the IaC space. Popular template languages include JSON, YAML, ARM template, HCL, Heat Scripts, and so on.

You can also use scripting tools to execute cloud infrastructure. Some popular ones include Bash and PowerShell. These sometimes come preinstalled on most personal computers.

Out of all these tools, though, Terraform is distinct for various reasons – and it’s the one we’ll be examining in this article.

What is Terraform?

Terraform is an open source tool developed by HashiCorp in 2014. It has evolved over the years and now serves as a cloud agnostic infrastructure tool that allows you to create infrastructure across multiple cloud service providers.

Terraform also offers Terraform Cloud, a cloud-based software as a service tool. It allows for cloud-based deployment of cloud tools, instead of using the old local-based methods we had in the defunct Terraform CLI tool.

Also, like other IaC tools which utilize template languages, the template framework used to create infrastructure in Terraform is the HashiCorp template language (HCL).

Benefits of Terraform

Now I’ll highlight some of the benefits of using Terraform as a cloud engineer, along with the tool’s key role in the cloud ecosystem.

1. Declarative Approach

This approach to cloud infrastructure automation ensures that all required infrastructure to be deployed (databases, servers, and so on) is stated explicitly and executed accordingly. This helps avoid conflicts.

2. Conflict Handling

In addition to its efficient cloud tool automation capabilities, Terraform has some robust conflict detection and handling properties. One of the ways it handles conflicts is via the Terraform plan function. This function highlights any perceived or potential conflicts of infrastructure orchestration which allows for easy correction before deployment. I’ll discuss this further in subsequent sections.

3. Cloud Agnostic

Terraform is a multipurpose, multi-cloud automation service provider with efficient infrastructure automation capabilities across the major cloud service providers (AWS, GCP and Azure). It also allows for hybrid and inter-provider automation.

4. User-friendly

Terraform is one of the largest cloud automation tools with the largest user communities out there. It has extensive beginner-friendly tutorials that help you get a quick hang of the tool. Here is a link to its documentation so you can dive in deeper.

5. File Management Capabilities

Terraform automatically creates a local backup of the automation states on your local computer to ensure immediate recall and file handling in case anything goes wrong. It also offers remote backup options to remote cloud service providers where necessary.

6. Version Control

Just like the Git version control system, Terraform has a built-in version control system which lets you track changes to a Terraform file. It also lets you go back to previous versions of your code if there are errors in the present version, for example.

7. Code Reusability

Terraform offers a wide variety of code templates for easy reuse on its developer documentation page.

Now that we’ve highlighted the benefits of Terraform, let’s learn some common terminologies used in Terraform and what they mean.

Common Terms Used in Terraform

Before you start using Terraform, you should be familiar with some key terms that come up a lot. Here’s what you need to know:

1. Providers: in Terraform, a Provider is a programming interface that lets Terraform interact with various APIs and cloud services. For example, you’d use a provider to interface with a cloud service provider like GCP or Azure.

2. Modules: Modules are specifically created within the Terraform framework and serve as reusable components that let you easily orchestrate cloud services. You can also store key information regarding cloud services in a module, and then modify it to ensure uniqueness using module variables.

3. Resources: Resources in Terraform refer to the cloud infrastructure components to be created. Examples include cloud networks, virtual machines, availability zones, and other infrastructures.

4. State: The concept of state in Terraform forms the basis for its efficiency. State keeps track of the current configuration of your infrastructure resources, and contains details about every resource Terraform has created, modified, or deleted. Terraforms version control system uses it to track any changes you make to a code file and uses that information to destroy and provision infrastructure as necessary.

5. Workspace: a Workspace functions sort of similarly to a version control system, as it creates a sort of constraint around a work file. Workspaces let you manage multiple instances of a single infrastructure configuration in a clean and isolated way within the same backend. You can use workspaces to separate environments like development, staging, and production while using the same Terraform configuration.

Demo Project: How to Write a Terraform Configuration

In this section, we’ll be diving deeper into writing our first Terraform file to orchestrate a Google Cloud program virtual machine with just a few lines of code. But before we begin, we’ll discuss the various commands that you should understand before we implement the demo project.

Common Terraform Commands

• Terraform init: This command initializes the Terraform tool and downloads essential cloud provider-specific files. It also establishes a connection between Terraform and the cloud provider in question. In our case, it’s between GCP and the Terraform provider.

• Terraform fmt: This command automatically ensures optimal code formatting and indentation. It ensures orderly execution of the code and minimizes any errors.

• Terraform plan: This command outlines the steps of execution of the Terraform code, and detects any errors that may occur during the process of execution. It also highlights any errors in the Terraform code that may hinder execution. Lastly, it works alongside Terraform state management to detect any change of state and de-provision or generate any additional cloud services if necessary.

• Terraform apply: This command executes the planned Terraform state implemented by the Terraform plan command.

• Terraform destroy: This command is the final command in the Terraform scheme which is used to deactivate or destroy all the cloud services created using the Terraform apply command. It's important to note that you should execute the commands listed above sequentially to ensure that your infrastructure gets created properly.

Creating an IaC-Powered GCP Virtual Machine

Now that you’ve learned these important commands, let’s test them all out by creating our first-ever IaC-powered GCP virtual machine.

In your code editor, type the following code:

provider "google" {
  project = "your-gcp-project-id"  # Replace with your GCP Project ID
  region  = "us-central1"          
  zone    = "us-central1-a"        
}

This code highlights the cloud provider we’re using to generate the cloud resources we need. In our case, it’s the Google cloud program. The name assigned to it is just “google”. Other cloud providers like AWS and Azure are “aws” and “azure” respectively.

The second line identifies the GCP subscription identifier, which is unique to each GCP account (and helps facilitate accurate integration). You should use yours in the space provided.

You’ll also need to include a suitable resource region and resource availability zone. This serves as the physical base for the virtual machine we’ll create so we can run it. In this scenario, I chose the USA central region and 1-a availability zone, respectively. You can read more here about cloud regions and availability zones.

resource "google_compute_instance" "vm_instance" {
  name         = "example-vm"      
  machine_type = "e2-medium"          

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11" 
    }
  }

The code snippet above specifies the exact compute resource that’ll be orchestrated, which in our case is a virtual machine instance coded as “vm_instance”. 'example-vm’ is the name we want to assign to the virtual machine we will be creating for this project. It is important to note that the virtual machine name must be unique too. The type of the virtual machine we opted for was the E2 (General purpose)-medium type VM. You can get more information on Virtual machine types here.

Going further, we also specify the expected booted Operating system (“boot_disk”) which is an image of the Debian Linux Operating system version 11 in my case.

  network_interface {
    network = "default"  # Attach to the default VPC network
    access_config {

    }
  }

output "instance_ip" {
  value = google_compute_instance.vm_instance.network_interface[0].access_config[0].nat_ip
}

To complete the creation of our virtual machine, we need to set up a Virtual Network to allow remote access to the VM. The network interface block connects the virtual machine to the default VPC (Virtual Private Cloud) network provided by GCP. We won’t be able to interface with our virtual machine without the VPC network. The output block also displays the default access IP address in the terminal, which we can use to connect to the virtual machine.

Here is the final expected code:

provider "google" {
  project = "your-gcp-project-id"  # Replace with your GCP Project ID
  region  = "us-central1"          
  zone    = "us-central1-a"       
}

resource "google_compute_instance" "vm_instance" {
  name         = "example-vm"         
  machine_type = "e2-medium"          

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"  
    }
  }

  network_interface {
    network = "default"  # Attach to the default VPC network
    access_config {

    }
  }

output "instance_ip" {
  value = google_compute_instance.vm_instance.network_interface[0].access_config[0].nat_ip
}

Going on from there, we’ll now be executing this code using the commands highlighted in the image below:

The command terraform -v confirms that Terraform has been successfully installed on the terminal. The expected output will be the version of the Terraform tool installed.

The next command executed is the terraform init function which initializes a communication with the cloud service provider, which in our case is GCP. All needed dependencies are also installed.

The terraform fmt command is also run to ensure adequate code formatting and indentation. Then the terraform plan command is sequentially executed.

From the image above, you can see the steps Terraform intends to use to generate the expected Virtual machine.

On successful execution of Terraform plan, we will then execute the terraform apply function to execute the steps outlined by Terraform plan.

This will generate a prompt asking for a confirmation of the Terraform execution as shown above. Typing “Yes” will allow the operation to proceed smoothly.

On successful execution, a success message will be displayed as shown above. With that, we have created our Cloud infrastructure with just code. The terraform destroy command can then be called to remove the created Virtual machines.

Conclusion

In this article, you’ve learned the basics about infrastructure as code. We discussed Terraform, its benefits, and some of its key features and commands. We also illustrated its use in a demo project.

To further enhance your knowledge, you can consult Terraform‘s documentation for more code examples. I would also recommend utilizing your newly gained knowledge to automate a project with real-life uses.

Feel free to message me with any comments or questions. You can also check out my other articles here. Till next time, keep on coding!

Object-relational mapping, as a programming concept, is an efficient standard protocol for facilitating seamless connection with databases. But what does it really mean, and how do you set it up as a developer? We’ll answer these questions and highlight more about object-relational mapping.

Here are the prerequisites:

• Knowledge of Node.js

• Use the Express framework

• An installed MySQL database

Table of Contents

• Table of Contents

• What is an ORM?

• How to Set Up Your Node.js Server

• How to Integrate Relevant Packages

• Demo Project

• Additional Information

What is an ORM?

Object Relational Mapping (ORM) is a database communication concept in programming that involves the abstraction of data types as compatible object-oriented programming variables. It simply eliminates the use of database-defined queries and storage types to allow ease of creating databases via the programming languages.

Its use has been widely adopted in the tech space as has more advantages than conventional database query methods. Here are some of them:

• It reduces the risk of data manipulation: SQL and non-SQL injections involve inputting malicious SQL syntaxes and queries into the database, which can compromise database security. Having an ORM in place adds an input validation scheme feature, and details the expected input variable syntax and processes it accordingly.

• Ease of database communication: ORM serves to simplify the use of databases as a data tool without undergoing the process of learning a different database query language. The ORM schema can be highlighted in an object-oriented fashion in the application language and can be configured to automatically translate the code to queries compatible with the database.

• This feature also allows easy code portability, achieving maintenance of a single database integration code base while changing the database without any adverse outcome. It is highly flexible and can be used in any database of choice.

• It also has additional features included to allow database interactions. Database migration features and version control processes are provided. With these, we have seen some of its benefits, we will then highlight popular ORM tools used globally.

Here are the popular ORM tools:

• SQLAlchemy

• Prisma ORM

• Sequelize

• ActiveRecord

• TypeORM

• Waterline

For this article, we’ll be streamlining our ORM use cases to a basic Node.js project linked to a MySQL database. We’ll use the Sequelize ORM as the tool of choice.

With an average package download of 8.5 million monthly and an active development community, Sequelize boasts robust features that seamlessly integrate databases with backend applications. It also provides a user oriented documentation which helps guide the user on setting up and using the tool.

Here is a link to the documentation. It also offers support for MySQL, DB2, and SQLite Microsoft SQL server, and it offers features such as read replication, lazy loading, and efficient database transaction properties.

Next, we’ll set up our web application and install Sequelize to connect us to a MySQL database hosted locally.

How to Set Up Your Node.js Server

In this section you’ll set up our Node server. Navigate to the command line and execute npm init. This command creates a new Node project structure for you.

Next, install the Express package – this will serve as the backend framework. You can do this by running the npm i express command.

How to Integrate Relevant Packages

For the purpose of this tutorial, we’ll install the Sequelize Node package manager in our Node application in order to set up the ORM communication to the database.

To set this up, execute npm i sequelize.

We’ll use a locally hosted MySQL database. To do this, we’ll install an npm package database driver. In this case, we will be installing mysql2. Here is a link to the package

Run npm i mysql2 to install it.

Let’s move on to configuring the connection to the database and building our demo project.

Demo Project

In this section we’ll build a simple backend server that performs Create-Read-Update-Delete operations, with the Sequelize library serving as the connection pipeline.

In order to begin the project, we’ll have to set up the database connection for our application. We’ll create a database connection file and set up our database credentials. You can name the file SequelizeConfig.

module.exports = {

    HOST: "localhost",

    USER: "root",

    PASSWORD: "",

    DB: "sequel",

    dialect: "mysql"

}

In the code above, the database credentials were specified, along with the host address. In our case, the database is locally hosted, so localhost is the default host.

The database login details were also provided. The user here is the root, while the password was set to an empty string. This should be tweaked to ensure database security. I also created a defunct database named “sequel”.

The dialect refers to the type of database the user intends to use. In our case, the dialect is MySQL. Note that this can also be replicated on a cloud hosted database with the credentials obtained. With that, let's integrate the connection file with the application.

const SequelConfig = require('../config/sequelize');

const Sequelize = require('sequelize');

const sequelize = new Sequelize(SequelCOnfig.DB, SequelCOnfig.USER, SequelCOnfig.PASSWORD, {

    host: SequelCOnfig.HOST,

    dialect: SequelCOnfig.dialect

});

In order to facilitate a connection to the database, the variables in the config file were imported and initialized in the Sequelize setup file.

const db = {};

db.Sequelize = Sequelize;

db.sequelize = sequelize;

db.user = require('../model/user.model')(sequelize, Sequelize);

db.token = require('../model/token.model')(sequelize, Sequelize)

module.exports= db;

This file above imports the config file created previously and initializes the Sequelize library. The code then fetches the database details inputted in the config file and, when executed, creates the database.

Furthermore, the various database models which will be discussed subsequently are then integrated with the defunct database and generates a SQL database table .

To get this up and running, the database file created is invoked using the sequelize.sync() method. Any error encountered is logged and the database connection gets terminated.

db.sequelize.sync().then(() => {

  console.log('user created ');

}).catch(err => {

  console.error(err)

})

We’ll go on to discuss the database models.

Models

const Sequelize = require("sequelize");

module.exports = (sequelize) => {

sequelize.define(

"user", {

firstName: {

type : Sequelize.DataTypes.STRING,

allowNull : false

},

lastName: {

type : Sequelize.DataTypes.STRING,

allowNull : false

},

email : {

type : Sequelize.DataTypes.STRING,

allowNull : false, unique: true

},

password: {

type : Sequelize.DataTypes.STRING,

allowNull : false

},

role:  {

type : Sequelize.DataTypes.STRING,

allowNull : false

}

}

)

}

In the code above, the user model was initialized in Sequelize ORM and the field details were specified: email, role, lastName, and password. The type of data to be received was also specified.

It also provides an option to ensure the uniqueness of the user details, and the option to prevent the user from leaving some fields empty via the use of allowNull = false.

On execution of the application, the Sequelize ORM creates an SQL equivalent of the model as a data table.

Next, we’ll work on the CRUD functions in Node.js.

Create Operation

const createUser = async (userInfo) => {

try {

// Check if the email already exists in the database

const ifEmailExists = await User.findOne({ where: { email: userInfo.email } });

if (ifEmailExists) {

throw new ApiError('Email has already been registered');

}

// Create the new user

const newUser = await User.create(userInfo);

return newUser; // Return the created user object

} catch (error) {

// Handle errors such as validation or uniqueness constraint

throw error;

}

};

The function above highlights the controller function for creating user entries in the Express server.

The function is asynchronous, which allows for execution of some commands before eventual execution. The code ensures that the user email doesn’t exist in the database before cresting a new user.

In addition, we also ensured that each email field is unique. If the user details are entered into the database successfully, a “successful” response is sent back to the server. Additionally, any error encountered leads to termination of the function and the error gets sent back to the server.

Read Operation

const FetchUser = async (userId) => {

let userDets;

if (userId) {

// Fetch a single user by ID if userId is provided

userDets = await User.findOne({ where: { id: userId } });

// Check if the user exists

if (!userDets) {

throw new ApiError(httpStatus.NOT_FOUND, 'User not found');

}

} else {

// Fetch all users if no userId is provided

userDets = await User.findAll();

// Check if any users were found

if (userDets.length === 0) {

throw new ApiError(httpStatus.NOT_FOUND, 'No users found');

}

}

The read operation fetches the desired query and sends it back to the user without modification. The user ID, which should be unique, is used to search for a specific user. In this scenario, we want access to all the users created in the database.

In case the requested query is not found, an appropriate error code is generated.

Update Operation

const updateUser = async (userId, userDetails) => {

// First, find the user by their ID

const user = await User.findOne({ where: { id: userId } });

if (!user) {

throw new ApiError(httpStatus.BAD_REQUEST, "User doesn't exist");

}

// Update the user with the new details

await User.update(userDetails, { where: { id: userId } });

// Fetch the updated user to return it

const updatedUser = await User.findOne({ where: { id: userId } });

console.log('Updated user:', updatedUser); // Log the updated user

return updatedUser; // Return the updated user object

};

The update operation aims to modify the data entered in previous operations. That is, to update some data fields.

In the case of Sequelize, the update method is invoked. To succeed with this, the particular user to be edited must be identified. The code above then generates the updated data field and sends it as the output of a successful request.

Delete Operation

const deleteUser = async (userId) => {

const user = await User.findOne({ where: { id: userId } });

if (!user) {

throw new ApiError(httpStatus.BAD_REQUEST, "User doesn't exist");

}

// Delete the user

await user.destroy();

console.log('Deleted user:', user); // Log the deleted user

return user; // Return the deleted user object (useful for confirmation)

};

The delete operation is invoked when data in the database table needs to be deleted. Sequelize makes provision for this via the use of the destroy method. This method deletes a specific user. When executed, a success response code is displayed.

Additional Information

So far, we have integrated an ORM library to serve as a connection between our backend application and our relational database. We also explored advanced concepts such as database migrations and CRUD operations. To learn more about this, you can explore the documentation and utilize it in building more complex projects, as hands-on learning is much encouraged.

Feel free to reach out to me on my blog and check out my other articles here. Till next time, keep on coding!

Some of the benefits include its use by developers to restrict malicious access to websites, prevent DDoS attacks, conserve website resources, and ensure optimal web server performance.

This article covers the practical aspects of implementing rate limits in a Strapi application using several packages and techniques.

Let's get started.

Table of Contents

• Demo Project

• Koa Rate Limiter

• Custom Strapi Api Rate Limiter

• Express-rate-limiter Implementation

• Conclusion

Demo Project

We'll be building an e-commerce site using Strapi as our backend framework. We'll then set up a rate limiter in our Strapi application to help guarantee our backend security. Postman will serve as our tool for testing the API endpoints. Let's go on to create a default Strapi application.

To create a strapi application, enter npx create-strapi-app@latest {project name} on the command line and follow the commands provided. To make the installation more straightforward, stick with the quick start installation method and your app should be ready.

This installation modality automatically sets up an easy-to-use SQLite database. However, you could choose to use any other SQL database supported by Strapi.

Alternatively, you can download the starter repo for the project from here and install the necessary dependencies via npm install. Thereafter, you can execute the Strapi application by navigating to the Strapi application code folder on the command line and run npm run develop.

On successful execution, you will be provided with the link to the localhost address to customize the application.

Navigating to the link will require you to create an admin login mail and password. Successful completion of this step will give you access to the backend dashboard.

You can utilize the Strapi dashboard UI to create APIs, or you can generate an API using npm generate. The APIs created will be used in completing the setup for the rate limiting functionality. We will be creating a product store for our e-commerce site. To easily set up products, kindly navigate to the Content-Type builder tab on the sidebar.

The content-Type builder manager allows you to create various collections which will come in handy when setting up your APIs. In this case, the product and category collections will be created to enable you set up your product catalogues.

After completing the creation of the collection types, you can easily add your products seamlessly into the backend database. In my case, I created phone brand products for sale.

Also noteworthy is that the collections we created in the Strapi dashboard automatically creates an API folder for us within our codebase. We will then be working on the project codebase subsequently.

The next step in this tutorial is to set up an efficient rate limiter for our Strapi APIs created in the repo using the tools discussed above.

koa2-rate-limit

In this section, we will be using the koa2-rate-limit package to build our project rate limiter. To install the package, navigate to your project folder on the command line and execute npm i koa2-rate-limit. On successful installation, navigate to the middleware subfolder within the API folder and create a code file. For ease of integration, name it as rateLimit.js.

After that, within the rate limit file, import and initialize the koa2-rate limit package.

const RateLimit = require("koa2-ratelimit").RateLimit;

Afterwards, we can configure the koa rate limiter to a specified time interval frame and the total number of requests.

module.exports = (config, { strapi }) => {
  // Configuring the rate limiter middleware
  const limiter = RateLimit.middleware({
    interval: { min: 1 }, // Time window in minutes
    max: 3, // Maximum number of requests per interval
 });

In the code above, the rate limiter middleware was invoked and the time interval in which the rate limit gets applied was set to 1 minute. The maximum number of requests (max) was set to 3 for this tutorial. You can tweak this to suit your preference.

  return async (ctx, next) => {


    try {
      // Apply the rate limiter to the current request
      await limiter(ctx, next);
 } catch (err) {
      if (err.status === 429) {
        // Handle rate limit exceeded error
        strapi.log.warn('Rate limit exceeded.');
        ctx.status = 429;
        ctx.body = {
          statusCode: 429,
          error: 'Too Many Requests',
          message: 'You have exceeded the maximum number of requests. Please try again later.',
 };
 } else {
        // Re-throw other errors to be handled by Strapi's error-handling middleware
        throw err;
 }
 }

The code above defines a middleware which gets executed whenever a function is made on any API. If the requests exceed the given maximum, an error code is outputted. Below is the full code.

'use strict';

/**
 * `RateLimit` middleware
 */
const RateLimit = require("koa2-ratelimit").RateLimit;

module.exports = (config, { strapi }) => {
  // Configuring the rate limiter middleware
  const limiter = RateLimit.middleware({
    interval: { min: 1 }, // Time window in minutes
    max: 3, // Maximum number of requests per interval
 });

  return async (ctx, next) => {

    try {
      // Apply the rate limiter to the current request
      await limiter(ctx, next);
 } catch (err) {
      if (err.status === 429) {
        // Handle rate limit exceeded error
        strapi.log.warn('Rate limit exceeded.');
        ctx.status = 429;
        ctx.body = {
          statusCode: 429,
          error: 'Too Many Requests',
          message: 'You have exceeded the maximum number of requests. Please try again later.',
 };
 } else {
        // Re-throw other errors to be handled by Strapi's error-handling middleware
        throw err;
 }
 }

 };
};

To ensure its seamless integration to all APIs within the Strapi project, the admin middlewares must also be configured.

cconst rateLimit = require('../middlewares/rateLimit');

module.exports = [
 'strapi::logger',
 'strapi::errors',
 'strapi::security',
 'strapi::cors',
 'strapi::poweredBy',
 'strapi::query',
 'strapi::body',
 'strapi::session',
 'strapi::favicon',
 'strapi::public',

 {
   name: 'global::rateLimit',
   config: {},
 },
];

With this, we have successfully configured the rate limiter powered by koa2-ratelimiter. Here are pictures of its execution.

Custom Strapi Api Rate Limiter

Within the rateLimit file in the API/middlewares folder, create a custom rate limiter by initializing a memory store.

const requestCounts = new Map();

Thereafter, define your rate limit function and then configure the rate limiter.

module.exports = (config, { strapi }) => {

  const rateLimitConfig = strapi.config.get('admin.rateLimit', {
    interval: 60 * 1000,  
    max: 3,  
 });

The time interval above is 1 minute while the maximum number of requests that can be made within the specified time interval is 3. You can tweak it to suit your preference.

return async (ctx, next) => {

    const ip = ctx.ip; 
    const currentTime = Date.now();

    if (!requestCounts.has(ip)) {

      requestCounts.set(ip, { count: 1, startTime: currentTime });
 } else {
      const requestInfo = requestCounts.get(ip);


      if (currentTime - requestInfo.startTime > rateLimitConfig.interval) {
        requestInfo.count = 1;
        requestInfo.startTime = currentTime;
 } else {

 }


      if (requestInfo.count > rateLimitConfig.max) {
        strapi.log.warn(`Rate limit exceeded for IP: ${ip}`);

        ctx.status = 429;
        ctx.body = {
          statusCode: 429,
          error: 'Too Many Requests',
          message: 'You have exceeded the maximum number of requests. Please try again later.',
 };
        return;
 }
 }

    await next();
 };
};

Afterwards, a middleware is defined which obtains the user IP address and then stores it in the memory store. The time interval is also set from the current time the request is made and the request count gets updated with every new request made.

If the requests made exceed the maximum expected requests within the time interval of 1 minute in our case, an error is thrown. Here is the full code below.

'use strict';
const requestCounts = new Map();

module.exports = (config, { strapi }) => {

  const rateLimitConfig = strapi.config.get('admin.rateLimit', {
    interval: 60 * 1000,  
    max: 3,  
 });

  return async (ctx, next) => {

    const ip = ctx.ip; 
    const currentTime = Date.now();

    if (!requestCounts.has(ip)) {

      requestCounts.set(ip, { count: 1, startTime: currentTime });
 } else {
      const requestInfo = requestCounts.get(ip);


      if (currentTime - requestInfo.startTime > rateLimitConfig.interval) {
        requestInfo.count = 1;
        requestInfo.startTime = currentTime;
 } else {

        requestInfo.count += 1;
 }


      if (requestInfo.count > rateLimitConfig.max) {


        ctx.status = 429;
        ctx.body = {
          statusCode: 429,
          error: 'Too Many Requests',
          message: 'You have exceeded the maximum number of requests. Please try again later.',
 };
        return;
 }
 }

    await next();
 };
};

Here is a demo of the project.

Express-rate-limiter Implementation

Express rate limiter is also another important package that can be used to implement rate limiting in our project. Right now, this package will be used to implement a route-specific API rate limiting.

The next step in this tutorial is setting up an efficient rate limiter for our Strapi APIs created in the repo.

To set up rate limiters on our Strapi applications, we'll be working mainly on the routes file. This can be navigated to by accessing the src folder within the project root directory. Within the src folder, navigate to the API folder which contains all the API files for the collections created in the Strapi dashboard.

The rate limiter will be enforced in the routes section of each API. For this tutorial, I will be using the products API as a demo API in this article.

'use strict';


/**
 * product router
 */

const { createCoreRouter } = require('@strapi/strapi').factories;

module.exports = createCoreRouter('api::product.product');

This is the initial code setup in the routes.js file in our product API folder. The rate limiting tool of choice for this tutorial is express-rate-limit as it offers much simplicity and user-friendliness coupled with its efficiency. Here is a link to its documentation. To get this installed, navigate to the command line of the project directory and run

npm install express-rate-limit

On completion of its installation, we will be initializing it in the products file already created within the routes folder as follows.

const { rateLimit } = require("express-rate-limit");

Go on and configure the rate limiter to your desired specifications.

const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 3 * 60 * 1000, // 3 minutes
  max: 2, // limit each IP to 2 requests per windowMs
  handler: async (req, res, next) => {
    const ctx = strapi.requestContext.get();
    ctx.status = 429;
    ctx.body = {
      message: "Too many requests",
      policy: "rate limit"
    };
    // Ensure the response is ended after setting the response body and status
    ctx.res.end();
  }
});

module.exports = limiter;

The code above serves to configure the rate limiting parameters we intend to use for the file.

windowMs represents the time interval in milliseconds for the number of requests. In our case, we specified a time of 3 minutes. Also, we specified the maximum number of requests that can be made within that same time frame. In our case, we used 2 for demo purposes.

However, the limit parameter also serves as an alternative to max parameter. Also included is the handler function that gets executed whenever the requests exceed the set number. It returns an Error 429 with an error body containing “Too many requests”.

const { createCoreRouter } = require('@strapi/strapi').factories;

module.exports = createCoreRouter('api::product.product', {
  config: {
    find: {
      middlewares: [
        async (ctx, next) => {
          await new Promise((resolve, reject) => {
            limiter(ctx.req, ctx.res, (error) => {
              if (error) {
                ctx.status = 429;
                ctx.body = { error: error.message };
                reject(error);
              } else {
                resolve();
              }
            });
          });
          await next();
        }
      ]
    }
  }
});

The above code illustrates the use of the Strapi API middleware which serves to ensure that the rate limit is fulfilled before the onward execution of the API requests. It also ensures that the request is terminated when the rate limit gets exceeded. Here is the final code for the project.

'use strict';

/**
 * product router
 */

const { createCoreRouter } = require('@strapi/strapi').factories;
const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 3 * 60 * 1000, // 3 minutes
  max: 2, // limit each IP to 2 requests per windowMs
  handler: async (req, res, next) => {
    const ctx = strapi.requestContext.get();
    ctx.status = 429;
    ctx.body = {
      message: 'Too many requests',
      policy: 'rate limit'
    };
    // Ensure the response is ended after setting the response body and status
    ctx.res.end();
  }
});

module.exports = createCoreRouter('api::product.product', {
  config: {
    find: {
      middlewares: [
        async (ctx, next) => {
          await new Promise((resolve, reject) => {
            limiter(ctx.req, ctx.res, (error) => {

              if (error) {
                ctx.status = 429;
                ctx.body = { error: error.message };
                reject(error);
              } else {
                resolve();
              }
            });
          });
          if (ctx.status !== 429) {
            await next();
          }
        }
      ]
    }
  }
});

Here is an image showing the rate limiting functionality.

You can also download the final code for the project here. Having completed this, you can then go ahead to test the rate limiting functionality of your API. The Strapi application can be run by executing npm run develop in the command line.

Conclusion

With this, we have come to the end of the tutorial. We hope you’ve learned essentially about rate limiting, its uses, tools and best practices.

You can also design multiple rate limiters within the code and implement them in any endpoint of your choice to test it out.

Feel free to drop any questions or comments. Happy coding!

These bad elements access the back-end servers through vulnerable points in the gateways to wreak havoc, stealing relevant info and negatively affecting application performance and efficiency via various forms of API attacks such as SQL and non-SQL-based injections, Distributed Denial-of-Service (DDoS) attacks, code malware, and other methods to exploit vulnerabilities.

In this article, I will be focusing on rate limiting, an important hack that helps protect the back-end API from being exploited by hackers via the use of DDoS, Brute-force attacks and other related malicious activities. But first of all, what does rate limiting mean?

Table of Contents

1. What is Rate Limiting?

2. Importance of Rate Limiting

3. Adoption and Usage of Rate-Limiting by Popular Sites

4. Other Real life Use-cases of API Rate Limiting

5. How Does Rate Limiting Work?

6. Examples of Rate Limiting Algorithms

7. Rate Limiting Best Practices

8. Conclusion

What is Rate Limiting?

Rate limiting is a mechanism put in place to regulate the frequency of requests made by a client to the back-end server. It prevents the repetition of a client request within a defined time frame.

Why do we have to implement rate limiting in the API development? We'll discuss that in the next section.

Importance of Rate Limiting

Here are some of the reasons why rate limiting is used in back-end application development.

DDoS Attacks

First of all, it serves as a preventive measure to mitigate against DDoS attacks. DDoS attacks are malicious attacks on servers which involves flooding the server endpoints with multiple requests, often millions, resulting in reduced server efficiency and disruption of server functions. In most cases, they occur with the use of automated bots.

These attacks can be volumetric, protocol-based or application layer-based. A key example of this form of attack occurred on the GitHub website in the year 2018.

Web Scraping

Rate limiting also plays a role in protecting web applications and web servers from unauthorized web scrapers and web crawlers. These tools, also automated, usually emit requests continually to collect relevant website data which can be exposed to unauthorized. Having a good rate limiter in place helps to prevent all these.

Brute Force Attack

This involves trying to gain access to a server's resources by trying all possible configurations to get access to the resource. This could be done manually but is mostly automated with the use of bots as it is resource-consuming. Rate limiting also proves effective in preventing these form of attacks by disabling the requests if they exceed the required number of requests within a specific time frame.

Resource Optimization

Server requests usually cost the API owners some expenses in terms of running and maintenance cost. Having a rate limiter in place helps to regulate the number of requests the server can handle, help conserve cost and maximize efficiency. Subsequently, we will highlight some algorithms which rate limiters are built on.

Adoption and Usage of Rate-Limiting by Popular Sites

Rate limiting as a security measure has been adopted by a lot of tech products, ranging from large-scale to small-scale applications. For example, Twitter (X) has a rale limit feature implemented in the application programming interfaces they provide to developers.

These interfaces allow access to Twitter sign-up extension and other features made available by Twitter. To guarantee the efficient running of these interfaces, Twitter imposed a rate limit of 50 tweet post requests per user every 24 hours. More details about this can be found here.

Other Real life Use-cases of API Rate Limiting

The use of an application programming interface isn't limited to just what popular sites like Twitter use it for. Here are some other real-life applications of rate-limiting in today's world.

Reducing the Incidence of Spamming

Research reveals that over 160 billion spam emails are sent daily. Hence this has prompted the implementation of rate-limiting to curb the spread of unsolicited messages and spam content via messaging and emailing platforms over a specific time range. By so doing, it encourages responsible use of these platforms.

Tackling Fraudulent Activities

Rate limiting is currently implemented across web applications to help detect unusual web application activities by some users which may possess fraudulent intents. This measure serves to prevent and mitigate the ongoing fraudulent transactions being performed over the application server.

Disabling Malicious User Authentication

Individuals with malicious intent may want to compromise the web servers by taking several measures such as brute force, DDoS and other techniques to take over other users' accounts.

However, several sites have efficient rate limit systems in places which limit the amount of login attempts an individual has to a site within a specific time range. This also contributes to web security measures.

How Does Rate Limiting Work?

Rate-limiting tools used in applications are implemented based on different algorithm structures. These algorithms guide the functionality of the rate-limiting tool whose end goal is to limit the number of requests a server receives per time to enhance its efficiency.

Examples of Rate Limiting Algorithms

Here are some of the most popular algorithms currently in use.

Fixed Window Algorithm

This algorithm is based on fixing a static definite time interval by the server for all clients, regulating the number of requests that can be made to the server, irrespective of the number of clients accessing the API.

For example, setting a request limit of five minutes prevents any client from accessing the endpoint until the expiration of the five minutes window. This model isn’t cost-effective.

Sliding Window Algorithm

This algorithm is similar in configuration to the fixed window algorithm but it provides a solution to the fixed window algorithm by individualizing client access to a given number of requests within a specific time interval by creating independent time intervals for each client.

For example, if Client A accesses the request by 10:00, the client is allowed to make 10 requests until the expiration of the time by 10:03, while Client B who accesses the request by 10:02 is allowed to make 10 requests until the expiration by 10:05.

Leaking Bucket Algorithm

This algorithm is based on the literal meaning of its name: the leaking bucket. It ensures that only a specific number of requests can be processed by the server at any given time. Any requests exceeding this number will be discarded and issued an "error 429". This is to ensure that the server is not overloaded and to guarantee maintenance of server efficiency and speed.

Token Bucket Algorithm

This model is similar to the leaking bucket as there is a hypothetical bucket which serves as the rate limiter. This bucket serves to manage tokens and new tokens are added periodically into the bucket.

When a request is made, a token gets discarded, and this continues until all the tokens in the bucket get depleted. At that point, any request made will be discarded with an "error 429". This also helps to prevent server congestion and ensure maximal efficiency.

Rate Limiting Best Practices

Efficient web API development is mainly achievable by following the best API development practices. To maximize the use of a rate limiter as an API security measure, the following needs to be implemented.

• Firstly, choose a compatible rate-limiting algorithm. Having a strong rate-limiting algorithm in place will be essential to achieve the desired result. Choosing the best rate-limiting algorithm in sync with your API endpoint will also be needed.

• Ensure that the limit sets are within the reasonable limit ranges. Having arbitrary rate limit parameters set could negatively affect user experience and that could defeat its purpose. Setting a reasonable time limit to maximize user experience and militate against attacks has proven to be much more effective.

• Ensure efficient error handling and provide necessary feedback to the client. The default rate limiting error code is error code 429. Appropriate handling of the errors that occur during the API usage especially due to the abuse of the API will be necessary to provide the necessary feedback to the user.

• Implement flexible rate-limiting mechanisms across several parameters. Setting a fixed time interval across all endpoints seems to be a bad practice as some API endpoints are much more data-sensitive than others. Hence, having a flexible rate limiter that sets parameters in order of relevance helps to maximize server efficiency and ensure security.

• Ensure the provision of appropriate application logging, monitoring and observability tools. Having in place API metrics, logging, monitoring and observability tools also helps to serve as an additional security hack for Web APIs as they help monitor server activity, and through the use of monitoring alerts, notify the server developer when suspicious requests are detected on the server.

• Ensure synchronicity of rate limiting and other API security measures. Proper synchronicity of rate limiters with other API security hacks should be harnessed to potentiate the API security measures. Adequate knowledge of the security measures and expertise is needed in order not to counteract the existing security measures.

• Ensure proper API documentation. Adequate API documentation is also needed to ensure users, other developers and clients alike are aware of the rate-limiting practice in place to ensure compliance with the rate-limiting rules.

Conclusion

In conclusion, we have highlighted rate limiting as an important API security hack and some of it's real life use cases.

Feel free to check out my other articles here. Till next time, keep on coding!

If not addressed, this can result in a disarray of communication among services, defeating the measures placed to ensure orderly database transactions. But thank goodness, we have message brokers to the rescue.

In this article, we'll highlight the importance of message queuing as a best practice for backend development, relevant use cases and popular message queuing tools, and how to implement message queuing in backend applications.

Here are some of the prerequisites to be able to follow along in this article:

• Knowledge of Node.js

• Knowledge of microservices architecture

What is Message Queuing?

In distributed systems, several requests and queues are sent at a time. The concept of a message queue enables the storage of messages in an orderly manner, allowing for the recipients of these messages and requests to process them accordingly.

It operates asynchronously, allowing the independent functioning of different components of a distributed system. Having these in place ensures that the messages sent to the recipient eventually get attended to irrespective of a system downtime. The messages are securely stored until they get acknowledged.

Relevant Use Cases of Message Brokers

Here are some of the real-life use cases of message brokers.

• They are actively used in modern fintech applications to ensure seamless and orderly execution and processing of financial transactions made on the application. This helps to prevent server overload and transaction errors.

• Message queueing is also used in our day-to-day notification applications, ensuring early reception of sent notifications from other services. This allows the recipient to get access to those notifications notwithstanding the time they were sent or when the recipient gets access to the notification application.

• It is also used in the financial markets for seamless and efficient execution of financial orders being made. Other uses of this feature are seen in media streaming and the healthcare industry.

In the next paragraph, we'll discuss more about the tools that offer message queueing features.

Examples of Popular Message Queue Services

A wide range of applications and services offer message queueing features. Some of these services are embedded in commercial cloud infrastructure providers. Here is a list of some commonly used message queueing services:

• RabbitMQ

• Apache Kafka

• Redis

• Amazon SQS

• Google Cloud Pub/sub

• NATS

• Pulsar

• IBM MQ

We'll be utilizing a Rabbit MQ Cloud-as-a-service application to power our messages due to its popularity and ease of use. Here is a link to the documentation. You can also check out other message queuing applications provided above.

Next, we'll develop a demo project that utilizes message queuing features.

Demo Project

In this project, we'll use Rabbit MQ as a service cloud platform to build a simple message broker system that allows for seamless, ordered communication between two Node.js servers.

In this tutorial, we'll create a message publisher that will serve as the sender, and a message consumer that receives the messages.

To begin with, we'll have to create both Node.js servers that will be communicating with each other.

You can create two different files and initialize a Node project using npm init.

Thereafter, you can install relevant packages that will aid in the implementation of the features. We'll use the amqplib library, a Node library implementation for Rabbit MQ.

This package allows us to swiftly communicate with RabbitMQ via the Node.js application. It seamlessly achieves this due to its built-in functions for creating queues, publishing messages, and consuming messages. More details regarding its usage will be discussed later.

To install this in our project, kindly execute:

npm i amqplib

The publish function will now be drafted. After that, we'll have to initialize amqplib in our project.

const amp = require(“amqplib”)

Also, we need to set up our RabbitMQ broker which will manage our messages.

There are several ways of creating RabbitMQ servers, the most popular being installing them on a home computer and then setting them up to interact with the backend servers. You can download the software here. However, for ease of usage, we will be utilizing a cloud-based RabbitMQ broker as a service application to generate our server.

To get this done, kindly navigate to https://www.cloudamqp.com/ and create an account. For this tutorial, an instance was created and configured to the closest region to me. On successful creation of the instance, the details of the Rabbit MQ will be made available.

Moving on, we will be creating a message queue in which both parties can use as a connection pipeline. We will begin by creating a function to send messages.

async function sendMessage(msg) {
  try {
    const connection = await amqp.connect(url);
    const channel = await connection.createChannel();

    await channel.assertQueue(queue);
    await channel.sendToQueue(queue, Buffer.from(msg));

In the code above, a connection was ensured and maintained. Thereafter, a communication channel was also created. The assert queue function is then declared when executed, ensuring that the existing queue is maintained, and creates the queue if it doesn't exist.

The message attached to the function gets buffered and then sent to the queue created.

async function receiveMessage() {
  try {
    const connection = await amqp.connect(url);
    const channel = await connection.createChannel();

    await channel.assertQueue(queue);
    await channel.consume(queue, (msg) => {
      console.log(`Received message: ${msg.content.toString()}`);
      channel.ack(msg);
    });

The receiver function also gets executed to receive any message that gets into the queue by executing the consume method at the exact queue. In our case, the message is outputted as a log message.

The ack function is now executed to acknowledge the message received from the queue.

Here is the full project code:

const amqp = require("amqplib");
const url = "amqp://localhost"; // Replace with your RabbitMQ server URL
const queue = "queue";

async function receiveMessage() {
  try {
    const connection = await amqp.connect(url);
    const channel = await connection.createChannel();

    await channel.assertQueue(queue);
    await channel.consume(queue, (msg) => {
      if (msg !== null) {
        console.log(`Received message: ${msg.content.toString()}`);
        channel.ack(msg);
      }
    });
  } catch (err) {
    console.error("Failed to receive message:", err);
  }
}

receiveMessage();

Here's the message publisher application code:

const amqp = require("amqplib");
const url = "amqp://localhost"; // Replace with your RabbitMQ server URL
const queue = "queue";

async function sendMessage(msg) {
  try {
    const connection = await amqp.connect(url);
    const channel = await connection.createChannel();

    await channel.assertQueue(queue);
    await channel.sendToQueue(queue, Buffer.from(msg));

    console.log(`Message sent to ${queue}: ${msg}`);

    await channel.close();
    await connection.close();
  } catch (err) {
    console.error("Failed to send message:", err);
  }
}

sendMessage("Hello, world!");

Here is the output of the code:

Additional info

So far, we have completed this tutorial on message queueing and its role in facilitating seamless communication across various systems. To further improve your skillset, here are some additional best practices that should be implemented when building complex services:

• Rate limiting

• Load balancing

• Application monitoring and logging

• Continuous integration and deployment

Summary

We've highlighted the importance of message brokers and how to implement message queueing in a backend application.

Feel free to check out my other articles here. Till next time, keep on coding!

Modern-day computing services also provide a wide range of features which make web apps easier to use and more efficient. So it's important for developers to have a basic understanding of how the cloud works.

This article is a beginner's guide to deploying backend applications to the cloud. We'll use the Azure platform as our cloud infrastructure and Node.js/Express for the backend web application. Before we go on, here are a few requirements:

• Basic understanding of cloud computing (you can check out this article to learn more about that).
• Knowledge of JavaScript.
• VS Code.

With that, let's get started.

Introduction to Azure

Azure is a cloud computing platform developed by Microsoft that acts as a server for deploying and hosting web applications, databases, file storage, and so on.

Compared to other cloud computing services, it is quite beginner-friendly and has an actively growing user base. Let’s explore the Azure portal.

How to Set Up Azure Account

Signing up on the Azure platform is the first step to hosting your application. First, navigate to the website and complete the signup process.

Azure sign in page

After signing up, you'll have access to the management console of the Azure application where all our activities can be carried out.

Before we go on, here are some of the services we'll be getting familiar with on this platform.

• Azure Resource groups
• Azure app services
• Storage accounts
• SQL databases
• Virtual networks

Azure services

Congratulations on successfully creating your Azure account.

Deployment Options on Azure

As a cloud computing platform, Azure boasts of its wide versatility. Depending on your skill level or preference, you can deploy web applications to Azure via the following options:

• Azure CLI
• Azure Virtual machines
• Azure Functions
• Azure Kubernetes Service
• Azure Storage.
• Azure DevOps
• Azure portal service

We'll utilize the Azure portal service for this tutorial and use its VS Code integration to deploy a simple Node.js application to the Azure cloud.

How to Set Up the Backend Application

We'll create our web application using Node.js via the command line and Visual Studio Code.

Firstly, navigate to the folder where your app will be created and initialize a Node project by executing npm init

Next, initialize the app by installing the Express framework. This can be done via npm i express.

Go on and paste the sample code for this tutorial:

const express = require("express");
const app = express();

app.use(express.json());

app.get("/", (req, res) => {
    res.send("Hello, World");
});

app.listen(process.env.PORT || 5000, () => {
    console.log("Server is running on port " + (process.env.PORT || 5000));
});

The code above outputs Hello World whenever it is executed.

Application Deployment

The backend code we wrote in the preceding paragraph will be deployed to Azure via the use of Azure’s VS Code extension.

VS Code homepage

Navigate to the Extensions tab, search for Azure App Services and install the extension. On successful installation, an Azure widget will appear on your taskbar where you can sign in to the Azure cloud.

Extensions marketplace

Subsequently, we'll be creating a cloud-based web application in which our Node.js code will later be deployed.

On the Azure resources tab, clicking the plus icon will display a drop-down menu where various development options can be seen. We'll be clicking on the Azure app services option.

Azure drop down menu

After clicking that, a prompt will pop up asking for a unique name for the cloud application. In my case, I went with newApp777.

creating a web application

However, you can use any other name you so wish. Subsequently, you will need to select the backend language of your choice. Any version of Node.js will be compatible with our application.

web stacks available

Also, the F1 service option will be used for this tutorial. However, you can pick anyone you so desire.

Various Azure tiers

On successful completion, your application will be created on the Azure portal.

Now to the crux of the matter. Let's install our Node.js code on this web application.

We'll click on the code folder which provides us with options to automatically deploy our code to an Azure web app service.

Deployment dropdown menu

As soon as this is done, the list of the cloud servers on your Azure account will be shown. You can then select the new app we just created.

app deployment

Your backend code should then be deployed on the NewApp cloud server we created. On successful completion, you'll receive a success message with a link to your cloud application.

command prompt interface

Congratulations, you have successfully hosted your first web application. Navigate here to see the hosted application.

The application webpage

Additional Information

So far, we have covered the basics of deploying an application via the use of VS code extensions on Azure portal services. As you progress in the field of cloud computing, other interesting fields can also be explored such as:

• App monitoring with Azure Monitor.
• Azure app networking essentials.
• Azure MySQL database integration.
• Node JS serverless functions deployment.

You can also interact with me on my blog and check out my other articles here. Till next time, keep on coding!

In this article, you'll learn how to optimize embedded links in your front-end development projects to meet your desired specifications.

The inspiration for this article came from a problem I encountered while creating a portfolio site and having issues embedding external sites—until I discovered an efficient tool to help.

To fully grasp this tutorial, there are some of the prerequisites:

• A good knowledge of JavaScript.
• Familiarity with React JS and npm package installation.
• Proficiency in CSS.

Let's get started!

The Concept of Micro-linking

Micro-linking entails embedding a link to an external website within a default web page. Embedding links are different from the default HTML “alt href” attributes commonly used by web developers.

This concept also entails extracting relevant site metadata, information and relevant images from the embedded external site link. This contributes to providing more knowledge about the external website in question.

This feature isn’t all new as it's commonly used among high-end blog posts, video streaming sites and e-commerce sites. This feature is also utilized by site owners to customize how their site information is viewed when embedded in advertisement sites.

How Does Micro-linking Work?

Ordinarily, embedding a URL using plain HTML code doesn’t generate any image or text preview. However, this isn’t the case for micro-links. This occurs due to the use of an application programming interface to extract relevant info from the site being referenced and then outputting it to the developer in JSON format for easy customization and usage.

There are quite a lot of micro-linking services available commercially. Some examples are Embedly, Open Graph protocol, microlink.io, and so on.

Introduction to the Microlink Package

In this tutorial, we'll create a site that uses a Node package known as Microlink to generate previews. Details regarding URL customization will also be illustrated.

This is customization allows developers to easily extract relevant link previews from the Microlink site, skipping the hassle of interacting with the website interface.

For further learning, the library has a rich documentation site which can be accessed here.

How to Set Up Your Project

To build the link preview site, we'll utilize the React Vite tool. Entering the command npm create-vite-app@latest links immediately spins up a folder named latest links which we'll use in building this project.

Also, the microlink package needs to be installed. To do that, navigate to the command line and run the npm i @microlinks/react command.

Demo Project

First of all, we'll be create the default JSX functional component for this project.

function App() {
return (
<>
</>
)
}
export default App

Next, we'll be import and initialize the installed microlink library:

import Microlink from '@microlink/react'

After that, we can initialize the package within the App function:

return (
<>
<Microlink
url= " https://tobilyn77.hashnode.dev/nodejs-clustering-and-load-balancing-comprehensive-overview "  />
</>
)

The URL variable I included is a link to an external site I intend to preview on my local site. This can be changed to whichever site you would like to preview.

On running this, you should see something similar to this.

microlink project webpage

We have successfully set up a micro-link in our application. But that’s not all, the power of the package is enormous, providing a level of flexibility in controlling the layout and size of the micro-links.

Size Tweaking

Size tweaking simply involves adjusting the size or dimensions of a variable to suit different specifications. In this case, we'll adjust the size of the embedded links.

<Microlink
url= "https://tobilyn77.hashnode.dev/nodejs-clustering-and-load-balancing-comprehensive-overview" />
<p></p>
<Microlink
url= "https://tobilyn77.hashnode.dev/nodejs-clustering-and-load-balancing-comprehensive-overview"  size="small"/>
<p></p>

micro-links with different sizes

The code above specifies the size of the micro-link to be rendered. If the size is indicated to be small, the dimension of the micro-link gets altered. It also allows for large and medium sizes, and by default, the medium size is what is usually seen when no size dimension is specified.

Also, the microlink self-enclosing tag allows for CSS styling. This can be achieved by including the style variable within the microlink tag.

<Microlink
url= "https://tobilyn77.hashnode.dev/nodejs-clustering-and-load-balancing-comprehensive-overview"
style= {{color: 'red'}} />

micro-link with the color set to red

As you can see in the code above, the microlink was styled with the color red and that reflected in the output. Other CSS styling can also be applied to give users an aesthetic feel to your site.

Conclusion

With this, we have come to the end of the tutorial. We hope you’ve learned link previewing via the microlink package.

Feel free to drop any questions or comments in the comment box below. You can also interact with me on my blog and check out my other articles here. Till next time, keep on coding!

Its introduction has helped to solve problems associated with the age-long monolithic software development approach, overcoming its cons and achieving scalability.

In this article, I hope to dive deep into what microservices entail and highlight its significant use cases. Also, I will dive deep into the SOLID principles and other best practices necessary to build efficient microservices. With that, let's get started.

First of all, what is a microservice?

What is a Microservice?

This is a type of system architecture in which an application gets structured as a confluence of several independent, loosely coupled independent services. This ensures that each aspect of the overall application gets managed individually and still functions irrespective of the current state of other independent services. These independent servers still allow for information sharing over a given network.

It actively mirrors the distributed system model which segregates the various computers on a network and shares resources amongst them. The adoption of this model by big enterprises has been seen to be advantageous as it has greatly reduced server downtime, minimized costs and maintained efficiency.

The microservice system architecture also provides an upper hand to these firms as it rapidly provides scaling opportunities in case of a spike in user visits. The scaling could either be horizontal which involves activating multiple servers to handle user requests or vertical which involves increasing the CPU power of the server to efficiently handle user requests.

Unlike conventional monolithic systems, microservice best practices deviate slightly from the conventional ACID principles designed for related databases. Hence it's important to learn about best practices and principles which serve as a basis for building resilient microservices.

This will take us into the world of the SOLID principles. The solid principles form the general basis of object-oriented programming and design but have been adapted in the context of building resilient microservices. But what does SOLID represent?

• Single Responsibility Principle
• Open-Close Principle
• Liskov Substitution Principle
• Interface Segregation Principle
• Dependency Inversion Principle

Let's discuss these in detail.

Single Responsibility Principle

This principle states that each service in the grand microservice architecture is responsible for a single functionality or possesses a single reason to change. This implies that the service in question is solely and wholly built to fulfil a specific application functionality and it is done cohesively.

This feature provides it with the liberty to scale bigger to effectively deliver on that given functionality. This forms the baseline for microservices development as it reduces the interference of several services due to service interdependency which is a side effect of monolithic applications.

Open-Close Principle

This principle was initially applied for object-oriented programming but is now also adapted for microservices development. It entails that services created within the overall microservice architecture are open to extension with additional service functionalities and communication via the services interface but should be closed to code modification.

This principle is necessary as code modification affects service functionality and stability and also serves as a risk for introducing bugs to the existing code which can ultimately cause errors in system function.

To ensure this, features such as code versioning allow for newer versions of an existing service to be created and deployed without affecting the older versions' functionality and maintain the system's efficiency. Also ensuring the implementation of APIs on each service and the concept of dependency inversion (which will be discussed in the subsequent section) helps to achieve this principle.

Liskov Substitution Principle

This principle is named after its originator, Barbara Liskov. It means that services built within complex microservice architecture can and should be easily substituted or replaced with no or minimal side effects to the entire microservice architecture. This feature also enables developers to build modular microservices applications.

It also enables the execution of the dependency inversion principle which will be discussed in subsequent paragraphs. Achieving this principle involves structuring the microservice architecture with the use of interfaces and classes which allows for the reuse and light coupling of services.

Interface Segregation Principle (ISP)

This principle builds on the Liskov substitution principle and it simply advocates for ensuring that interfaces used for each service are specific for the users who interact with them solely. This ensures that the interface delivers the specific function intended by the service created. This would in turn minimize service interdependency among various services and ensure service application autonomy, enabling it to achieve the desired scalability and overall efficiency possible.

This, alongside the Liskov substitution principle, allows for seamless microservice application evolution over a given cycle. To achieve this, it is important to ensure a minimal reliance of the service on external dependencies and also declare explicit and distinct functions for each service.

Dependency Inversion Principle

This principle negates the age-long tradition in which high-level modules and services tend to depend on smaller low-level services to achieve the necessary efficiency and correctly perform their designated function. It now implies that the high-level services/modules should not depend on anything from the low-level services and both should only interact based on the existing abstraction. In our case, this implies that the interfaces already discussed earlier.

This principle, in line with the other principles, permits easy scaling of each service or module in question and also allows for service reuse or substitution whenever such is needed. This principle has also revolutionized the way applications get built as they now delineate the functions and autonomy of each service in the application.

Additional Information

So far, we have shed light on the SOLID microservices design principles. However, other additional tips which could be of great help when building microservices include:

Availability Over Consistency Principle

This principle is based on the CAP theorem (consistency, availability and partition tolerance. Hypothetically, a system should have all these components implemented and fully functional but in reality, this isn’t so. Ensuring these works often results in network lags, which affects system efficiency, resulting in the need for tradeoffs among these components.

In the case of microservices development, the need for a service to be continually consistent in providing an updated response to a request gets overridden by the need for the service to be available with minimal downtime. Eventually, the overall microservice consistency is achieved during a given period via conflict resolution techniques and other consensus protocols.

Easy Deployability Principle

Unlike conventional monolithic applications, deploying microservices is a bit complex as it requires ensuring seamless communication across various deployments. However, this can be achieved by mastery of some techniques.

Firstly, it's important to possess knowledge of containerization and containerization tools such as Docker. Additionally, knowledge of orchestration tools like Kubernetes is an additional advantage. Adequate knowledge of Infrastructure as Code tools such as Terraform also helps as it gives the developers great control over the application and allows for easy versioning. Provision of monitoring and observability tools to help detect any anomaly in the operations.

Conclusion

With this, we have come to the end of the tutorial. We hope you’ve learned essentially about the principles and other best practices to have in mind when building microservices and other distributed applications.

Feel free to drop comments and questions in the box below, and also check out my other articles here. Till next time, keep on coding!

Database management is a critical skill a developer must possess in building scalable applications that have a high level of efficiency. If not handled properly, it can result in data loss and mismanagement on the part of the database developer.

Hence, databases must be structured and built with the users in mind and built utilizing the best practices available.

This article aims to highlight general principles of database best practices and also explain each peculiarity. But before we discuss that in detail, let’s review what database transactions are all about.

What are Database Transactions?

Database transactions are simply groups of operations which can be termed as a unit of a work process performed on a database within a database management system.

It encompasses basic operations such as CRUD operations to more advanced operations such as database indexing, caching, and normalization.

With so many users performing many transactions at the same time, it’s important to ensure that the database is concurrency-enabled to prevent data interference between two or more users accessing the same resource.

Hence, there is the need for the ACID principle. What then does ACID represent?

• Atomicity
• Consistency
• Isolation
• Durability

Subsequently, we will be discussing each point in detail. First on our list is atomicity.

What is the Database Atomicity Principle?

What does database atomicity entail? The atomicity of a database simply means that a database operation can’t be broken down further as a unit. This means that the database operation or transactions gets executed completely, and in case any error comes up during the execution process, the entire operation gets completely cancelled, preventing room for partial operation execution.

If the database isn’t atomic, this can result in the provision of misleading incomplete data and ultimately result in entire system chaos. How does the database ensure atomicity? It does this by creating a copy of the existing database before the operation gets executed and then initiates a crash recovery and backup restoration operation in the event of an operation failure.

It is also important to note that other database principles such as consistency and durability rely on the need for the database to be atomic to be truly fulfilled.

Having discussed this, let’s move on to the database consistency principle.

What is the Database Consistency Principle?

This principle entails that the database has certain constraints, cascades, triggers and other requirements in place, which needs to be fulfilled while making changes to an established database. Failure to fulfill this requirement will lead to consistency errors, returning the database to its previous stable state.

Also, consistency as a principle ensures that the data updated by a user is made available as the latest version of the data in the database to all users who desire to read the database. Having this in place eliminates the occurrence of inconsistencies and aids faster information retrieval.

Understanding what it means for a database to be consistent involves ensuring the operation performed on the database passes the integrity check before being successfully executed. Having exhausted this in detail, let's discuss the database isolation principle.

What is the Database Isolation Principle?

Why should we isolate a database and how does one make a database operation independent from other database operations?

Isolation is necessary in a database management system to ensure that the user's access to information on the database is not interfered with by other concurrent transactions undertaken by other users on the database. To enforce this, the use of isolation levels in each database operation helps to preserve information integrity.

To effectively guarantee the database integrity, specific database isolation levels must be used. Here are some of the isolation levels ranked in order of hierarchy:

• Read uncommitted
• Read committed
• Repeatable read
• Serializability

Read Uncommitted Isolation Level

The read uncommitted database isolation level allows other users to have access to read current database transactions which has not yet been completely or successfully executed. It allows access to read what is being referred to as dirty read, which is one of the data inconsistencies that can be seen. This level of data isolation isn’t advised.

Read Committed Isolation Level

This database isolation level disallows other users to read or have access to a database transaction that has not yet been committed. Hence it prevents other users from seeing, updating or overwriting it until it has been completely executed.

Repeatable Read Isolation Level

This isolation level exclusively isolates a transaction from other transactions occurring concurrently, preventing other users access to read and update the transactions.

Serializability Isolation Level

This is the highest level of data isolation and is referred to as the strictest level. It isolates the multiple transactions performed concurrently and executes them efficiently as they are executed serially. It also prevents database inconsistencies.

Without these levels in place, inconsistent database mishaps such as dirty reads, non-repeatable reads, phantom reads and many others may be experienced. With this, let's move on to the last point about database durability and discuss it in detail.

What is the Database Durability Principle?

What does it imply when we describe a database as durable and how do we ensure the durability of a database? Durability as it sounds is a principle which ensures that databases have a high level of immortality.

Irrespective of any adverse outcomes that the database management system might face such as outages and crashes, there shouldn't be any loss of database information.

How do databases try to achieve this? The database creates a transactional log that contains the recorded data before any new operation gets executed. In the event of any of these adverse events, the transaction log serves as the backup store, ensuring that the database info is well preserved up to the point before the operation occurred, thereby mitigating against data breaches and loss.

We'll also highlight other helpful database operations best practices that can also be implemented.

Other Database Operations Best Practices

The BASE principle, which is more suited for NoSQL databases such as MongoDB, Redis, and Cassandra, and so on. It entails a database to be:

• Basically available
• Existing in a soft state
• And be eventually consistent.

Basically Available

This entails that the database prioritizes the availability of the database operations over consistency and concurrency. This is quite applicable to distributed systems which rely on a high level of efficiency to function effectively.

Soft State

This ensures easy flexibility of the database, allowing for size scaling, operations and increased concurrency for optimal database performance at all times. This allows the data to maintain resiliency.

Eventually Consistent

This entails that, irrespective of how the transactions get executed in the sequences, it eventually achieves efficient consistency. This is achieved by conflict resolution and reconciliation. This eventually contributes to the building of a resilient data system.

Conclusion

With this, we have come to the end of the tutorial. We hope you’ve learned essentially about optimizing database operations and their efficiency using the ACID principle and other best practices available.

Feel free to drop comments and questions in the box below, and also check out my other articles here. Till next time, keep on coding!

The act of application security is a shared responsibility amongst the client and server developers and negligence of one’s role can be disastrous. Statistics show that data breaches in 2023 resulted in exposure of over 8 million data records worldwide.

In this article, I'll be highlighting key areas of API security, which involves data validation. This concept is quite crucial in helping you protect your API from web attacks via malicious user data. This tutorial is well-suited for all backend developers regardless of years of experience.

To be able to follow this tutorial, here are some prerequisites:

• Knowledge of Node.js
• Knowledge of npm and package installation

With that in place, let’s get started.

How Does Data Validation Work?

First of all, what is data validation? Data validation simply entails ensuring the accuracy and reliability of the data received from external sources before onward data processing.

It is a key component of web API security as it is essential for preventing the occurrence of web injection attacks, SQL attacks and NoSQL attacks. To know more about these, you can check this link.

Note that data validation is needed but not limited to the following backend operations.:

• User login and sign up
• Response query
• Updating server databases

All these can be used as avenues by mischievous black hat hackers to gain access to the server database and obtain sensitive user details or even wreak havoc by formatting the entire database.

Popular Data Validation Tools

So far, there are lot of tools that can help the programmer achieve efficient data validation in API development.

They help you avoid reinventing the wheel of using long regex code to validate data. They provide a whole lot of features, including error handling and validation customization functionalities.

Some of these tools include:
• Joi
• Zod
• Yup
• AJv
• Valibot
• Validator.js
• Superstruct

To further shed light on these tools, we'll compare some of the most popular data validation tools mentioned above.

Pros and Cons of Data Validation Tools

To further enlighten you about these JavaScript validation tools, I will be highlighting some pros and cons of three of these popular JavaScript validation tools.

Joi

Pros

• It has a strong, large user community and development support
• It has built-in capabilities to handle complex validations

Cons

• It’s syntax is quite verbose

Zod

Pros

• It is easily compatible with Typescript projects
• It has efficient error-handling capabilities

Cons

• Async validation isn’t supported.

Yup

Pros

• It mainly uses declarative syntax to set its validation tool which confers its simplicity
• It has a comparable fast performance.

Cons

• It doesn’t provide customization features
• It has limited ability to handle complex validations

For the purpose of this tutorial, we'll use Joi as our data validation tool.

Introduction to Joi

Joi is a simple and efficient JavaScript-based data validation tool that is based on the schema-type configuration.

It has built-in capabilities for validating the occurrence of data in various forms, but not limited to Booleans, strings, functions and intervals. It can also handle complex validation operations.

Additionally, it provides minimal caching functionalities. More information about the tool can be found here.

How to Set Up Joi

In this section, we'll set up Joi in our local environment. To install Joi, navigate to the code folder via the command line and run this:

npm i joi

A message confirming successful installation should be displayed. With that completed, we can demonstrate the power of Joi in validating user registration in our demo API.

Demo Project

In this project, you'll use Joi to validate the input received from the client with the intent to sign up on the server. The default code for the user sign-up function for the Node.js application can be found here.

Go on and import the installed Joi package into your code:

const Joi = require("joi");

Before writing our signup controller, we'll initialize the Joi library within the code file:

const SignUpSchema = Joi.object({});

In this project, we'll validate the email, password and username parameters received from the client.

const SignUpSchema = Joi.object({
    email: Joi.string().email({
        minDomainSegments: 2,
        tlds: {
            allow: ['com', 'net']
        }
    }),
    username: Joi.string().alphanum().min(3).max(15).required(),
    password: Joi.string().min(8).required()
});

The email parameter object ensures that the email address is a string, and the domain site is limited to .com and .net, disallowing other forms of domains.

For the username parameter, it ensures that it is a string containing both letters and numbers with a minimum character count of 3 and a maximum character count of 15. The required function ensures that these conditions must be met or the entire request won't be validated.

The password parameter ensures that the password supplied is in a string format with a minimum character count of 8, and it is also required.

To apply it to our endpoints, we include this within the controller function:

const { error, value } = SignUpSchema.validate(req.body, { abortEarly: false });
if (error) {
    res.status(400).json(error.details);
    return;
}

This function gets executed before inserting the user details into the database. The schema tries to validate the received input and then proceeds to the database if successfully validated.

The abortEarly feature is included to allow for all parameters to be assessed. All the errors will be displayed if there is any.

The above can also be replicated in the Login controller function. You can also check out the documentation for other examples of complex validation options using Joi.

The final code for the project is displayed below:

const jwt = require("jsonwebtoken");
const userSchema = require("../Schema/User");
const Joi = require("joi");
const bcrypt = require("bcrypt");
const { createNewColumn, checkRecordsExists, insertRecord } = require('../utils/sqlSchemaFunction');

const generateAccessToken = (use) => {
    return jwt.sign({ userID: use }, process.env.JWT, { expiresIn: "1d" });
}

const SignUpSchema = Joi.object({
    email: Joi.string().email({ minDomainSegments: 2, tlds: { allow: ['com', 'net'] } }),
    username: Joi.string().alphanum().min(3).max(15).required(),
    password: Joi.string().min(8).required()
});

const loginSchema = Joi.object({
    email: Joi.string().email({ minDomainSegments: 2, tlds: { allow: ['com', 'net'] } }),
    password: Joi.string().min(8).required()
});

const register = async (req, res) => {
    const email = req.body.email;
    const password = req.body.password;

    if (!email || !password) {
        res.status(400).json("Please supply the email or password");
        return; 
    }

    const { error, value } = SignUpSchema.validate(req.body, { abortEarly: false });
    if (error) {
        res.status(400).json(error.details);
        return;
    }

    const salt = await bcrypt.genSalt(10);
    const hashedPassword = await bcrypt.hash(password, salt);
    const user = {
        username: req.body.username,
        email: email,
        password: hashedPassword
    };

    try {
        const userAlreadyExists = await checkRecordsExists("users", "email", email);
        if (userAlreadyExists) {
            res.status(400).json("Email must be unique");
        } else {
            await insertRecord("users", user);
            res.status(200).json("User created successfully");
        }
    } catch (err) {
        res.status(500).json({ err: err.message });
    }
};

module.exports = { register };

API testing in Postman

Ensuring that the code followed our defined schema resulted in it being successfully executed.

Conclusion

With this, we have come to the end of the tutorial. I hope you’ve learned about data validation, various data validation tools and data validation best practices.

You can also reach out to me and check out my other articles here. Till next time, keep on coding!

So far, web development has become much more advanced, and data migration and backup is an essential skill every web developer should have to help ensure continuity of data and preserve user information regardless of any circumstances.

For developers like me who prefer to test all their code features locally in a development environment before final deployment to the cloud, this tutorial would come in handy for you when faced with the challenge of migrating your locally stored MongoDB data to a cloud platform or any other platform.

The inspiration for this article is a result of the need for me to migrate the total JSON data on my local MongoDB Compass server to the cloud (MongoDB Atlas) for an ongoing project. I waded through the endless internet resources and documentation but couldn’t get a suitable, straightforward fix. With multiple trials and errors, I was able to succeed with the migration. This article should serve as the definitive guide you need to save you time and achieve fast results.

In this article, we'll delve into data management in MongoDB, data backup, and efficient data migration tools that ease up the entire process. This article is suited for people with intermediate to advanced knowledge of MongoDB and backend development.

If you have any difficulty with the terminologies used in this article, I would suggest studying the MongoDB documentation here.

Let’s begin.

Introduction to MongoDB

MongoDB is a non-relational NoSQL-based database that stores data via the document model in a JSON format. It’s quite popular, and ranks top among the most used databases worldwide.

It's also more code-friendly as it's currently the default database used by many JavaScript-based full stack and backend developers.

It has various database options, comprising both local database servers and a cloud-based Database-As-A-Platform.

A good example of this is the MongoDB Atlas. MongoDB Atlas is a flexible and scalable MongoDB implementation with strong cloud security.

It provides indexing, load balancing, and sharding, among other features. More information regarding it can be found here.

How to Initialize the MongoDB Server

You should have the MongoDB local server already installed. However, if you haven’t installed it, it can be downloaded here.

The MongoDB server can then be easily executable by adding its path to the environmental variables on Windows.

To initialize the MongoDB application, activate the command prompt and type in
Mongod

MongoDb server running

To explore the databases on the MongoDB server, open the MongoDB command shell and type show dbs

The databses on the MongoDB server

This displays all the databases on the MongoDB server.

With this, let's go on to migrate one of these databases to the cloud.

How to Back Up Data

To efficiently upload the database collections to the cloud, they must first be backed up.

To back up your database, you'll have to install an additional tool. Navigate back to the MongoDB downloads site and download the MongoDB database administration tools.

This package contains a lot of database tools, such as MongoExport, MongoImport, MongoDump, and MongoRestore.

The above settings are applicable for users who have a version of MongoDB above version 4.4. MongoDB decided to release it as a stand-alone tool. For users whose version is less than version 4.4, these tools can be pre-installed in the MongoDB package.

Now, let's go on to the two major tools that would help facilitate the migration process: MongoDump and MongoRestore.

When MongoDump is executed on a database, it helps to create a backup database file in a binary-encoded JSON format (BSON). While the MongoRestore helps to return the backed-up database to MongoDB for usage.

Moving on, ensure that their package path is included in the environmental variables so as to guarantee efficient execution.

In order to back up a local database, open the MongoDB shell and then enter this command:

Mongodump --db={TheNameOFYourDB} --collection={TheNameOfYourCollection} –out={The name of the folder the backed up JSON files would be located}

With this execution, you have successfully backed up your local MongoDB database.

How to Set Up a MongoDB Atlas

MongoDB Atlas is MongoDB cloud Database-As-A-Service cloud option accessible to its users at all times with little to no disruption.

Also, for most production apps that utilize the power of MongoDB on the server as a database tool, the easiest and most convenient provider of this solution is the MongoDB Atlas.

Hence, it is our choice to back up our MongoDB data to the cloud. I would assume you already have a MongoDB Atlas account, but you can still create one and then create a database that would serve as the recipient of the local database.

To set the account up, kindly navigate to the Atlas home page here. Create an account and you'll be directed to a dashboard where you can create new databases.

MongoDb Atlas home page

You can, however, name it with any name you want. To complete the upload of the local MongoDB data to the cloud, navigate to the settings tab on the MongoDB Atlas database and click on the migrate data option. With this successfully done, let's now go on to complete the migration process.

How to Migrate Your Data to the Cloud

To migrate your backed-up database to the cloud, MongoImport and MongoExport will be invoked.

In the command prompt, paste the migration code copied from the Atlas, and then execute it on the command prompt. With that, you should see a success text.

You can then check and refresh your cloud database to see the new files that have been uploaded there. Thereafter, you can connect the new cloud database to your backend application and run it successfully to get the same results on your local computer.

Conclusion

With this, we have come to the end of the tutorial. We hope you’ve learned about data management and migration in MongoDB, tools involved and all its intricacies.

Feel free to drop comments and questions, and also check out my other articles here. Till next time, keep on coding!