In this article, I will use a simple event management app as an example to show you how to build an application using Node.js, Express.js, and MongoDB.

By the end, you’ll know how to set up a Node.js project, create a server with Express.js, show dynamic pages with embedded JavaScript, and connect to a MongoDB database to handle your data.

What You'll Learn

• Setting up a Node.js project

• Creating a server with Express.js

• Rendering dynamic pages with ejs

• Connecting to a MongoDB database

• Creating models and schemas for your data

• Handling HTTP requests and responses

Table of Contents

• Set Up Your Development Environment

• Set Up the Server

• Install and Set Up Express.js

• Create a Dynamic Template

• Save Your Data to MongoDB

• Connect to the Database

• Create the Model for the Document Structure

• Create HTML Pages

• Create Partials

• Create an Environment Variable File

• Further Steps

• Conclusion

Let's get started 🚀

Prerequisites

• Node.js installed on your system.

• A good understanding of MongoDB.

• A code editor that you prefer, such as Visual Studio Code or Sublime Text.

Step 1: Set Up Your Development Environment

Install Node.js and npm

First, you'll need to download and install Node.js from nodejs.org. Then you can verify the installation by running: node -v and npm -v.

Initialize a New Project

Create a new directory for your project. Then initialize the project with npm: npm init -y in your terminal.

mkdir event-app
cd event-app
npm init -y

Initializing the project

Running npm init -y creates the package.json file, as shown above. This file is crucial. It stores and tracks all third-party libraries (dependencies) needed for your application.

Step 2: Set Up the Server

To set up your server, create a file called server.js or app.js. These are common names. They are used for their descriptive nature. But, you can name the file whatever you prefer.

The server.js file will be used to create a server which will be used to manage, control, and route to the necessary page in our application.

Step 3: Install and Set Up Express.js

Express.js is a popular web application framework for Node.js and a third-party library that we use in our application.

Express simplifies the handling and definition of various routes for HTTP requests. It enables you to manage the application's routing and connect it to the server.

To use Express:

Install Express.js by running the following command in your terminal:

npm install express

Require Express in your server.js file.

const express = require('express')

Initialize Express so you can use it in your application.

const app = express()

Create a routing path to get the HTTP request.

//routing path
app.get('/', (req, res) => {
  res.send('Hello World!');
});

Lastly, we need to ensure that the connection to the server is set up correctly. When we start the server in the terminal, it will open in the browser.

To do this, use the listen() method.

// Start the server
app.listen(3000, () => {
  console.log('Server started on port 3000');
});

This method will listen() to requests from the server.

Here's the complete code process:

const express = require('express');


// Next initialize the application
const app = express();

// routing path
app.get('/', (req, res) => {
  res.send('Hello World!');
});

// Start the server
app.listen(3000, () => {
  console.log('Server started on port 3000');
});

Note: The routing path above was only for testing purposes to confirm that the server is working and connected. We will provide a different file for the event app we are creating.

With Express.js installed in your application, you can now create a server that will handle all your routing and connections.

To start the server, go to your terminal.

Use the keyword node, then type --watch, a flag to start and automatically restart the server whenever you make changes:

node --watch server.js

Or you can install [nodemon](https://www.npmjs.com/package/nodemon) for the same purpose. Nodemon detects changes in the directory and restarts your application.

npm install -g nodemon

Then run your server with:

nodemon server.js

Step 4: Create a Dynamic Template

We need a templating engine to render HTML code in the browser using Node.js. We'll use ejs (Embedded JavaScript) for this tutorial but there are others such as Pug (formerly known as Jade) and Express Handlebar, which also render HTML on the server.

ejs allows you to embed JavaScript in HTML to create dynamic web pages.

To install ejs, run:

npm install ejs

To set up ejs in server.js, require and set ejs as the templating engine:

Requiring ejs template in your application

const express = require('express');
const app = express();

app.set('view engine', 'ejs');

With this setup, you can now enable dynamic rendering of HTML code in your Node.js application.

Step 5: Save Your Data to MongoDB

To save the data you create for your application, you will use MongoDB.

MongoDB is a "Not Only SQL" (NoSQL) database that's designed for storing document collections. Traditional SQL databases organize data into tables, but MongoDB is optimised for handling large volumes of data.

To read more about this, check out this article.

Step 6: Connect to the Database

Now we need to connect to the database which will be MongoDB for this tutorial.

Using MongoDB provides you with a Uniform Resource Locator (URL) to connect to your application. This URL connect you and acts as a communicator between the database and your application.

How to get the URL

To get the URL, follow these simple steps:

1. Sign Up/Log In: Go to the MongoDB website and sign up for an account or log in if you already have one.

2. Create a Cluster: Once logged in, create a new cluster. This will set up your database.

3. Connect to Your Cluster: After your cluster is created, click the "Connect" button.

4. Choose a Connection Method: Select "Connect your application".

5. Copy the Connection String: MongoDB will provide a connection string (URL) like this:

mongodb+srv://<username>:<password>@cluster0.mongodb.net/<dbname>?retryWrites=true&w=majority

6. Replace the Placeholders: Replace <username>, <password>, and <dbname> with your actual username, password, and database name.

Now that you have the URL, you can easily connect to your database.

To make this connection easier, we will use a tool called Mongoose.

What is Mongoose?

Mongoose is a JavaScript library that makes it easier to work with MongoDB in a Node.js environment. It provides a simple way to model your data. You can define schemas, do data validation, and build queries as well.

How to make a connection

MongoDB has already provided you with a URL for connection. Now you'll use Mongoose to send your documents to the database.

To use Mongoose in your project, follow these steps:

Install Mongoose using npm.

npm i mongoose

In your server.js file, you need to require Mongoose to use it as a connector to the database.

const mongoose = require('mongoose');

After you require Mongoose, you need to define the connection URl provided in your server.js file.

server.js:

const mongoose = require('mongoose');

// Replace <username>, <password>, and <dbname> with your actual credentials
const dbURL = 'mongodb+srv://<username>:<password>@cluster0.mongodb.net/<dbname>?retryWrites=true&w=majority';

mongoose
  .connect(process.env.dbURL)
  .then((result) => {
    console.log('Connected to MongoDB');
    app.listen(3000, () => {
      console.log('Server started on port 3000');
    });
  })
  .catch((err) => {
    console.error('Could not connect to MongoDB:', err);
  });

This setup ensures that Mongoose acts as the connector. It connects your application to the MongoDB database.

Step 7: Create the Model for the Document Structure

Next, we need to create a model document called a Schema so that when you post data to your database it will be saved accordingly.

To create this model:

1. Create a folder named models to keep your application organized.

2. Inside the model's folder, create a file called event.js.

In theevent.js file, you will use Mongoose to define the schema for the event documents. You'll specify the structure and data types for the documents you will send to your database.

Here's the event.js file created inside the model folder:

const mongoose = require('mongoose');

// Schema
const EventSchema = new mongoose.Schema(
  {
    title: {
      type: String,
      required: true,
    },
    date: {
      type: Date,
      required: true,
    },
    organizer: {
      type: String,
      required: true,
    },
    price: {
      type: String,
      required: true,
    },
    time: {
      type: String,
      required: true,
    },
    location: {
      type: String,
      required: true,
    },
    description: {
      type: String,
      required: true,
    },
  },
  { timestamps: true }
);

const Event = mongoose.model('event', EventSchema);

module.exports = Event;

When this is done, export so you can use it in your server.js file by simply using the require keyword.

With the schema created, it can now be exported to the server.js file.

Your server.js will look like this:

const express = require('express');
const ejs = require('ejs');
const mongoose = require('mongoose');
const Event = require('../models/Events');// the event.js file

Step 8: Create HTML Pages

As we talked about earlier, we're using ejs in step 4 to render HTML code, allowing us to view the code in the browser.

Form Page

First, let's create a form page. With the form page created, you'll be able to make POST requests which will enable you to send data to your MongoDB database.

To create a basic form, ensure it includes:

• An action attribute which specifies the route to send the data.

• A method attribute which specifies the HTTP request method – in this case, the POST request.

A basic form:

<form action="/submit-event" method="POST">
    <h2>Event Creation Form</h2>
  <label for="title">Title</label>
  <input type="text" id="title" name="title" required>

  <label for="date">Date</label>
  <input type="date" id="date" name="date" required>

  <label for="organizer">Organizer</label>
  <input type="text" id="organizer" name="organizer" required>

  <label for="price">Price</label>
  <input type="text" id="price" name="price" required>

  <label for="time">Time</label>
  <input type="text" id="time" name="time" required>

  <label for="location">Location</label>
  <input type="text" id="location" name="location" required>

  <label for="description">Description</label>
  <textarea id="description" name="description" rows="4" required></textarea>

  <button type="submit">Submit</button>
</form>

NB: Make sure to add the name attribute to each input, or it won't post.

The form created above will let you post data to the specified route. You will then process and store it in your database.

Here's the result:

The form page

After creating the form page, we need to go back to the server.js file and create a POST request to handle the form submission.

server.js file:

// posting a data

app.post('/submit-event', (req, res) => {
  const event = new Event(req.body);
  event.save()
    .then((result) => {
      res.redirect('/');
    })
    .catch((err) => {
      console.error(err);
    });
});

The Homepage

Now that the form can post data to the database, we can create the homepage to display the created events in the browser.

First, in your server.js file, you need to create a function. It will fetch all the events posted from the form and stored in the database.

Here’s how to set it up:

This is a function created at server.js to fetch all data from the database:

// To get all the event

router.get('/', (req, res) => {
  Event.find()
    .then((result) => {
      res.render('index', { title: 'All event', events: result })
    })
    .catch((err) => {
      console.error(err); 
  })
})

Next, we will dynamically loop through each part using a forEach loop in the homepage file. Since we are using ejs, the HTML file extension will be .ejs.

<div>
  <h2>All events</h2>
  <div>
    <% if (events.length > 0) { %>
      <% events.forEach(event => { %>
        <div>
          <h3><%= event.title %></h3>
          <p><%= event.description %></p>
          <a href="/event/<%= event.id %>">
            Read More
          </a>
        </div>
      <% }) %>
    <% } else { %>
      <p>No events available at the moment.</p>
    <% } %>
  </div>
</div>

Here is an explanation of what each part of the code does:

• Heading (<h2>All events</h2>): Displays "All events" as a heading.

• Event List (<div>): Container for displaying a list of events.

• Conditional Check ( <% if (events.length > 0) { %> ... <% } else { %> ... <% } %>): Checks if there are any events (events.length > 0). If events exist, it loops through each event (events.forEach) to display its details.

• For each event, it creates a <div> containing the event's title (event.title) in a<h3> tag, the event's description (event.description) in a <p> tag, and a link (<a>) to view more details about the event (Read More). The link directs to /event/event.id, where event.id is the unique identifier of the event.

• No Events Message (<% } else { %> ... <% } %>): If no events are present (events.length <= 0), it displays a message saying "No events available at the moment."

Step 9: Create Partials

Remember that you installed ejs into your application to facilitate more dynamic components. It allows you to break your code down further to be more dynamic.

To further organize your code, you'll use something called Partials.

Partials let you break down your code into scalable, modular, and manageable parts, keeping your HTML organized.

First, let's create a partial for the navbar.

How to Create a Partial:

• Inside your views folder, create a new folder named partials

• Inside the partials folder, create a new file called nav.ejs.

• Cut out the navbar code from your homepage file and paste it into nav.ejs.

Example:

First, create the Partials folder and file:

nav partial

Use the <%- include() %> syntax from ejs to include the nav.ejs partial across pages in your application where you want the navbar to appear.

Include () syntax

Here's the code:

<!DOCTYPE html>
<html lang="en">
    <%- include('./partial/head.ejs') %>

<body>
    <%- include('./partial/nav.ejs') %>
    <main>
      hello
    </main>
      <%- include('./partial/footer.ejs') %>
</body>
</html>

With this setup, your HTML code will be organized. It will be easy to manage and update components like the navbar across different pages. You can use this approach on other parts of your application. For example, the head tag, footer tag, and other reusable components.

Step 10: Create an Environment Variable File (.Env)

In this tutorial, we'll upload the project to GitHub. You'll protect your port number and MongoDB URL with secure storage. You'll also use an environment variable file, a configuration file known as .env. This file keeps sensitive information safe. It includes passwords and API URLs and prevents exposure.

Here's how to set it up using Node.js:

First, install the [dotenv](https://www.npmjs.com/package/dotenv) package.

npm i dotenv

Then create a .env file. Inside it, add your PORT number and MongoDB URL. It should look something like this:

PORT=3000
dbURl='mongodb+srv://<username>:<password>@cluster0.mongodb.net/<dbname>?retryWrites=true&w=majority';

Then update your .gitignore file:

/node_modules
.env

Adding .env to your .gitignore ensures that it is not included in your GitHub repository. This tells Git to ignore the .env file when uploading your code.

Then in your server.js file, require the dotenv package. Load the variables with this line at the top of the file:

To require it, simply type:

require('dotenv').config();

This way, you don't need to hardcode the PORT number and MongoDB URL in your server.js file. Instead, you can access them using process.env.PORT and process.env.dbURl.

So your server.js file will be cleaner and not messy 😵‍💫

require('dotenv').config();
const express = require('express');
const ejs = require('ejs');
const mongoose = require('mongoose');

mongoose
  .connect(process.env.dbURL)
  .then((result) => {
    console.log('Connected to MongoDB');
    app.listen(3000, () => {
      console.log('Server started on port 3000');
    });
  })
  .catch((err) => {
    console.error('Could not connect to MongoDB:', err);
  });

Further Steps

To expand on this basic application, consider adding features such as:

• User authentication

• Event search and filter functionality

• Event editing and deletion

• Notifications for upcoming events

How to Style the Application

If you want to add some styling to your application, follow these steps:

First, create a public folder. Inside this folder, create a style.css file where you will write your custom CSS.

Then in your HTML file, link the style.css file in the <head> tag as you normally would:

<link rel="stylesheet" href="/style.css">

To ensure your CSS file is served correctly, add the following line to your server.js file:

app.use(express.static('public'));

This application uses Tailwind CSS for styling. But using Tailwind is optional. You can use any CSS framework or write custom CSS to achieve your desired layout.

How to Include Images

All images should be stored in the public folder and referenced in your HTML files. You should also ensure that the public folder is correctly set up in your server.js file to serve static files.

Here's an example of how to serve static files in server.js:

const express = require('express');
const app = express();


// Serve static files from the 'public' folder
app.use(express.static('public'));

Conclusion

Congratulations! You've built a simple application using Node.js, Express.js, ejs, and MongoDB. With these fundamentals, you can expand and enhance your application to meet more specific needs and features.

Feel free to share your progress or ask questions if you encounter any issues.

If you found this article helpful, share it with others who may also find it interesting.

Stay updated with my projects by following me on Twitter, LinkedIn and GitHub

Thank you for reading 💖.

Happy coding!

The Node.js event loop is a crucial mechanism in Node.js that makes Node.js programs run concurrently and asynchronously. Mastering the Node.js event loop helps a Node.js developer understand how Node.js programs run under the hood.

In this article, you will learn the basics of the event loop, starting with threads and processes, then how the JavaScript event loop works, and finally, how the Node.js event loop works.

What are Threads and Process?

To master the Node.js event loop, you must understand processes and threads.

A programmer can write programs that perform different tasks with different programming languages. While some programming languages can run just one task at a time, other programming languages can run several tasks simultaneously.

A process involves several tasks running in a program from start to finish, while a thread is the running of an individual task.

A process consists of all the steps a program takes to run till completion. It is a currently executing program. A program may have one or more independent processes, each having its own memory space or address. A process may have one or more threads in it.

A thread is a single unit of execution that is part of a process, like a task in a program. A thread has a thread ID, a register set, and a stack. A thread also shares its code section, data section, operating system resources, and memory space with other threads in a process.

The code below contains an isNumberEven function that checks if a number is even and an isNumberOdd function that checks if a number is odd. A process involves running this code from start to finish, while a thread involves running individual functions.

function isNumberEven(number) {
  if (number % 2 === 0) {
    return true;
  } else {
    return false;
  }
}

function isNumberOdd(number) {
  if (number % 2 !== 0) {
    return true;
  } else {
    return false;
  }
}

isNumberEven(6);
isNumberOdd(1);

What are Single-threads and Multi-threads?

All programming languages have a runtime engine that runs their code. Some runtime engines are single-threaded (which means they can only run one thread at a time), while some are multi-threaded (which means they can run more than one thread at a time).

The diagram below shows a single-threaded process and a multi-threaded process:

Single-threaded and Multi-threaded Processes

A single-threaded programming language has a single-threaded runtime engine that runs tasks in a program sequentially. A multi-threaded programming language has a multi-threaded runtime engine that runs tasks in a program simultaneously. A multi-threaded runtime engine is more performant than a single-threaded runtime engine.

Programming languages like Java, C#, and so on are multi-threaded, while languages like JavaScript, Python, and so on are single-threaded.

Single-threaded programming languages are synchronous, which means they run the task in their programs sequentially. JavaScript is synchronous, but its event loop makes it asynchronous.

In the upcoming sections, you will learn how the JavaScript event loop works and then master the Node.js event loop.

How the JavaScript Event Loop Works

You must understand how the JavaScript runtime engine runs JavaScript code for you to understand the JavaScript event loop.

The JavaScript runtime engine consists primarily of the:

• Memory heap and

• Call stack

The memory heap is where variables declared in the program are allocated memory space, and the call stack is where the runtime engine stores functions in the program for execution.

The JavaScript runtime engine runs the code below synchronously and in this step-by-step process:

• It allocates memory space for all the variables in the code.

• It executes the exponentiation function after pushing onto the call stack.

• It executes the validatePassword function after pushing it onto the call stack.

function exponentiation(base, exponent) {
  let result = 1;
  for (let i = 0; i < exponent; i++) {
    result *= base;
  }
  return result;
}

function validatePassword(password) {
  const hasUppercase = /[A-Z]/.test(password);
  const hasLowercase = /[a-z]/.test(password);
  const hasNumber = /[0-9]/.test(password);
  const isValidLength = password.length >= 8;

  if (hasUppercase && hasLowercase && hasNumber && isValidLength) {
    return "password is valid";
  } else {
    return "password is invalid";
  }
}

exponentiation(5, 3);
validatePassword("Ab01234");

If your code contains a blocking function, which is a function that takes a lot of time to run, the function will block other functions from running until it completes. Users can’t interact with a website that has a blocking function as part of its code while the function runs.

The code below contains the fibonacci function, which takes time to run. The runtime engine starts running the factorial function first, then the fibonacci function, during which the findMin function can’t run.

function factorial(n) {
  if (n === 0 || n === 1) {
    return 1;
  }
  let result = 1;
  for (let i = 2; i <= n; i++) {
    result *= i;
  }
  return result;
}

function fibonacci(num) {
  if (num <= 1) {
    return num;
  }
  return fibonacci(num - 1) + fibonacci(num - 2);
}

function findMin(numbers) {
  if (!numbers || numbers.length === 0) {
    throw new Error("Empty array provided");
  }

  let min = numbers[0];
  for (let i = 1; i < numbers.length; i++) {
    if (numbers[i] < min) {
      min = numbers[i];
    }
  }
  return min;
}

let numbers = [4, 2, 8, 1, 6];

factorial(5);

fibonacci(45);

findMin(numbers);

To make the program in the example above run asynchronously, you should make the fibonacci function non-blocking with JavaScript web APIs.

JavaScript web APIs do not run on the main thread but instead create their threads, which enables them to run concurrently and not block the execution of other functions in your code.

You can use these JavaScript web APIs to make your functions non-blocking:

• setTimeout

• AJAX and

• DOM

So, if you add a setTimeout to your code like this:

function factorial(n) {
  if (n === 0 || n === 1) {
    return 1;
  }
  let result = 1;
  for (let i = 2; i <= n; i++) {
    result *= i;
  }
  return result;
}

function fibonacci(num) {
  if (num <= 1) {
    return num;
  }
  return fibonacci(num - 1) + fibonacci(num - 2);
}

function findMin(numbers) {
  if (!numbers || numbers.length === 0) {
    throw new Error("Empty array provided");
  }

  let min = numbers[0];
  for (let i = 1; i < numbers.length; i++) {
    if (numbers[i] < min) {
      min = numbers[i];
    }
  }
  return min;
}

let numbers = [4, 2, 8, 1, 6];

factorial(5);

setTimeout(fibonacci(45), 3000);

findMin(numbers);

The factorial and findMin functions will run on the main thread, while the fibonacci function runs concurrently on a separate thread.

To properly understand how the program above runs, you must understand how the JavaScript event loop works. The JavaScript event loop is a mechanism by which tasks in a JavaScript program run asynchronously.

The JavaScript event loop has a callback queue that stores functions that take time to execute.

The event loop sends functions that execute immediately to the callback queue for execution and sends blocking functions to web API threads for execution.

Then, the event loop sends the blocking function back to the callback queue when the set time elapses. The event loop then checks if the call stack is empty before pushing the function in the callback queue to the call stack for execution.

The diagram below explains how the event loop in JavaScript works:

The JavaScript event loop

The code above that contains factorial, fibonacci, and findMin functions is executed like this after adding a setTimeout function.

The event loop pushes the factorial function onto the call stack for execution. Then, the event loop pushes the fibonacci function onto the call stack, but the fibonacci function has a setTimeout function that prevents it from executing immediately. So, the event loop pushes the fibonacci function to a separate thread for web APIs to run concurrently.

Then, the event loop pushes the findMin function onto the call stack for execution. When the time set in the setTimeout elapses, the event loop pushes the fibonacci function to the call stack for execution.

How The Node.js Event Loop Works

Node.js is a JavaScript runtime environment that enables JavaScript to run outside the browser, like on the command line interface, servers, and hardware.

Node.js has an event loop that is similar to the JavaScript event loop. The Node.js event loop and the JavaScript event loop have a call stack and a callback queue. The Node.js event loop is implemented and managed by a library named libuv written in C.

The Node.js event loop has six phases, which are:

• Timer phase

• Pending Callbacks Phase

• Idle Phase

• Poll Phase

• Check Phase

• Close Callbacks Phase

The diagram below shows how the Node.js event loop works:

The Node.js event loop

There is a microtask queue that exists outside of the Node.js event loop. The microtask queue consists of the nextTick queue and the Promise queue. The nextTick queue runs the process.nextTick function, while the Promise queue runs .then, .catch, and other promises.

In the upcoming sections, you will learn about each phase of the Node.js event loop.

Timer Phase

There are three timers in Node.js: setTimeout, setInterval, and setImmediate. setTimeout and setInterval run in the timer phase. The code sample below runs during the timer phase:

setTimeout(() => {
  console.log("setTimeout callback executed");
}, 1000);

setInterval(() => {
  console.log("setInterval callback executed");
}, 2000);

Pending Callbacks Phase

I/O operations execute in the poll phase of the event loop. During the poll phase, some specific I/O operations callbacks defer to the pending phase of the next iteration of the event loop. I/O operations callbacks deferred from the previous iteration run in the pending callbacks phase.

The code sample below runs during the ‘pending callbacks’ phase:

const fs = require("fs");

fs.readFile(__filename, (err, data) => {
  if (err) throw err;
  console.log("File data:", data);
});

Idle Phase

The idle phase is not a normal phase of the Node.js event loop. It is a period whereby the event loop has nothing to do but perform background tasks like checking for low-priority results or running garbage collection.

To skip the idle phase and not perform background tasks, you can call the idle.ignore() method from the idle-gc package in your code.

const { idle } = require("idle-gc");

idle.ignore();

The idle.ignore() method ensures that the code continues to run without any idle period till completion. However, due to the performance issues it causes, the idle.ignore() method should be used sparingly.

Poll Phase

The poll phase is where I/O operations execute. I/O operations transfer data to or from a computer. The event loop checks for new I/O operations and executes them in the poll queue.

The code sample below runs during the poll phase:

const http = require("http");

http.get("http://jsonplaceholder.typicode.com/posts/1", (res) => {
  console.log("HTTP request response received");
  res.on("data", (chunk) => {
    // Do something with the data
  });
});

Check Phase

The check phase is where the setImmediate timer runs. The Node.js event loop goes to the check phase when there is a setImmediate in the program, and the poll phase becomes idle or when the poll phase completes.

The code sample below runs during the check phase:

setImmediate(() => {
  console.log("setImmediate callback executed");
});

Close Callbacks Phase

The close callbacks phase is the last phase of the Node.js event loop. The close callback phase is where callbacks from the close event of a socket and the closing of an HTTP server run.

The code sample below runs during the check phase:

const http = require("http");

const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello World\n');
});

server.listen(3000, () => {
  console.log('Server listening on port 3000');
  server.close(() => {
    console.log('Server closed');
  });
});

Conclusion

The Node.js event loop is the mechanism that enables asynchronous programming in Node.js.

As a Node.js developer, you can understand how your Node.js code runs under the hood if you master the Node.js event loop.

This article explained processes and threads, the JavaScript event loop, and the Node.js event loop.

What set this app apart was that instead of users entering a URL to generate a QR code, they would initiate a request through the app. Then a unique QR code would be generated which would be associated with their account details. The QR code would then be displayed on their screen while the payer scans the QR code using their mobile device's camera.

In this tutorial, you'll learn how to develop a custom QR code generator for URLs using Node.js and Next.js. I'll walk you through the process step by step, including setting up Azure Blob Storage to store the generated URLs. These URLs will then be displayed in the form of QR codes in your Next.js frontend application.

We'll build the backend of the application using Node.js and the Express framework, and the frontend (which interacts with the backend) with Next.js.

I'll also provide explanations on QR codes, the concept of buffers for handling binary data in Node.js, and how it is being used to stream the QR code image data to Azure Blob Storage.

So, let dive in.

Prerequisites

Before you begin you'll need an active Azure account and subscription to create an Azure blob storage.

Table of Contents

1. What is a QR Code?

2. How to Set Up Azure Blob Storage

3. Node.js QR Code

4. Code Overview

5. How to Connect the Frontend Application

6. Next.js Code Walkthrough

7. How to Start the Application Locally

8. Conclusion

What is a QR Code?

According to Investopedia, a Quick Response (QR) code functions as a specialized barcode, scannable by digital devices, that stores data within a grid of square pixels.

QR codes are extensively employed in digital payments, cryptocurrency, and transmitting web addresses to mobile devices. They can encode URLs, facilitating webpage access.

Now, let's delve into the process of programmatically generating QR codes. Today, I'll demonstrate this step-by-step. To begin, you'll set up an Azure Blob Storage instance in your Azure Portal.

How to Set Up Azure Blob Storage

Azure Blob Storage is a cloud-based storage service provided by Microsoft Azure. It is part of the Azure Storage suite, which also includes services such as Azure Files, Azure Queues, and Azure Tables.

Azure Blob Storage is designed to store large amounts of unstructured data, such as text or binary data, in the form of objects called blobs. If you are familiar with AWS, Azure Blob Storage is similar to an S3 bucket. Storage accounts are primarily accessed via REST API.

Step 1: Create a storage account

You can create your storage account by simply searching for Storage account in the search bar at the top of the Azure Portal.

Create Storage Accounts in Azure Portal

You can then run through the steps to create your storage account. Just note that this name needs to be unique and it also needs to be all lower case – no spaces but it can include numbers.

Create a New Resource Group

Step 2: Create a container

After creating your storage account, you can now create a container.

Create a New Container

When accessing blob storage with the QR codes stored in Azure Storage, the URL typically follows a structure like https://<storage_account_name>.blob.core.windows.net/<container_name>/<blob_name>.

Having a container allows us to structure the URLs in a meaningful and organized way, making it easier to manage and share the generated QR codes.

Step 3: Obtain Azure Storage connection string

In the Security + networking section, select "Access keys."

Make sure to copy the connection string and save it somewhere as it's required to establish a secure connection between the Azure Storage account and the Node.js application.

Obtain Access Key for the Container

That concludes the discussion on Microsoft Azure Storage. I must say, I thoroughly enjoyed exploring and solving these challenges.

Next, you will be diving into coding, specifically around Nodejs and and then moving on to frontend development, where you'll be using Next.js.

Node.js QR Code

First, you need to install Node.js and npm on your computer. Go to the Node.js website and download the version for your computer if you don't have it already.

Once you've installed them, check if Node.js and npm are installed correctly by typing these commands in your terminal:

node -v
npm -v

Next, go to this GitHub link to fork the project and then clone it to your preferred directory on your computer.

Github Repository Project

Once you've cloned the project repository, open the project with your code editor. I'm using VS Code. You'll notice that the project has of two folders: server and frontend. You'll start by navigating to the server folder by typing cd server in your terminal and then press Enter.

Change the current directory to a directory named "server."

Now you can install all the necessary dependencies by running npm install. This command will download and install all the required packages for the server-side application.

installation of dependencies

If everything went well, you should see something like this below after npm install:

Successful Installation

Next, you need to create a .env file in the server directory to store your environment variables. It's not advisable to hardcode sensitive credentials. You can do this easily by running touch .env in your terminal.

Create a new file named ".env"

In the newly created .env file, you'll define three variables that your application depends on: CONTAINER_NAME , AZURE_STORAGE_CONNECTION_STRING, and PORT .

Assign 'qrcode' as the value for CONTAINER_NAME. This was the name of the container you created inside the Azure Storage account. Also, set the PORT to 8000, which is the port your backend application will be listening on.

Now, for the AZURE_STORAGE_CONNECTION_STRING, you'll need to obtain the secret key from the access key you obtained earlier. Copy the connection string and paste it as the value for AZURE_STORAGE_CONNECTION_STRING in the .env file.

Store Secret Variables

Once you've added these environment variables to the .env file, save it and you're all set to run the server-side of the application!

Before you run the application, let me quickly explain the code. Click on the index.js file.

View Index.js file

Code overview:

Below is the code snippet containing the necessary logic for generating QR codes and establishing a connection with the Azure storage account you previously created.

const express = require('express');
const { BlobServiceClient, generateBlobSASQueryParameters, BlobSASPermissions } = require('@azure/storage-blob');
const qrcode = require('qrcode');
const { v4: uuidv4 } = require('uuid');
const { Readable } = require('stream');
const dotenv = require('dotenv');

dotenv.config();

const app = express();
const port = process.env.PORT || 5000;

// Allowing CORS for local testing
const origins = [
    "http://localhost:3000"
];

app.use((req, res, next) => {
    res.header('Access-Control-Allow-Origin', origins.join(','));
    res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept');
    next();
});


const containerName = process.env.CONTAINER_NAME;

const blobServiceClient = BlobServiceClient.fromConnectionString(process.env.AZURE_STORAGE_CONNECTION_STRING);

app.use(express.json());

app.post('/generate-qr', async (req, res) => {
    const { url } = req.body;

    // Generate QR Code
    console.log('Received URL:', url);
    const qrCode = await qrcode.toBuffer(url);

    const bufferStream = new Readable();
    bufferStream.push(qrCode);
    bufferStream.push(null);

    // Generate unique file name for Azure Blob Storage
    const fileName = `qr_codes/${uuidv4()}.png`;

    try {
        const containerClient = blobServiceClient.getContainerClient(containerName);
        const blockBlobClient = containerClient.getBlockBlobClient(fileName);

        await blockBlobClient.uploadStream(bufferStream, 4 * 1024 * 1024, 20, {
            blobHTTPHeaders: {
                blobContentType: 'image/png'
            }
        });

        // Generate SAS token for blob
        const sasToken = generateSasToken(blockBlobClient);

        // Generate the Blob URL with SAS token
        const blobUrlWithSasToken = `${blockBlobClient.url}?${sasToken}`;

        // Send response with the Blob URL containing SAS token
        res.json({ qr_code_url: blobUrlWithSasToken });
    } catch (error) {
        console.error('Error generating QR Code:', error);
        res.status(500).json({ error: 'Internal Server Error' });
    }
});

// Function to generate SAS token for blob
function generateSasToken(blobClient) {
    const blobSAS = generateBlobSASQueryParameters({
        containerName: blobClient.containerName,
        blobName: blobClient.blobName,
        permissions: BlobSASPermissions.parse("r"), // Read permission
        startsOn: new Date(),
        expiresOn: new Date(new Date().valueOf() + 86400) // Token expires in 24 hours
    }, blobClient.credential);

    return blobSAS.toString();
}

app.listen(port, () => {
    console.log(`Server is running on port ${port}`);
});

Now I'll give a detailed explanation of the code structure, functionalities, and key components of the application.

Importing required modules:

• const express = require('express'): This line imports the Express.js framework, which is a Node.js web application framework for building web applications and APIs. It allows you to define routes, handle HTTP requests, and more.

• const { BlobServiceClient, generateBlobSASQueryParameters, BlobSASPermissions } = require('@azure/storage-blob'): This line imports specific modules from the @azure/storage-blob package, which is the Azure Blob Storage SDK for JavaScript. It allows you to interact with Azure Blob Storage from our Node.js application.

• const qrcode = require('qrcode'): This line imports the qrcode module, which is a popular Node.js library for generating QR codes.

• const{ v4: uuidv4 } = require('uuid'): This line imports the uuid module and specifically extracts the v4 function as uuidv4. The uuid module is used to generate universally unique identifiers (UUIDs) in Node.js.

• const{ Readable } = require('stream'): This line imports the Readable class from the built-in Node.js stream module. The Readable class is used to create readable streams, which are useful for handling data that can be read sequentially.

Configuring environment variables:

• dotenv.config();: This line loads environment variables from a .env file into process.env. The .env file typically contains CONTAINER_NAME and AZURE_STORAGE_CONNECTION_STRING you specify in your .env file.

Initializing the Express application:

• const app = express();: This line initializes an Express application instance, which you'll use to define routes, middleware, and other configurations for your web application.

Defining port configuration:

• const port = process.env.PORT || 5000: This line sets the port number for the Express application. It retrieves the port number from the process.env.PORT environment variable, if it exists. If not, it defaults to port 5000. This allows flexibility for deploying the application in different environments where the port may be specified externally.

Allowing CORS for local testing:

• CORS (Cross-Origin Resource Sharing) is a security feature implemented by web browsers to restrict resources from being requested from another domain or application.

• In this section, CORS is being configured to allow requests from a specific origin (http://localhost:3000), which is typically used during local development.

• The app.use() function is used to add middleware to the Express application. Here, a middleware function is defined that sets the necessary CORS headers on every HTTP response.

• res.header('Access-Control-Allow-Origin', origins.join(',')): Sets the value of the Access-Control-Allow-Origin header to allow requests from the specified origins (in this case, http://localhost:3000).

• res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept'): Sets the allowed headers for the CORS request.

• next(): Calls the next middleware function in the stack.

Azure Blob Storage configuration:

• containerName and blobServiceClient are initialized using environment variables (process.env.CONTAINER_NAME and process.env.AZURE_STORAGE_CONNECTION_STRING) configured earlier.

• blobServiceClient is initialized using the fromConnectionString() method from the BlobServiceClient class provided by the @azure/storage-blob package. This allows the application to interact with Azure Blob Storage using the provided connection string.

Express application configuration:

• app.use(express.json()): Adds middleware to parse JSON bodies of incoming requests. This enables the application to handle JSON data in requests.

Endpoint for generating QR codes:

• Defines a POST endpoint at /generate-qr to handle requests for generating QR codes.

• Upon receiving a request, the endpoint extracts the URL from the request body and generates a QR code image using the qrcode.toBuffer() function.

• The generated QR code image is then uploaded to Azure Blob Storage as a blob with a unique file name.

• After successfully uploading the image, a Shared Access Signature (SAS) token is generated for the blob, which provides temporary access to the blob with specified permissions (in this case, read-only).

• Finally, the endpoint responds with a JSON object containing the URL of the generated QR code image along with the SAS token.

Function to generate SAS token for blob:

• Defines a function generateSasToken() to generate a SAS token for a given blob client (block blob client in this case). The SAS token is generated with read permissions and an expiration time set to 24 hours.

Listening on port:

• The Express application listens on the configured port (port) for incoming HTTP requests. When the server starts, it prints a message indicating the port it is listening on.

Now, you can start the application locally.

To start the application, simply run npm start as shown below. If all goes well, you'll observe the message Server is running on port 8000 printed on your console.

Start the Application locally

How to Connect the Frontend Application

Now it is time to connect the frontend application with the backend application that is listening on Port 8000.

A typical full-stack application usually consists of at least two main components: a frontend (client-side) and a backend (server-side).

Frontend Component: This is the part of the application that users interact with directly. It's typically built using technologies like HTML, CSS, and JavaScript frameworks like React, Angular, or Next.js.

Backend Component: This is the part of the application that handles data storage, retrieval, and server side connectivity. It's usually built using server-side programming languages like Node.js (with frameworks like Express.js or Nest.js), Python (with frameworks like Django or Flask), Java (with frameworks like Spring), or Ruby (with frameworks like Ruby on Rails).

The backend communicates with the frontend, processes requests from users, interacts with databases, and generates responses.

To navigate to the frontend folder, open a new terminal by clicking the + , then use cd frontend to enter the frontend folder.

Navigate to Frontend Directory

Now, you can install all the necessary dependencies by running npm install. This command will download and install all the required packages for the Nextjs client-side application.

If everything went well, you should see something like this after npm install.

Successful Installation of Dependencies

Before you run the application, let me quickly explain the code.Navigate to the frontend folder, then inside the src/app directory, click on the page.js file.

View page.js file

Next.js Code Walkthrough

This code represents a React component serving as the frontend for the QR code generator backend application you've recently built. This component allows users to input a URL, submit it, and receive the corresponding QR code image for display.

'use client'

import { useState } from 'react';
import axios from 'axios';

export default function Home() {
  const [url, setUrl] = useState('');
  const [qrCodeUrl, setQrCodeUrl] = useState('');

  const handleSubmit = async (e) => {
    e.preventDefault();
    try {
      const response = await axios.post('http://localhost:8000/generate-qr', { url });
      setQrCodeUrl(response.data.qr_code_url);
    } catch (error) {
      console.error('Error generating QR Code:', error);
    }
  };

  return (
    <div style={styles.container}>
      <h1 style={styles.title}>QR Code Generator</h1>
      <form onSubmit={handleSubmit} style={styles.form}>
        <input
          type="text"
          value={url}
          onChange={(e) => setUrl(e.target.value)}
          placeholder="Enter URL like https://www.google.com"
          style={styles.input}
        />
        <button type="submit" style={styles.button}>Generate QR Code</button>
      </form>
      {qrCodeUrl && <img src={qrCodeUrl} alt="QR Code" style={styles.qrCode} width="200" height="200" />}
    </div>
  );
}

// Styles
const styles = {
  container: {
    minHeight: '100vh',
    display: 'flex',
    flexDirection: 'column',
    alignItems: 'center',
    justifyContent: 'center',
    backgroundColor: '#121212',
    color: 'white',
  },
  title: {
    margin: '0',
    lineHeight: '1.15',
    fontSize: '4rem',
    textAlign: 'center',
  },
  form: {
    display: 'flex',
    flexDirection: 'column',
    alignItems: 'center',
  },
  input: {
    padding: '10px',
    borderRadius: '5px',
    border: 'none',
    marginTop: '20px',
    width: '300px',
    color: '#121212'

  },
  button: {
    padding: '10px 20px',
    marginTop: '20px',
    border: 'none',
    borderRadius: '5px',
    backgroundColor: '#0070f3',
    color: 'white',
    cursor: 'pointer',
  },
  qrCode: {
    marginTop: '20px',
  },
};

Now I'll give a detailed explanation of the frontend application's code structure, functionalities, and key components.

State management:

• import { useState } from 'react' imports the useState hook from React to manage state within the component.

• const [url, setUrl] = useState('') and const [qrCodeUrl, setQrCodeUrl] = useState(''): This state variables, url and qrCodeUrl, are initialized using the useState hook. These variables hold the input URL and the generated QR code URL, respectively.

Form submission:

• When the form is submitted, the handleSubmit function is triggered.

• This function prevents the default form submission behavior by using e.preventDefault().

• It sends a POST request to the server (http://localhost:8000/generate-qr) with the input URL using the Axios library.

• Upon successful response, the generated QR code URL is stored in the qrCodeUrl state variable.

Rendering:

• The component renders a title, a form with an input field for entering the URL, and a button to generate the QR code.

• When the QR code URL is available (qrCodeUrl is not empty), an image element is rendered to display the generated QR code.

Styling:

• The component includes inline styles defined using JavaScript objects.

• Styles are applied to the container, title, form, input field, button, and QR code image.

How to Start the Application Locally

Now, you can start the application locally.

To start the application, simply run npm run dev as shown below. If all goes well, you'll observe the message http://localhost:3000 printed in your console.

Application ready to start

Open your browser and paste the URL http://localhost:3000. The browser should render the application and look exactly as shown below.

Application running in Browser

Paste in the URL of a website – either your portfolio website or any website you wish to generate a QR code for. I pasted my portfolio website URL, https://wilfred-portfolio.vercel.app/, into the URL box. See the result below:

QR Code generated successfully

Conclusion

When it comes to picking the right tools, technology choices are important. Next.js is great for making the front end of a website, while Node.js works well for handling server-side tasks. Also Azure blob storage is great for storing unstructured data such as binary data like QR code.

But remember, this journey is not just about writing code. It's also about learning about different technologies and picking the best ones for what you need to do.

As I finish up this tutorial, I'd like to keep asking for feedback to ensure that this tutorial stays helpful. Feel free to share your thoughts or comments with me.

Thanks for reading!

Happy coding! 🚀

Contact Me:

• Twitter

• LinkedIn

• GitHub

In this fast-paced development landscape, deploying via containers, particularly with Kubernetes, is becoming increasingly popular. Some companies perform numerous deployments daily, so it's crucial for you to learn these technologies.

Kubernetes is a popular choice to deploy containerized applications like web servers, databases, and APIs. You can set up Kubernetes either locally or in the cloud. In this tutorial, we'll explore setting up Kubernetes on a cloud platform, specifically Azure.

I'll walk you through the process of setting up Kubernetes using Azure Kubernetes Service (AKS). You'll configure your YAML file using StatefulSet, Persistent Volume, and Services to deploy a PostgreSQL database on Azure Kubernetes. Then, you'll obtain the PostgreSQL database credentials running inside the AKS and use them to establish a connection with a Node.js application.

We'll cover key concepts such as deployment, stateful sets, persistent volumes, and services, preparing you to deploy a Postgres container effectively on AKS. I'll also help you connect your Node.js Express app to the Postgres container within the AKS cluster.

So find a comfortable seat and get ready, as we're about to dive in.

Prerequisites

Before you begin, it's important to understand some basic concepts in Kubernetes like pods, deployments, services, and nodes.

If you're new to this, I recommend checking out the Stashchuk freeCodeCamp video for a beginner-friendly tutorial.

You'll also need an active Azure account and subscription to follow along.

Table of Contents

• Challenges We're Trying to Solve
  – Deployments
  – StatefulSets
  – Persistent Volumes

• Azure Kubernetes Service (AKS)
  – Sign in to Your Azure Portal
  – Create a Resource
  – Create a new container
  – Create a new Azure Kubernetes Service(AKS)
  – Create a new resource group
  – Give your Kubernetes cluster a name
  – Navigate to the node pool page
  – Enable container logs and set up alerts
  – Advanced Section
  – Tags

• Connect to Your AKS Cluster
  – Download Azure CLI and kubectl
  – Verify if Azure CLI is installed
  – Verify if kubectl is installed
  – Login to Azure account
  – Configure kubectl

• How to Create Resources with YAML
  – Clone the Repository
  – Open the Repository
  – Install Dependencies

• YAML Configuration
  – StorageClass
  – PersistentVolumeClaim
  – ConfigMap
  – StatefulSet
  – Service

• How to Deploy YAML Resource to Azure
  – Deploy the YAML resource

• Node.js Application
  – Configure Nodejs
  – Run Nodejs Application
  – Test the Application
  – Open Postman
  – Confirm the Data
  – Delete Pod
  – Data Persistence

• Conclusion

Challenges We're Trying to Solve

Firstly, what is Kubernetes? Well, it's like a manager for your software containers. It helps you run and manage lots of containers like web servers, databases, microservices, and APIs which are like little packages holding your applications.

Kubernetes takes care of things like starting, stopping, and scaling these containers, so your apps run smoothly even when there is more load on your application. It's popular because it makes running software in the cloud easier and more reliable.

Now, let's talk about how to tackle some challenges you might face with a real-world application running Postgres in a Kubernetes production cluster.

Imagine that the infrastructure hosting your Postgres crashes, causing you to lose all the services and data stored in the database. Or, picture a scenario where the Postgres database becomes corrupted, leading to data loss.

In both cases, you need a way to back up your application so you can restore it to a working state if disaster strikes.

So, how do you capture a comprehensive application backup that includes all the necessary data? This backup should allow you to restore the entire application, including the database, if you lose your cluster or encounter data loss.

In Kubernetes, think of a Pod as the tiniest unit that you can deploy. It's like a small box that holds one thing, like a web server or a database. So, if your Pod isn't running, your web server or database isn't either.

This means that if the cluster where your Pod runs gets destroyed, all the data in the Pod disappears too. All the nodes (virtual machines that run your application over the network) will also be wiped out.

How can you make a pod stay on one specific node where the data is and never move? And how can you make sure that each pod can be found separately when you're using a load balancer?

One solution is to consider how you deploy your application on Kubernetes. Typically, you create a deployment and expose it using a service, specifying the service type as either Cluster type, NodePort, or LoadBalancer.

But not all applications are the same when it comes to state. Some applications, known as stateless applications, don't rely on storing data locally, so losing their state isn't a big issue.

But for applications like databases or caches, maintaining state is crucial because they rely on storage. In Kubernetes, deploying stateful applications like databases using just deployment isn't ideal. You need a solution that ensures your application's data is safely stored and can be recovered in case of failure.

Deployments

You might be wondering why we can't just use a Kubernetes deployment to deploy Postgres in the Kubernetes cluster? Well, the thing is, many people aren't aware of the difference between a deployment and a stateful set.

Let's imagine you have a pod running in your cluster that you created using a deployment. Then you scaled up to two pods, so you now have Pod A and Pod B.

The problem arises because, by default, pods created as part of the same deployment share the same persistent volume (PV) across the cluster. So, when you scaled up, both instances of Postgres would write to the same storage, which could lead to data corruption.

Another issue arises from a networking perspective. Pods A and B don't have a dependable way to communicate with each other over the network. By default, Kubernetes pods don't have their own DNS names. Instead, you rely on services to expose ports to other applications in the cluster.

If you take a closer look at pod names, you'll notice that pods are assigned a random hash at the end of their names. Because of this, pods lack a consistent network identity. Every time a pod is destroyed and recreated, it receives a new randomized name. This inconsistency isn't ideal for reliable networking.

Postgres isn't naturally made for Kubernetes, and Kubernetes can be tough when handling stateful tasks. To set up a Postgres instance, you've got to know the right Kubernetes setup. You can't just throw it in a pod, because if the pod goes down, so does your data. But, for a quick integration, a pod could work fine.

Deployments aren't ideal either, since you don't want your pod randomly placed on a node. But for testing, deployments are handy if you just need a Postgres instance to run temporarily.

What you really want is a pod that sticks to a particular node where your data resides, and stays put. Plus, you also want your pod to be individually addressable. for this we need what we called a statefulSet.

StatefulSets

When you update your deployment to become a StatefulSet, Kubernetes introduces some improvements for deploying stateful workloads. One major change is how it handles scaling.

If you specify that you want three replicas of your StatefulSet, Kubernetes won't create all three pods at once. Instead, it creates them one by one. Each pod gets its own unique DNS name, starting with the pod's name followed by an ordinal number starting from zero. So, when you scale up, the ordinal number increases for each new pod.

Here's the cool part: if a pod like Pod-0 is destroyed and needs to be remade, it will return with the same name. This means each pod has a specific address, even if it's replaced.

And here's another cool feature: each pod in a StatefulSet gets its own persistent volume (PV). This lets you keep the same storage even if you scale up or down. This brings us to another concept called persistent volumes.

Persistent Volumes

Let's forget about pods, deployment, and containers for a moment. What exactly is "state"? In simple terms, state is the data that your applications need to work properly.

Now, when we talk about processes, there are two types: stateless and stateful. Stateless processes don't rely on any data to work. They just do their thing without needing any specific information. On the other hand, stateful processes need data or state to function properly.

Now, where do you store this state? There are two main places: memory and disk. Memory allows for quick access to data, which is great for applications like Redis, MongoDB, Postgres, or MySQL. They store their state on memory for quick access. But for persistent, they store it on disk on the file system (for more permanent storage).

Why the file system? Because it's the only way to keep the state persistent even when the system reboots. So, when a process dies and gets recreated, it can read its state from the file system.

I like breaking things down because I used to teach tech stuff. Now, let's get into setting up Kubernetes in Azure.

Azure Kubernetes Service (AKS)

In this section, I'll guide you through setting up a Kubernetes cluster on Azure.

Step 1: Sign in to your Azure portal

To begin, you will have to sign into your Azure portal. Once logged in, you should see a dashboard similar to this:

Azure portal homepage

Step 2: Create a resource

Click on "create a resource" to create a resource.

Resources are the various services, components, and assets that you can create and manage within the Azure cloud platform. These resources can include virtual machines, databases, storage accounts, networking components, web applications, and more.

Creating a resource in Azure portal

Step 3: Create a new container

Next, navigate to the "Containers" category from the options available on the left pane. Click on Containers as shown by the arrow in the screenshot.

Again, Kubernetes is a container orchestration platform. It manages and orchestrates the deployment, scaling, and operation of application containers across clusters of machines. Kubernetes provides a framework for automating the deployment, scaling, and management of containerized applications.

Creating new container (Kubernetes) in Azure

Step 4: Create a new Azure Kubernetes Service (AKS)

Select "Azure Kubernetes Service (AKS)" from the list of available container services and click Create. This will take you to the AKS creation page.

Creating a new Azure Kubernetes service

Step 5: Create a new resource group

In the "Resource group" section, click on "Create new" to create a new resource group for your Azure Kubernetes Service (AKS) deployment.

In Azure, a "resource group" is a logical container used to group together related Azure resources. It serves as a way to organize and manage these resources collectively, rather than individually.

When you create resources such as virtual machines, databases, storage accounts, or any other Azure service, you typically associate them with a resource group.

Creating a new resource group in azure portal

Let's name the resource group "AZURE-POSTGRES-RG" as shown below. You can name it anything you like. Then click ok.

Inputting name for the resource group

Step 6: Give your Kubernetes cluster a name

Now let's name the session for configuring the Kubernetes cluster "Kubernetes Cluster Name".

In Azure, a Kubernetes cluster is a managed container orchestration service provided by Azure Kubernetes Service (AKS). It allows you to deploy, manage, and scale containerized applications using Kubernetes without having to manage the underlying infrastructure.

Give it a name like "AZURE-POSTGRES-KC" and and select a region that's close to you. In my case I select (Asia Pacific) East Asia and click next.

Naming the Kubernetes cluster name

Step 7: Navigate to the node pool page

Now it's time to configure the node pool session by clicking on the agentpool.

In Azure, a node pool is a group of virtual machines (VMs) that are provisioned and managed together within an Azure Kubernetes Service (AKS) cluster. Each node pool runs a specific version of Kubernetes and has its own set of configurations, such as VM size, OS image, and node count.

Editing agentpool

Set the minimum node count to 1, maximum node count to 2, and the maximum pods per node to 30 to minimise cost. Then click update.

These parameters help control the size and behavior of the node pool in an Azure Kubernetes Service (AKS) cluster:

1. Minimum Node Count: Ensures a minimum number of nodes are always available for consistent performance and availability, even during low-demand periods.

2. Maximum Node Count: Sets an upper limit on the number of nodes in the node pool to manage costs and prevent over-provisioning.

3. Maximum Pods per Node: Defines the maximum number of pods that can run on each node, optimizing resource utilization and preventing overcrowding.

Updating agentpool details

Once you've clicked "Update," you'll be directed to the "Networking" section as shown below. Keep the page as is and proceed by clicking "Next." This will take you to Integration session.

Navigating to Next Page

Azure Container Registry (ACR) is a fully managed private Docker registry service provided by Microsoft Azure. It enables developers to store, manage, and deploy Docker container images securely within their Azure environment.

You will need a place to store the Docker image that's pulled.

To begin, select "Create New" to set up a new container registry. This action will bring up a page where you can input the necessary details, as illustrated on the right side of the image below. Enter the details as indicated by the arrows and then click "Okay." Once you're done, proceed by clicking "Next."

Naming and editing Azure Container Registry details

Step 8: Enable container logs and set up alerts

The Enable Container Logs option allows you to turn on logging for your containers. Logging records important information about what's happening inside your containers, like errors, warnings, and other events. It's useful for troubleshooting and monitoring your applications.

Choosing container logs

Step 9: Advanced section

Keep the Monitoring section unchanged and proceed by clicking "Next."

Navigating to Next Page

Step 10: Tags

Keep the Tags section unchanged and proceed by clicking "Next."

Navigating to Next Page

Step 11: Click "Review + create" to finalize the deployment

Once completed, your resource group, Azure Kubernetes Service (AKS), Azure Container Registry, and Kubernetes cluster will be created.

Completing Azure Kubernetes Setup

The screenshot below shows that the deployment was successful.

Successful Deployment

You've just successfully created an Azure Kubernetes Service from the Azure portal. Congrats!

How to Connect to Your AKS Cluster Using the Command Line

After successfully creating a new AKS in the Azure portal, the next step is to establish a connection to that cluster.

In this section, I'll guide you through Azure login, configuring kubectl to use the current context, and creating the YAML file for our Postgres container. This file will include StatefulSet, persistent volume, persistent volume claim, config map, and using Azure File for data storage.

I'll also show you how to run a Node.js Express application locally, use Postman to test the endpoints, and receive a response confirming that data was sent to the database successfully.

Download Azure CLI and kubectl

To start, you'll need to download the Azure CLI and kubectl.

• Azure CLI (Command-Line Interface): a command-line tool provided by Microsoft for managing Azure resources. It allows users to interact with Azure services and resources directly from the command line, making it easy to automate tasks, create scripts, and manage Azure resources programmatically.

• kubectl: a command-line tool for managing Kubernetes clusters, used to deploy, scale, and manage containerized applications. It allows users to perform operations like deploying applications, managing pods, services, and deployments, inspecting cluster resources, scaling applications, and debugging issues, simplifying management of containerized workloads in a Kubernetes environment.

I'm using the warp terminal. Warp is the terminal reimagined with AI and collaborative tools for better productivity. You can run the command using PowerShell on Windows or Terminal on Mac. I'm using a MacBook.

Verify if the Azure CLI is installed by typing the command az --version

Once the download finishes, verify whether Azure CLI is installed on your computer by running the command az --version. If the installation is successful, you should see an output similar to this:

Verifying Azure CLI Installation

Verify if kubectl is installed

To check if kubectl is installed, just type kubectl version in the command line.

Verifying Kubectl Installation

Login to your Azure account

Enter az login in the command line. This will open your browser and prompt you to sign in to your Azure account.

Logging into Azure

After signing in, it shows details about your Azure subscription, including the subscription name, ID, and user information.

Select an Azure subscription

Azure subscriptions are logical containers used to provision resources in Azure. You'll need to locate the subscription ID that you plan to use in this module. Use the command to list your Azure subscriptions:

az account list --output table

Use the following command to ensure you're using an Azure subscription that allows you to create resources for the purpose of this module, substituting your subscription ID (SubscriptionId):

az account set --subscription "Name of the subscription"

Configure kubectl to connect to your Azure Kubernetes

Replace Your_Azure_Resource_groups_name in the code below with the name you chose when creating a resource group. Also, replace your_azure_kubernetes_service_name with the name of your Kubernetes cluster. Then, execute the following command:

az aks get-credentials --resource-group [Your_Azure_Resource_groups_name] --name [your_azure_kubernetes_service_name]

The output should look like this:

Merging kubectl with Azure Kubernetes Service

Verify if kubectl has been merged successfully

Run the following command kubectl get nodes:

Verifying if merged is successful

When you run this command, Kubernetes communicates with the cluster's control plane to fetch a list of all the nodes that are part of the cluster you created. As you can see, this is the node that was running in the Kubernetes cluster we created inside Azure.

Virtual Node running in AKS cluster

Run the command kubectl get pods

Displaying Pod Information

When you run the command kubectl get pods, Kubernetes attempts to retrieve information about all pods within the default namespace of your cluster. But in this case, the output indicates that there are no resources (pods) found within the default namespace, implying that no pods currently exist in that namespace.

A namespace in Kubernetes is a virtual cluster environment within which resources like pods, services, and deployments are organized and isolated. It's a way to divide cluster resources between multiple users, teams, or projects. Namespaces provide a scope for names and make it easier to manage and control access to resources.

By default, Kubernetes starts with a "default" namespace, but you can create additional namespaces to organize and manage resources more effectively. Namespaces help prevent naming conflicts and provide a logical separation of resources, allowing different teams or projects to work independently within the same Kubernetes cluster.

Create a namespace

Creating Namespace

When you run the command kubectl create namespace database, Kubernetes creates a new namespace named "database." The output "namespace/database created" confirms that the namespace has been successfully created.

You can now use this namespace to organize and manage resources related to databases within the Kubernetes cluster.

Confirm the namespace

The command kubectl get namespace lists all namespaces in the Kubernetes cluster including the database namespace we just created, showing their names, status (active), and age.

Confirming namespace

Get pod information in database namespace

Displaying Pod Information related to database namespace

This command, kubectl get pods -n database, attempts to fetch information about pods specifically within the "database" namespace. But the output No resources found in database namespace indicates that there are currently no pods deployed in the "database" namespace.

How to Create Resources with YAML

Let's explore creating resources with YAML to provision our PostgreSQL database running in an Azure Kubernetes cluster. But first, what exactly is YAML?

Kubernetes YAML is a configuration file written in YAML (YAML Ain't Markup Language). They define how Kubernetes resources like pods, deployments, and services should be set up within a cluster. These files are easy to read and specify details like resource names, types, specifications, labels, and annotations. They're crucial for deploying applications and infrastructure on Kubernetes clusters.

YAML is what you will use to create Kubernetes resources that will run Postgres.

First, you need to clone this GitHub repository. Inside, you'll find a Node.js Express application and a YAML file. The Node.js app allows users to register with their email, password, and full name, and also enables them to log in by verifying their details in the database. If their details are found, it displays a success message.

Clone the repository

Create a new folder on your computer and then clone this repository into it.

Open your terminal or PowerShell, go to the folder you want, and use the command below to clone the repository into your computer in that location.

git clone https://github.com/ayowilfred95/Azure-k8s-postgres.git

Open the cloned repository in any text editor

I'm using Visual Studio Code, but feel free to use any text editor you prefer. Here's the structure of the project:

Project folder structure

Install project dependencies

Open the terminal in VS Code and go to the main directory of the project. Next, execute the command npm install to install all the required packages and dependencies for the project:

npm install

Since the backend application is a Node.js Express app, you use npm to install dependencies (similar to how we use maven clean install in Java).

After the dependencies are installed, open the file named "postgres.yaml". It holds all the YAML configurations required to set up your PostgreSQL database that will run in the Kubernetes cluster.

YAML Configuration

In the postgres.yaml file, there are five configurations separated by ---. It's important to use this "---" symbol when declaring different types of Kubernetes resources. If you forget to do this, you'll encounter an error.

StorageClass

The first one is the StorageClass. This YAML configuration defines a StorageClass in Kubernetes for managing storage resources.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azuredisk-premium-retain
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Retain   # Retain or Delete
volumeBindingMode: WaitForFirstConsumer   # WaitForFirstConsumer or Immediate
allowVolumeExpansion: true    # true or false
parameters:
  storageaccounttype: Premium_LRS   # Premium or Standard
  kind: Managed

Let's break down what each part means:

• kind: StorageClass: Indicates the type of Kubernetes resource being defined, which is a StorageClass. A StorageClass defines the class of storage offered by a cluster.

• apiVersion: storage.k8s.io/v1: Specifies the Kubernetes API version being used for this resource.

• metadata: name: azuredisk-premium-retain: Provides metadata for the StorageClass, including its name, which in this case is "azuredisk-premium-retain".

• provisioner: kubernetes.io/azure-disk: Specifies the provisioner responsible for provisioning storage. In this case, it's "kubernetes.io/azure-disk", indicating that Azure Disk will be used as the storage provisioner.

• reclaimPolicy: Retain: Defines the reclaim policy for the storage resources. It specifies what action should be taken when the associated persistent volume is released. Here, it's set to "Retain", meaning the volume is retained even after it's no longer used by a pod.

• volumeBindingMode: WaitForFirstConsumer: Specifies the volume binding mode, which determines when volume binding should occur. In this case, it's set to "WaitForFirstConsumer", meaning the volume will be bound when the first pod using it is created.

• allowVolumeExpansion: true: Indicates whether volume expansion is allowed. Setting it to "true" means that the size of the volume can be increased if needed.

• parameters: Contains additional parameters specific to the provisioner. Here, it specifies the storage account type as "Premium_LRS" and the kind of storage as "Managed".

Overall, this configuration sets up a StorageClass named "azuredisk-premium-retain" using Azure Disk as the provisioner, with specific policies and parameters tailored for Azure storage.

PersistentVolumeClaim

The second configuration in the postgres.yaml file is the persistent volume claim.

This YAML configuration defines a PersistentVolumeClaim (PVC) in Kubernetes, which is used to request storage resources.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azure-managed-disk-pvc
spec:
  accessModes:
  - ReadWriteOnce   # ReadWriteOnce, ReadOnlyMany or ReadWriteMany
  storageClassName: azuredisk-premium-retain
  resources:
    requests:
      storage: 4Gi

Let's break down what each part means:

• apiVersion: v1: Specifies the Kubernetes API version being used for this resource.

• kind: PersistentVolumeClaim: Indicates the type of Kubernetes resource being defined, which is a PersistentVolumeClaim. A PVC is used by pods to request storage resources.

• metadata: name: azure-managed-disk-pvc: Provides metadata for the PersistentVolumeClaim, including its name, which is "azure-managed-disk-pvc".

• spec: Describes the desired state of the PersistentVolumeClaim.

• accessModes: - ReadWriteOnce: Specifies the access mode for the volume. Here, it's set to "ReadWriteOnce", meaning the volume can be mounted as read-write by a single node at a time.

• storageClassName: azuredisk-premium-retain: Specifies the StorageClass to use for provisioning the volume. This PVC will use the StorageClass named "azuredisk-premium-retain" defined previously.

• resources: requests: storage: 4Gi: Specifies the desired storage capacity for the volume. Here, it requests 4 gigabytes (Gi) of storage.

Overall, this configuration sets up a PersistentVolumeClaim named "azure-managed-disk-pvc" requesting storage resources with specific access modes, storage class, and storage capacity.

ConfigMap

The third configuration in the postgres.yaml file is the config map. This YAML configuration defines a ConfigMap in Kubernetes, which is used to store configuration data in key-value pairs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_DB: freecodecamp
  POSTGRES_USER: freecodecamp1
  POSTGRES_PASSWORD: freecodecamp@
  PGDATA: /var/lib/postgresql/data/pgdata

Let's break down what each part means:

• apiVersion: v1: Specifies the Kubernetes API version being used for this resource.

• kind: ConfigMap: Indicates the type of Kubernetes resource being defined, which is a ConfigMap. A ConfigMap is used to store non-confidential data in key-value pairs.

• metadata: name: postgres-config: Provides metadata for the ConfigMap, including its name, which is "postgres-config".

• labels: app: postgres: Labels are key-value pairs used to organize and select resources. Here, a label "app" with the value "postgres" is applied to the ConfigMap.

• data: Contains the key-value pairs of configuration data.

• POSTGRES_DB: pisonitsha: Specifies the name of the PostgreSQL database as "pisonitsha".

• POSTGRES_USER: pisonitsha1: Specifies the username for accessing the PostgreSQL database as "pisonitsha1".

• POSTGRES_PASSWORD: pisonitsha@: Specifies the password for accessing the PostgreSQL database as "pisonitsha@".

• PGDATA: /var/lib/postgresql/data/pgdata: Specifies the location of PostgreSQL data directory as "/var/lib/postgresql/data/pgdata".

Overall, this configuration sets up a ConfigMap named "postgres-config" containing key-value pairs of configuration data, such as database name, username, password, and data directory location, which can be used by other Kubernetes resources.

Note: It's recommended to avoid hardcoding secret variables such as POSTGRES_DB, POSTGRES_PASSWORD,PGDATA and instead store them in secret files, for the sake of simplicity in this tutorial, we'll keep them hardcoded.

StatefulSet

The fourth configuration is the stateful set.This YAML configuration defines a StatefulSet in Kubernetes, which is used to manage stateful applications like databases.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  selector:
    matchLabels:
      app: postgres
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:10.4
          imagePullPolicy: "IfNotPresent"
          ports:
          - containerPort: 5432
          envFrom:
          - configMapRef:
              name: postgres-config
          volumeMounts:
          - name: azure-managed-disk-pvc
            mountPath: /var/lib/postgresql/data
      volumes:
      - name: azure-managed-disk-pvc
        persistentVolumeClaim:
          claimName: azure-managed-disk-pvc
    ```

Let's break down what each part means:

* `apiVersion: apps/v1`: Specifies the Kubernetes API version being used for this resource.
* `kind: StatefulSet`: Indicates the type of Kubernetes resource being defined, which is a `StatefulSet`. `StatefulSets` are used to manage stateful applications by providing unique **identities** and stable **network** identities to each pod.
* `metadata: name: postgres`: Provides metadata for the `StatefulSet`, including its name, which is "postgres".
* `spec`: Describes the desired state of the `StatefulSet`.
* `serviceName: postgres`: Specifies the name of the Kubernetes service that will be used to access the `StatefulSet` pods.
* `selector: matchLabels: app: postgres`: Selects the pods controlled by this `StatefulSet` based on the label "app: postgres".
* `replicas: 1`: Specifies the desired number of replicas (instances) of the StatefulSet, which is 1 in this case.
* `template`: Defines the pod template used to create pods managed by the `StatefulSet`.
* `metadata: labels: app: postgres`: Labels applied to the pods created from this template.
* `spec`: Describes the specification of the containers within the pod.
* `containers`: Specifies the containers running in the pod.
* `name: postgres`: Defines the name of the container as "postgres".
* `image: postgres:10.4`: Specifies the Docker image used for the container, which is "postgres:10.4".
* `imagePullPolicy: "IfNotPresent"`: Specifies the policy for pulling the container image, which is "IfNotPresent", meaning it will only pull the image if it's not already present on the node.
* `ports: containerPort: 5432`: Specifies the port that the PostgreSQL service inside the container is listening on.
* `envFrom: configMapRef: name: postgres-config`: Injects environment variables from a ConfigMap named "**postgres-config**" that you defined earlier.
* `volumeMounts: name: azure-managed-disk-pvc mountPath: /var/lib/postgresql/data`: Mounts a persistent volume claim named "azure-managed-disk-pvc" to the container at the specified path.
* `volumes: name: azure-managed-disk-pvc persistentVolumeClaim: claimName: azure-managed-disk-pvc`: Defines the persistent volume claim named "azure-managed-disk-pvc" to be used by the pod.

Overall, this configuration sets up a StatefulSet named "postgres" with one replica, running a PostgreSQL container with specific settings and mounted persistent storage.

### Service

The fifth configuration is the **service**. This YAML configuration defines a **Service** in Kubernetes, which is used to expose the `StatefulSet` we declared earlier as a network service.

```bash
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: LoadBalancer
  selector:
    app: postgres
  ports:
    - protocol: TCP
      name: https
      port: 5432
      targetPort: 5432

Let's break down what each part means:

• apiVersion: v1: Specifies the Kubernetes API version being used for this resource.

• kind: Service: Indicates the type of Kubernetes resource being defined, which is a Service. Services allow pods to be accessed by other pods or external users.

• metadata: name: postgres: Provides metadata for the Service, including its name, which is "postgres".

• labels: app: postgres: Labels are key-value pairs used to organize and select resources. Here, a label "app" with the value "postgres" is applied to the Service.

• spec: Describes the desired state of the Service.

• type: LoadBalancer: Specifies the type of Service, which is "LoadBalancer". This type allows the Service to be exposed externally with a cloud provider's load balancer.

• selector: app: postgres: Selects the pods controlled by the Service based on the label "app: postgres".

• ports: Specifies the ports that the Service will listen on.

• protocol: TCP: Specifies the protocol used for the port, which is TCP.

• name:https : Specifies a name for the port, which is "https".

• port: 5432: Specifies the port number on which the Service will listen, which is 5432.

• targetPort: 5432: Specifies the target port on the pods to which traffic will be forwarded, which is also 5432. This means that traffic received on port 5432 of the Service will be forwarded to port 5432 on the pods.

Overall, this configuration sets up a Service named "postgres" with a LoadBalancer type, forwarding traffic on port 5432 to pods labeled with "app: postgres".

How to Deploy YAML Resource to Azure Kubernetes Service (AKS)

You've previously connected "kubectl" with the Azure Kubernetes Service (AKS) you set up. Let's double-check it.

In your VS Code terminal, rerun the command kubectl get nodes. You'll see an output like this, though your node's value will be different.

Displaying node information running in Azure Kubernetes cluster

Next, verify the namespace you previously created by executing the command: kubectl get namespace database. Your output should resemble this:

Retrieving Namespace Information

Deploy the YAML resource

Once you've confirmed everything is set, you can deploy the YAML resource. This will establish your PostgreSQL database in the Azure Kubernetes cluster you've configured.

Run the below command in the main directory where the configuration file is located. Currently, I'm in the project's root directory (azure-k8s-postgres). To deploy the database, just execute this command below:

kubectl apply -n database -f postgres.yaml

Your output should look like this. This output confirms that all these components have been successfully created in Kubernetes.

Applying Configuration to Namespace

Execute the command below to verify that the pod is running:

kubectl get pods -n database

Your output should look like this:

Fetching Pods in Namespace

This output confirms that a pod name "postgres-0" is running in your Azure Kubernetes Cluster. But it was not the only pod you created. As I said earlier, to connect to a pod, you need what is called service. And you have declared a service resource in our configuration file which has also been deployed into your Kubernetes.

To get the status of the service, run this command:

kubectl get services -n database

Your output should look like this:

Retrieving Services in Namespace

This output displays the services in the "database" namespace, including a service named "postgres" with the type "LoadBalancer," its internal cluster IP, external IP, and port mappings. You'll utilize the external IP along with the Postgres port "5432" to connect your database with the Node.js application. Note that your external IP will differ from mine.

Node.js Application

In this section, I'll guide you through setting up your Node.js app to connect to a PostgreSQL database in your Azure Kubernetes Service.

We'll cover sending data into the database and retrieving it using Postman. Also, I'll demonstrate how to check if the data remains in the database even if the pod running PostgreSQL in the cluster is deleted.

Configure your Node.js application

Go to the database folder and open the database.js file. Replace the host with your EXTERNAL-IP obtained from the service, and leave the rest unchanged since you've already defined those variables in your config map.

Your database.js file should resemble the CodeSnap below:

CodeSnap of database.js Configuration

Run your Node.js application

In your VS Code terminal, execute this command to start the Node.js application locally:

npm start

Your output should look like this if the connection is established successfully.

Server Listening on Port 4000

If your output looks the same as mine, it indicates that you've successfully connected your Node.js application to the PostgreSQL database running in your Azure Kubernetes cluster. Congratulations! 🎉

Test the application

Testing is a fundamental principle in DevOps operations. It helps us understand the state of the application we've built before releasing it to users. Any application that doesn't pass the testing stage will not be deployed. This is a rule in DevOps.

For this tutorial, you'll be using Postman. You can download Postman here. Postman enables you to test API endpoints by receiving status responses.

Check out this post on how to use Postman to test APIs. If you want to learn more, here's a full course on the subject.

Open your Postman application

To begin using Postman, start by creating a new API request in your preferred workspace. Choose POST. POST requests add new data to the database or server. Then, paste the endpoint URL (localhost:4000/api/v1/admin/register) for your Postman test.

The below screenshot illustrates how you will create a POST request.

Postman URL Endpoint Configuration

In the body, paste the JSON data shown below inside it as shown below:

{   
    "fullName":"Azure postgres freecodecamp",
    "email":"freecodecamp@gmail.com",
    "password":"freecodecamp"
}

Postman Request Body

Once you've set up the request, just click the "Send" button to send it. Postman will then show you status codes, and the response payload as shown below.

Postman API Response

Confirm the data

To confirm that the data you sent into the database exists, make a GET request to this endpoint URL: localhost:4000/api/v1/admin/freecodecamp@gmail.com

Postman GET Request URL Endpoint

When you click send, Postman will then show you status codes and the response payload as shown below. Notice that we didn't put anything in the body because this is a GET request.

Postman GET Request by Email Retrieval Response

Delete the pod to confirm data persistence

We chose to create our PostgreSQL database using a StatefulSet to ensure that data persists even if the pod is destroyed. Let's test this by deleting the pod and checking if the data remains intact.

In your VS Code terminal, execute the command: kubectl delete pod -n database postgres-0.

This command deletes a pod named "postgres-0" in the "database" namespace from your Kubernetes cluster. Your output should look like this.

Deleting Pod in Namespace:

Pod recreation

Kubernetes has a built-in feature called replication controllers or replica sets that ensure a specified number of pod replicas are running at any given time. If a pod is deleted, Kubernetes will automatically recreate it to maintain the desired number of replicas, ensuring high availability

If you run kubectl get pods -n database, you'll notice that Kubernetes has created a new pod with the same name, "postgres-0", to replace the one that was deleted. This ensures that the application remains available and continues to function as expected.

Pod Recreated back in Namespace

Data persistence

Navigate back to Postman and make a GET request to the endpoint URL localhost:4000/api/v1/admin/freecodecamp@gmail.com.

You should get the same response as before. So under the hood, when we delete the pod, the storage disk was not deleted. The storage disk is inside the Azure disk. How do we know that? If you run this command:

kubectl get pvc -n database

you should get this output:

Persistennce Volume Claim details in namespace

This shows details about a storage called "azure-managed-disk-pvc" in your Kubernetes. It's currently in use and has 4 gigabytes of space available. It's set up to be read and written to by one system at a time. This storage is provided by a service called "azuredisk-premium-retain" that we configured earlier.

Clean up resources

In this tutorial, you created Azure resources in a resource group. If you won't need these resources later, delete the resource group from the Azure portal or run the following command in your terminal:

az group delete --name AZURE-POSTGRES-RG --yes

This command might take a minute to run.

Conclusion

We've gone on quite a journey here! You've learned how to deploy a Postgres container in Azure Kubernetes Service (AKS) and integrate it with a Node.js application.

In this tutorial, I guided you through the process of configuring Kubernetes using Azure Kubernetes Service (AKS). You learned to customize YAML files utilizing StatefulSet, Persistent Volume, and Services to deploy a PostgreSQL database on Azure Kubernetes. You also acquired PostgreSQL database credentials running within AKS to establish connectivity with a Node.js application. I then provided detailed instructions on connecting your Node.js Express app to the Postgres container within the AKS cluster.

Thank you for reading!

So let's get started.

Want to watch the video version of this tutorial? You can check out the video below:

Table Of Contents

• How You Usually Debug Node.js Applications
• How to Add a Debugger to Debug Your Code
• How to Run the Application for Debugging
• How to Access Variables During Debugging
• How to Create a Script to Debug Node.js Apps
• Quick Recap

How You Usually Debug Node.js Applications

If you want to debug a Node.js application, usually you add a console.log statement in the code that you want to debug to find out the value of any variable.

This works, but you need to keep checking the console log to see the value that you're trying to print.

And if the data printed in the console contains nested objects or if it's a lot of data, then using console.log is not feasible.

Fortunately, there's a better way.

How to Add a Debugger to Debug Your Code

Instead, you can add a debugger; statement in the code that you want to debug.

So let's say you have an Express.js API route for registering a user as shown in the below code:

// controllers/auth.js

const register = async (req, res) => {
  try {
    const { email, password } = req.body;
    const existingUser = await User.findOne({
      email,
    });
    if (existingUser) {
      return res.status(400).send('User with the provided email already exist');
    }
    // some more code
    return res.status(201).send();
  } catch (error) {
    console.log(error);
    return res
      .status(500)
      .send('Error while registering a new user. Try again later.');
  }
};

module.exports = { register };

// routes/auth.js
const { register } = require('../controllers/auth');

const Router = express.Router();

Router.post('/api/register', register);

And there's some issue while registering a user, so you want to debug the register function's code.

In that case, you can just add a debugger; statement inside the register function code like this:

const register = async (req, res) => {
  try {
    const { email, password } = req.body;
    debugger;
    const existingUser = await User.findOne({
      email,
    });
    if (existingUser) {
      return res.status(400).send('User with the provided email already exist');
    }
    // some more code
    return res.status(201).send();
  } catch (error) {
    console.log(error);
    return res
      .status(500)
      .send('Error while registering a new user. Try again later.');
  }
};

How to Run the Application for Debugging

Normally, you start your Node.js application by executing the following command:

node index.js

But instead, you can execute the following command:

node inspect index.js

Here, we just added an inspect keyword in between.

If your main application file's name is server.js, you can execute the node inspect server.js command.

Once you execute the above command, you will see the output displayed as shown below:

Debugger Attached

As you can see from the output, the debugger is attached, so now you can start debugging the code.

To do that, open the Chrome browser and enter chrome://inspect in the browser URL.

You will see the output as shown below:

Chrome Inspect Page

Since you executed the node inspect index.js command to start inspecting, you can see a new target entry displayed under the Remote Target section.

Now, if you click on the displayed blue inspect link, then you will see a new browser dev tool opened as shown below:

Debugger Paused

As you can see in the right panel in the above image, the Debugger paused message is displayed. The debugging control is at the first line of code, as you can see from the highlighted yellow line.

But you don't want to start debugging from the first line of code. Instead, you want to just debug the registration code. To do this, click on the blue triangle icon which is displayed just above the Debugger paused message as shown below:

Continuing Debugging

Now don't close this window – instead, try registering a user from the application or making an API call using Postman, so the /register route handler code that we added previously will be executed.

Registering User & Debugging Code

As you can see above, when you click on create new account button, you're automatically redirected to the code where you added the debugger; statement.

Now, you can debug the code line by line and see the values of each variable during debugging to find out and fix the issue.

How to Access Variables During Debugging

Sometimes when you mouse over any variable while debugging to see its actual value, the value might be too long (because it might be an object with many properties), so you can't see it easily by mousing over it.

In that case, while the debugger is still active, you can open the console tab and type the name of the variable whose value you want to see – as you can see in the below Gif:

Logging Variables In The Console

So that's how you can easily debug any of your Node.js application's code.

How to Create a Script to Debug Node.js Apps

If you don't want to manually type the node inspect index.js command every time in the terminal, you can create a new debug script inside the package.json file like this:

"scripts": {
    "start": "node index.js",
    "debug": "node inspect index.js",
    "dev": "nodemon index.js"
},

So now, you can execute the npm run debug command to start your application in debug mode.

Quick Recap

To debug a Node.js application, you need to follow the below steps:

• Add a debugger statement inside the code that you want to debug.
• Run the node inspect index.js or node inspect server.js command to start the application in debug mode.
• Access the URL chrome://inspect in your Chrome browser.
• Click on the inspect link under the Remote Target section.
• Click on the blue triangle icon to skip debugging if you don't want to start debugging your application from the first line of the index.js or server.js file.
• Make an API call or do something that will trigger the code where you added the debugger; statement. This way you can debug the code line by line and find out the issue.

Thanks for Reading

That's it for this tutorial. I hope you learned something new.

Want to watch the video version of this tutorial? You can check out this video.

If you want to master JavaScript, ES6+, React, and Node.js with easy-to-understand content, check out my YouTube channel. Don't forget to subscribe.

Want to stay up to date with regular content on JavaScript, React, and Node.js? Follow me on LinkedIn.

Security is critical in the field of web development, particularly when dealing with user credentials such as passwords. One security procedure that's critical in web development is password hashing.

Password hashing guarantees that plaintext passwords are difficult for attackers to find, even in a situation where a database is compromised. But not all hashing methods are created equal, and this is where bcrypt stands out.

Node.js, a popular framework for developing web applications, provides a robust ecosystem for constructing secure authentication systems. In this article, we'll look at using bcrypt in Node.js to hash passwords. We'll look at how bcrypt may be smoothly incorporated into Node.js applications to improve security and safeguard user credentials effectively.

Whether you're an experienced Node.js developer looking to strengthen your authentication practices or a beginner looking to learn the best techniques for secure password management, this article will be helpful to you. Let's have a look at how you can use bcrypt to hash passwords in Node.js.

Here's what we'll cover:

1. What is Hashing?
2. What is Bcrypt?
3. How to Install Bcrypt in Nodejs
4. How to Set Up Bcrypt in Node.js
5. How to Hash Passwords With Bcrypt
6. How to Verify Passwords With Bcrypt
7. Security Best Practices with Bcrypt
8. Conclusion

What is Hashing?

Hashing involves converting a given key or string of characters into another value. This is typically represented by a shorter, fixed-length value or key that represents the original value and facilitates the retrieval.

What is Password Hashing?

Password Hashing is a process of converting an input password into a fixed-length string of characters, typically for the purpose of securely storing and transmitting the password.

Password hash functions are designed to be one-way functions. This means it should not be computationally possible to reverse the process and get the original input password from the hashed value.

For example, suppose we want to hash a password like "password123". The password will be transformed into a fixed-length character string using a hash algorithm like bcrypt. And we'll get a hashed result once the hash function has processed our password.

The hashed output of "password123" using bcrypt, for instance, would look like this:

e234dsdom3k2kmdl3l43iwes9vjro44223m3n32kn5n2ksdo4

Now that you understand the basics of how password hashing works, it's time to dive deeper into the practical application of hashing a password using the bcrypt algorithm.

But before we proceed with that, let's learn a bit more about bcrypt so you understand its workings and installation process, as well as how to integrate it into a Node.js project.

Firstly, let's gain insight into bcrypt – what it is, how it operates, and its significance in password security. Then we'll discuss how to install bcrypt and integrate it seamlessly within a Node.js environment. This will include a detailed walkthrough on setting up bcrypt within your project and leveraging its functionalities effectively.

By the end of this article, you'll have a comprehensive understanding of bcrypt, equipped with the knowledge to securely hash passwords in your Node.js applications. So, let's embark on this journey to enhance the security of our projects through bcrypt integration.

What is bcrypt?

bcrypt is a type of cryptographic algorithm used to securely store passwords. It scrambles a user's password into a unique code. This way, even if a thief takes the database, they won't be able to recover the original passwords readily.

How Does bcrypt Work?

bcrypt works by combining hashing and a technique known as salting, which is specifically developed to make stored passwords more safe.

Here's a breakdown of the procedure:

1. Hashing: Bcrypt processes a user's password using a sophisticated mathematical function. This function converts the password to a fixed-length string of characters that appear random and meaningless. The hashed value is what is kept in the database, not the original password. Because the hashing function is one-way, reversing the hash will not produce the original password.
2. Salting: To improve security, bcrypt incorporates a random number called a salt. This salt is unique to each password and is attached to it before hashing. The combined value (password + salt) is then passed to the hashing function.

How to Install Bcrypt in Nodejs

Before you install bcrypt, you'll need to have a Node.js project already set up. If you haven't created one yet, follow these steps to create a new Node.js project:

Create a directory:

This command creates a new directory (folder) where your Node.js project will reside. It's named bcrypt-password-hash.

mkdir bcrypt-password-hash

• mkdir: This command stands for "make directory." It's used to create a new directory.
• bcrypt-password-hash: This is the name of the directory you're creating. You can choose any name you prefer for your project directory.

Change into the newly created directory:

This command navigates you into the newly created directory so that you can start working on your project within it.

cd bcrypt-password-hash

• cd: This command stands for "change directory." It's used to move from one directory to another.
• bcrypt-password-hash: This is the name of the directory you want to navigate into.

Initialize a new Node.js project:

This command initializes a new Node.js project within the directory you created. It creates a package.json file, which is used to manage dependencies and configuration for your Node.js project.

npm init -y

• npm init: This command initializes a new Node.js project using npm (Node Package Manager).
• -y: This flag automatically accepts all default values for the package.json file, so you don't have to manually provide input for each field.

After running these commands, you should have a new directory (bcrypt-password-hash) with a package.json file, indicating that you successfully created a new Node.js project. You can now go ahead and install dependencies and write code.

Create a file named index.js where you will write your code:

To create a file named index.js where you will write your code, you can use the touch command in your terminal. Here's how to do it:

touch index.js

• touch: This command is used to create a new file. (Note that you must have already installed touch on your machine to use it. If
  you haven't, you may run this command in your terminal to install
  touch: npm install touch-cli -g.)
• index.js: This is the name of the file you want to create. In this case, you're creating a JavaScript file named index.js.

After running this command, you'll have a new file named index.js in your project directory where you can write your Node.js code just like you can see in the image below:

Now that we've correctly constructed a Node.js project, we can install bcrypt in our project.

Install the required dependencies (bcrypt):

To install bcrypt, you'll use npm, the Node.js package manager. Here's the command to install bcrypt:

npm install bcrypt

• npm install: This command is used to install packages from the npm registry.
• bcrypt: This is the name of the package you want to install. bcrypt is a popular package for hashing passwords securely in Node.js.

When you run this command, npm will download and install the bcrypt package and its dependencies in the node_modules directory of your project. This directory will include all of the dependencies required for your project, including bcrypt.

How to Set Up Bcrypt in Node.js

Once Bcrypt is installed in your Node.js project, you can seamlessly integrate its functionality into your application. Here's how to proceed:

Firstly, after installing the bcrypt package using npm, make sure you import it into your Node.js application index.js file to utilize its features effectively.

Here's how to do it:

const bcrypt = require('bcrypt');

This line of code ensures that the bcrypt package is accessible within your application, allowing you to leverage its powerful capabilities for secure password hashing and verification.

With bcrypt integrated into your project, you can enhance the security of user authentication and data protection.

bcrypt provides two primary functions for password hashing and comparison:

1. bcrypt.hash(): This function is used to generate a hash of a plaintext password. It takes the plaintext password and a salt factor (optional) as input parameters and returns the hashed password asynchronously.
2. bcrypt.compare(): This function is used to compare a plaintext password with its hashed counterpart. It takes the plaintext password and the hashed password as input parameters and returns a boolean value indicating whether the passwords match.

How to Hash Passwords With Bcrypt

Having delved into the significance of password hashing, as well as the concepts of hash and salt, let's put theory into practice within our index.js file.

How to Generate a Salt and Hash the Password

As we've learned, a key aspect of secure password hashing involves incorporating a unique salt into the hashing process. bcrypt simplifies this by handling salt generation and password hashing seamlessly.

To begin, we require the bcrypt module in our Node.js application:

const bcrypt = require('bcrypt');

To ensure the strength of our password hashes, we determine the number of salt rounds. This value dictates the computational cost of hashing and, consequently, the level of security:

const saltRounds = 10; // Typically a value between 10 and 12

With our configuration established, we can generate a salt asynchronously using the bcrypt.genSalt() function. This salt will be unique for each password hash, enhancing security:

bcrypt.genSalt(saltRounds, (err, salt) => {
if (err) {
    // Handle error
    return;
}

// Salt generation successful, proceed to hash the password
});

Once the salt is generated, we combine it with the user's password to compute the hash using the bcrypt.hash() function. This results in a securely hashed password ready for storage:

const userPassword = 'user_password'; // Replace with the actual password
bcrypt.hash(userPassword, salt, (err, hash) => {
    if (err) {
        // Handle error
        return;
    }

// Hashing successful, 'hash' contains the hashed password
console.log('Hashed password:', hash);
});

By leveraging bcrypt for password hashing in our Node.js application, we ensure the robust security of user credentials. The incorporation of unique salts for each password hash, coupled with the computational complexity of bcrypt, fortifies our defense against unauthorized access and malicious attacks.

In the next section, we'll explore how to verify passwords and discuss best practices for securely managing hashed passwords.

How to Verify Passwords With Bcrypt

Now that we've covered the process of hashing passwords using bcrypt within our Node.js application, let's shift our focus to verifying passwords during user authentication.

In this section, we'll explore how bcrypt facilitates password verification, ensuring a secure and seamless authentication process.

How to Retrieve a Hashed Password from the Database

Before we can verify a user's password, we need to retrieve the hashed password associated with the user's account from the database.

Assuming you have a user authentication system in place, you'll typically query the database to fetch the hashed password based on the user's username or email.

Once you have retrieved the hashed password from the database, you're ready to proceed with the password verification process.

How to Verify Passwords

To verify a password using bcrypt, use the bcrypt.compare() function. This function compares a plaintext password provided by the user during login with the hashed password stored in the database.

Here's how you can implement password verification using bcrypt in your Node.js application:

const storedHashedPassword = 'hashed_password_from_database';
const userInputPassword = 'password_attempt_from_user';

bcrypt.compare(userInputPassword, storedHashedPassword, (err, result) => {
    if (err) {
        // Handle error
        console.error('Error comparing passwords:', err);
        return;
    }

if (result) {
    // Passwords match, authentication successful
    console.log('Passwords match! User authenticated.');
} else {
    // Passwords don't match, authentication failed
    console.log('Passwords do not match! Authentication failed.');
}
});

In this code snippet, storedHashedPassword represents the hashed password retrieved from the database, while userInputPassword is the plaintext password provided by the user during login. The bcrypt.compare() function compares these two passwords and returns a boolean value indicating whether they match.

In the next section, we'll discuss best practices for securely managing hashed passwords, including considerations for password storage and handling.

Security Best Practices with bcrypt

Now that we've discussed the principles of password hashing and verification with bcrypt, let's look at some important security best practices to ensure the integrity of our authentication system.

Robust Password Guidelines

Encourage users to create strong and complex passwords that are resistant to dictionary attacks. Provide guidance on password length, the inclusion of alphanumeric characters, symbols, and the avoidance of common patterns.

Salting

Always use a unique salt for each password hash. This prevents attackers from using precomputed rainbow tables to crack passwords. bcrypt automatically handles salt generation, ensuring that each hash is unique.

Adaptive Hashing

bcrypt employs adaptive hashing, allowing developers to adjust the computational cost of hashing over time. Periodically increase the number of hashing rounds to keep pace with advancements in hardware and computational power.

Secure Storage

Store hashed passwords securely in your database. Ensure that access controls are in place to prevent unauthorized access to user credentials. Avoid storing plaintext passwords or using reversible encryption algorithms.

Error Handling

Implement proper error handling mechanisms when working with bcrypt functions. Handle errors gracefully and avoid leaking sensitive information that could aid attackers in exploiting vulnerabilities.

Conclusion

In conclusion, we have explored the essential aspects of password security and the role of bcrypt in safeguarding user credentials within Node.js applications. From understanding the fundamentals of password hashing and salting to implementing secure authentication mechanisms, we have covered a wide array of topics aimed at enhancing the security posture of our applications.

By leveraging bcrypt for password hashing and verification, we ensure that sensitive user data remains protected against unauthorized access and malicious attacks. bcrypt's robust algorithm, combined with adaptive hashing and salt generation, provides a reliable defense mechanism against common password-based vulnerabilities.

We also discussed security best practices, including strong password policies, secure storage practices, and error handling. By adhering to these best practices and staying vigilant against evolving threats, we can create a secure authentication system that instills confidence in our users and upholds the integrity of our applications.

Let's continue to prioritize security and strive for excellence in our pursuit of building robust and trustworthy applications.

Thank you for joining me on this exploration of password security with bcrypt. Together, we can create a safer digital environment for all users.

Happy Coding!

An API gateway serves as a central entry point for all client requests. It provides various functionalities such as routing, load balancing, authentication, and rate limiting.

In this article, we’ll explore how you can build out a custom API gateway using Node.js.

Here's what we'll cover:

1. What is an API Gateway?
2. Security in API Gateways
3. How to Build a Custom API Gateway with Node.js
4. Conclusion

Prerequisites

This is a beginner's guide that should be relatively easy to follow. But to fully understand and get the most out of it, basic knowledge of Node.js such as installation, setting up, and spinning up a server is vital.

Without further ado, let's dig in!

What is an API Gateway?

API gateways act as intermediaries between clients and back-end services in a Microservices architecture. They abstract the complexity of the underlying services and expose a unified API to clients.

By consolidating multiple service endpoints into a single entry point, API gateways simplify client-side code and improve the overall scalability and performance of the system.

Compared to other popular API gateway solutions like Kong, AWS API Gateway, and Tyke, building a custom API gateway using Node.js offers flexibility and customization options tailored to your specific project requirements.

To get a little more understanding of what an API gateway is, I recommend you check out this article if you haven’t.

Benefits of Using an API Gateway:

• Improved scalability and performance through request routing and load balancing: API gateways facilitate request routing and load balancing, distributing incoming traffic across multiple backend services to ensure optimal performance and scalability.
• Simplified client-side code by providing a unified API endpoint: With a unified API endpoint provided by the API gateway, clients can interact with multiple services seamlessly, reducing complexity and improving the maintainability of client-side code.
• Enhanced Security: API gateways offer robust security features such as authentication, authorization, and rate limiting, protecting backend services from unauthorized access and potential security threats.

Security in API Gateways

Security is paramount in modern software development, especially when dealing with distributed systems and microservices. API gateways play a crucial role in enforcing security measures to safeguard sensitive data and prevent unauthorized access to APIs.

Common security features implemented in API gateways include:

• JWT Authentication: Verifying the identity of clients using JSON Web Tokens (JWT) to ensure secure communication between clients and backend services.
• OAuth2 Integration: Providing secure access control and authorization mechanisms using OAuth2 protocols to authenticate and authorize client requests.
• SSL Termination: Encrypting traffic between clients and the API gateway using SSL/TLS protocols to protect data in transit from eavesdropping and tampering.

Now you should have a general overview of what an API gateway is and why it's important.

In the next section, we will delve into the process of building a custom API gateway using Node.js. I'll demonstrate how to implement security features using the http-proxy-middleware package.

How to Build a Custom API Gateway with Node.js

As I've already discussed, we'll be using Node.js for this tutorial. In my opinion, Node.js is by far the easiest and most popular web framework. Anyone can learn how to use it.

For this guide, I assume you already know or have a basic understanding of Node.js and how to set up a server.

Getting Started – Installations and Setup

To get started, create a new folder called “API-gateway” entirely outside your front-end or your back-end code. Once the folder is created, open it on your terminal and run npm init -y. This will set up npm and then you’re ready to roll things out!

We’ll be using a couple of NPM packages, and it’s best to install them now. The most important one is the http-proxy-middleware. This middleware or package is what will route our requests from one endpoint (www.domain.com/auth ) to each corresponding endpoint (www.externaldomain.com/v1/bla/auth, www.externaldomain.com/v1/bla/projects ) as defined in our microservices.

To install the http-proxy-middleware, simply run npm i http-proxy-middleware on the root folder on your terminal. If it's installed, you’re good to go.

Next, we’ll need the remaining packages. Simply run npm install express cors helmet morgan on your terminal in the root folder of the API gateway.

The above command installs the following:

• Express: our Node.js library for creating our server and running our code
• Cors: middleware to manage and control any cross-origin requests
• Helmet: yet another middleware for securing our HTTP response headers
• Morgan: a logging tool we can use to track both success and error logs

Lastly, install Nodemon. This is a tool that spins up your server whenever you save a file using npm install --save-dev nodemon.

Now, go to your package.js file and update the scripts section. It should look like this:

"scripts": {
 "start": "node index.js",
 "dev": "nodemon index.js",
 "test": "echo \"Error: no test specified\" && exit 1"
},

To finally start testing things out, create a new file called index.js in that same api-gateway folder.

If you get everything right, you should have the following files:

An image showing the file structure of our code base

Putting it All Together

A good code practice is to break things down as much as possible into smaller components.

But for this guide, we’re going to break that rule and put all the code into that one index.js file we created from the steps above. We'll be doing it this way because having too many files and an overly complex set up here might be confusing, especially while you're learning how things work.

First thing first, open the index.js file you’ve created and paste the following code into it:

const express = require("express");
const cors = require("cors");
const helmet = require("helmet");
const morgan = require("morgan");
const { createProxyMiddleware } = require("http-proxy-middleware");

In the code above we're just importing packages.

Next up, initialize and set up the imported packages like this:

// Create an instance of Express app
const app = express();


// Middleware setup
app.use(cors()); // Enable CORS
app.use(helmet()); // Add security headers
app.use(morgan("combined")); // Log HTTP requests
app.disable("x-powered-by"); // Hide Express server information

Remember that an API gateway is a single source of truth for all your services or external URLs. This means you must have other services or URLs you want to forward the requests to.

Assuming you already have your other services running either locally or deployed, let’s move to the next section of the code.

// Define routes and corresponding microservices
const services = [
 {
   route: "/auth",
   target: "https://your-deployed-service.herokuapp.com/auth",
 },
 {
   route: "/users",
   target: "https://your-deployed-service.herokuapp.com/users/",
 },
 {
   route: "/chats",
   target: "https://your-deployed-service.herokuapp.com/chats/",
 },
 {
   route: "/payment",
   target: "https://your-deployed-service.herokuapp.com/payment/",
 },
 // Add more services as needed either deployed or locally.
];

In the above code, we created a services array list and defined objects each containing routes (where we’ll make requests to) and targets (where the requests will be forwarded to).

Make sure to update the routes and targets to suit your needs.

Can you guess what’s next…?

Well, it’s finally time to create the simple logic to forward the requests to our target URL, setting up a rate limit and timeouts. And do you know what’s coming next? A code sample, lol:

// Define rate limit constants
const rateLimit = 20; // Max requests per minute
const interval = 60 * 1000; // Time window in milliseconds (1 minute)

// Object to store request counts for each IP address
const requestCounts = {};

// Reset request count for each IP address every 'interval' milliseconds
setInterval(() => {
  Object.keys(requestCounts).forEach((ip) => {
    requestCounts[ip] = 0; // Reset request count for each IP address
  });
}, interval);

// Middleware function for rate limiting and timeout handling
function rateLimitAndTimeout(req, res, next) {
  const ip = req.ip; // Get client IP address

  // Update request count for the current IP
  requestCounts[ip] = (requestCounts[ip] || 0) + 1;

  // Check if request count exceeds the rate limit
  if (requestCounts[ip] > rateLimit) {
    // Respond with a 429 Too Many Requests status code
    return res.status(429).json({
      code: 429,
      status: "Error",
      message: "Rate limit exceeded.",
      data: null,
    });
  }

  // Set timeout for each request (example: 10 seconds)
  req.setTimeout(15000, () => {
    // Handle timeout error
    res.status(504).json({
      code: 504,
      status: "Error",
      message: "Gateway timeout.",
      data: null,
    });
    req.abort(); // Abort the request
  });

  next(); // Continue to the next middleware
}

// Apply the rate limit and timeout middleware to the proxy
app.use(rateLimitAndTimeout);

// Set up proxy middleware for each microservice
services.forEach(({ route, target }) => {
  // Proxy options
  const proxyOptions = {
    target,
    changeOrigin: true,
    pathRewrite: {
      [`^${route}`]: "",
    },
  };

  // Apply rate limiting and timeout middleware before proxying
  app.use(route, rateLimitAndTimeout, createProxyMiddleware(proxyOptions));
});

I added a bunch of good code comments to help you understand what's going on.

Congratulations if you know what’s happening above. If you don’t, you can read about the http-proxy-middleware package.

But let’s get serious, we’re not done yet.

The above code still won’t work, as we need one more thing: writing a function to start the server when called upon.

Add the following code sample to the bottom of the index.js after all of the code you’ve added above:

// Define port for Express server
const PORT = process.env.PORT || 5000;


// Start Express server
app.listen(PORT, () => {
 console.log(`Gateway is running on port ${PORT}`);
});

With that, when you run npm run dev, it spins up your server and you should be able to test this out using tools like Postman or any other tool you use to test APIs.

Now, before we go, let’s try to make this a little bit spicy!

Let’s add a 404 function to track and return a nice 404 message to a user if they navigate or send a request to a URL that doesn’t exist.

So on our services array defined above, we don’t have any routes defined for products. This means that if a user sends a request to /product, they’d get a server error because the request can’t be handled.

To tell the user that the URL is not found, we can add the following code sample just before we define the port and listen to it:

// Handler for route-not-found
app.use((_req, res) => {
 res.status(404).json({
   code: 404,
   status: "Error",
   message: "Route not found.",
   data: null,
 });
});


// Define port for Express server

Conclusion

Building a custom API gateway with Node.js offers developers a flexible and customizable solution for managing, routing, and securing API calls in a microservices architecture.

Throughout this tutorial, we've explored the fundamental concepts of API gateways, including their role in simplifying client-side code, improving scalability and performance, and enhancing security.

By leveraging the power of Node.js and the http-proxy-middleware package, we've demonstrated how to implement a basic API gateway that proxies requests to multiple backend services. We've also enhanced our gateway with essential features such as rate limiting and timeouts to ensure reliable and secure communication between clients and services.

As you continue to explore the world of microservices and distributed systems, remember that API gateways play a crucial role in orchestrating communication and enforcing security measures. Whether you choose to build a custom solution or utilize existing gateway platforms, understanding the principles and best practices outlined in this tutorial will empower you to architect robust and scalable systems.

I encourage you to experiment with the code samples provided and explore further customization options to suit your project's unique requirements. The complete source code for this tutorial can be found here: https://github.com/irorochad/api-gateway.

Thank you for joining me on this journey to explore the intricacies of API gateways with Node.js. Happy coding!

Observability involves collecting and analyzing data from sources within a system to monitor its performance and address problems effectively.

Why is Observability Useful?

1. Detecting and Troubleshooting Problems: Observability plays a role in identifying and diagnosing issues within a system. When something goes wrong, having access to data helps pinpoint the cause and resolve problems more quickly.

2. Optimizing Performance: Through monitoring metrics and performance indicators, observability helps in optimizing the performance of your system. This includes identifying bottlenecks, improving resource utilization, and ensuring operation.

3. Planning for Future Capacity: Understanding how your system behaves over time is vital for planning capacity requirements. Observability data can reveal trends, peak usage periods, and resource needs, helping your decisions regarding scaling.

4. Enhancing User Experience: By observing user interactions with your system through logs and metrics, you can improve the user experience. It assists in recognizing patterns, preferences, and potential areas that can be enhanced for user satisfaction.

Why Should I Use OpenTelementary?

Observability is essential for ensuring the reliability and availability of your Node.js applications. But manually instrumenting your code to collect and export telemetry data, such as traces, metrics, and logs, can become very stressful.

Manual instrumentation is very tedious, error-prone, and inconsistent. It can also introduce additional overhead and complexity to your application logic.

In this guide, you will learn how to use OpenTelemetry’s auto-instrumentation to help you achieve effortless Node.js monitoring.

Prerequisites

Before you go through this guide, make sure you have the following:

• A Node.js application

• A Datadog account and an API key. If you don't have one, you can sign up here to get one.

• A Backend service. You can use a backend service like Zepkin or Jaeger to store and analyze trace data. For this guide, we'll be using Jaeger.

• Some basic knowledge of Linux commands. You should be familiar with using the command line and editing configuration files.

Prepare Your Application

In this guide, you will be using a Node.js application that has two services that transfer data between themselves. You will use OpenTelemetry’s Node.js client library to send trace data to an OpenTelementay collector.

Firstly, clone the Repo Locally:

$ git clone https://github.com/<github-account>/nodejs-example.git

Then run the application:

npm install

Go to the directory of the first service using this command:

$ cd <ServiceA>

And start the first service.

$ node index.js

Then go to the directory of the second service

$ cd <ServiceB>

And start the second service.

$ node index.js

Open Service A, in this case port 5555, and input some information. Then repeat the same for Service B.

How to Set Up OpenTelementary

After starting the services, it's time to install the OpenTelementary modules you'll need for auto-instrumentation.

Here are what we need to install:

$ npm install --save @opentelemetry/api

$ npm install --save @opentelemetry/instrumentation

$ npm install --save @opentelemetry/tracing

$ npm install --save @opentelemetry/exporter-trace-otlp-http

$ npm install --save @opentelemetry/resources

$ npm install --save @opentelemetry/semantic-conventions

$ npm install --save @opentelemetry/auto-instrumentations-node

$ npm install --save @opentelemetry/sdk-node

$ npm install --save @opentelemetry/exporter-jaeger

Here's break down of what each module does:

• @opentelemetry/api: This module provides the OpenTelemetry API for Node.js.

• @opentelemetry/instrumentation: The instrumentation libraries provide automatic instrumentation for your Node.js application. They automatically capture telemetry data without requiring manual code modifications.

• @opentelemetry/tracing: This module contains the core tracing functionality for OpenTelemetry in your Node.js application. It includes the Tracer and Span interfaces, which are important for capturing and representing distributed traces within your applications.

• @opentelemetry/exporter-trace-otlp-http: This exporter module enables sending trace data to an OpenTelemetry Protocol (OTLP) compatible backend over HTTP.

• @opentelemetry/resources: This module provides a way to define and manage resources associated with traces.

• @opentelemetry/semantic-conventions: This module defines a set of semantic conventions for tracing. It establishes a common set of attribute keys and value formats to ensure consistency in how telemetry data is represented and interpreted.

• @opentelemetry/auto-instrumentations-node: This module simplifies the process of instrumenting your application by automatically applying instrumentation to supported libraries.

• @opentelemetry/sdk-node: The Software Development Kit (SDK) for Node.js provides the implementation of the OpenTelemetry API.

• @opentelemetry/exporter-jaeger: This exporter module allows exporting trace data to Jaeger. Jaeger provides a user-friendly interface for monitoring and analyzing trace data.

Configure the Node.js Application

Next, add a Node.js SDk tracer to handle the instantiation and shutdown of the tracing.

To add the tracer, create a file tracer.js:

$ nano tracer.js

Then add the following code to the file:

"use strict";

const {
    BasicTracerProvider,
    SimpleSpanProcessor,
} = require("@opentelemetry/tracing");
// Import the JaegerExporter
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");
const { Resource } = require("@opentelemetry/resources");
const {
    SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");

const opentelemetry = require("@opentelemetry/sdk-node");
const {
    getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");

// Create a new instance of JaegerExporter with the options
const exporter = new JaegerExporter({
    serviceName: "YOUR-SERVICE-NAME",
    host: "localhost", // optional, can be set by OTEL_EXPORTER_JAEGER_AGENT_HOST
    port: 16686 // optional
});

const provider = new BasicTracerProvider({
    resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]:
            "YOUR-SERVICE-NAME",
    }),
});
// Add the JaegerExporter to the span processor
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

provider.register();
const sdk = new opentelemetry.NodeSDK({
    traceExporter: exporter,
    instrumentations: [getNodeAutoInstrumentations()],
});

sdk
    .start()
    .then(() => {
        console.log("Tracing initialized");
    })
    .catch((error) => console.log("Error initializing tracing", error));

process.on("SIGTERM", () => {
    sdk
        .shutdown()
        .then(() => console.log("Tracing terminated"))
        .catch((error) => console.log("Error terminating tracing", error))
        .finally(() => process.exit(0));

Here is a simple breakdown of the code:

• The code starts by importing the modules BasicTracerProvider and SimpleSpanProcessor for setting up tracing from the OpenTelemetry library

• It then imports the JaegerExporter module for exporting trace data to Jaeger.

• The code creates a new instance of the JaegerExporter, specifying the service name, host, and port.

• It then creates a BasicTracerProvider and adds the JaegerExporter to the span processor using SimpleSpanProcessor.

• The provider is registered, setting it as the default provider for the application.

• An OpenTelemetry SDK instance is created, configuring it with the JaegerExporter and enabling auto-instrumentations for Node.js.

• The OpenTelemetry SDK is started, initializing tracing.

• A handler for the SIGTERM signal is set up to shut down tracing when the application is terminated.

• The code then configures the trace provider with a trace exporter. To verify the instrumentation, ConsoleSpanExporter is used to print some of the tracer output to the console.

How to Set Up OpenTelemetry to Export the Traces

Next, you'll need to write the configurations to collect and export data in the OpenTelemetry Collector.

Create a file config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  datadog:
    api: # Replace with your Datadog API key
      key: "<YOUR_DATADOG_API_KEY>"
    # Optional:
    #   - endpoint: https://app.datadoghq.eu  # For EU region

processors:
  batch:

extensions:
  pprof:
    endpoint: :1777
  zpages:
    endpoint: :55679
  health_check:

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog]

The configuration sets up OpenTelemetry with the OTLP (OpenTelemetry Protocol) receiver and the Datadog exporter. Here’s a break down of the code:

• receivers: Specifies the components that receive the telemetry data. In this case, it includes the OTLP receiver, which supports both gRPC and HTTP protocols.

• exporters: Defines the components responsible for exporting telemetry data. Here, it configures the Datadog exporter, providing the Datadog API key. Additionally, an optional endpoint is provided for using Datadog's EU region.

• processors: Specifies the data processing components. In this case, the batch processor is used to batch and send data in larger chunks for efficiency.

• extensions: Defines additional components that extend the functionality. Here, it includes extensions for pprof (profiling data), zpages (debugging pages), and a health check extension.

• service: Configures the overall service behavior, including the extensions and pipelines. The extensions section lists the extensions to be used, and the pipelines section configures the telemetry data pipeline. Here, the traces pipeline includes the OTLP receiver, the batch processor, and the Datadog exporter.

This code is configured by the collector with the Datadog exporter to send the traces to Datadog Distributed Tracing services. However, there are other distributed tracing services that you can use like New Relic, Logzio, and Zipkin.

How to Start the Application

After correctly setting up auto-instrumentation, start the application again to test and verify the tracing configuration.

Begin by starting the OpenTelemetry Collector:

./otelcontribcol_darwin_amd64 --config ./config.yaml

The collector will start on port 4317.

Next, go to the directory of the first service:

$ cd <ServiceA>

Then start the first service with the “--require './tracer.js'” parameter to enable the application instrumentation.

$ node --require './tracer.js' index.js

Repeat this to start the second service.

Using a browser like Chrome, go to the endpoints of your two applications' services, add some data, and send some requests to test the tracing configuration.

Once the requests are made, these traces are picked up by the collector, which then dispatches them to the distributed tracing backend specified by the exporter configuration in the collector's configuration file.

It's worth noting that our tracer not only facilitates the transmission of traces to the designated backend, but also exports them to the console at the same time.

This dual functionality allows for real-time visibility into the traces being generated and sent, helping us in the monitoring and debugging processes.

Now, let’s use Jaeger UI to monitor the traces:

Start Jaeger with the following command:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.32

Using a browser, start Jaeger UI at the http://localhost:16686/ endpoint.

There you have it! The initiation of the trace starting from the inception point of one service, navigating through a sequence of operations.

This path is created as the service starts its operations, resulting in the set up of the other service to fulfill the original request you initiated earlier.

The trace provides a visual narrative of what happens between these services, offering insights into each step of the process.

How Can You Use Observability Data?

1. Monitoring Metrics: Keep an eye on key metrics such as response times, error rates, and resource usage. Sudden spikes or anomalies can indicate issues that require attention.

2. Logging: Log data provides detailed information about events and actions within a system. Analyzing logs helps in understanding the sequence of activities and tracing the steps leading to an issue.

3. Tracing: Tracing involves tracking the flow of requests or transactions across different components of a system. This helps in understanding the journey of a request and identifying any bottlenecks or delays.

4. Alerting: Set up alerts based on specific conditions or thresholds. When certain metrics exceed predefined limits, alerts can notify you in real-time, allowing for immediate action.

5. Visualization: Use graphical representations and dashboards to visualize complex data. This makes it easier to identify patterns, trends, and correlations in the observability data.

Observability, when implemented effectively, empowers teams to proactively manage and improve the performance, reliability, and user experience of their systems. It's a crucial aspect of modern software development and operations.

Conclusion

In this guide you learned how to auto-instrument Node.js applications with little code by:

• Installing and configuring the OpenTelemetry Node.js SDK and the auto-instrumentation package

• Enabling automatic tracing and metrics collection for your Node.js applications and their dependencies

• Exporting to visualize your telemetry data on a backend, Jaeger.

Using OpenTelemetry’s auto-instrumentation can help you gain valuable insights into the performance and behavior of your Node.js applications without having to manually instrument each library or framework.

According to its creator, Zoltan Kochan, pnpm "allows you to save gigabytes of disk space."

Many popular projects including Next.js, Vite, Svelte, and even freeCodeCamp use pnpm. So now is a great time try out this tool if you haven't yet. I'm sure your time will not be wasted.

In this article, I won't go into details of why pnpm is faster and more efficient than npm. You can check out the official documentation if you want to know more about that.

The goal of this article is to quickly get you started with pnpm so you can perform your day to day tasks that you previously did with npm or yarn. Grab your favorite cup of tea or coffee ☕️, and let's dive right in! 🚀

How to Install pnpm

I assume you already have a modern version of Node.js installed on your machine. These modern versions come with a command called corepack. It let's you manage your Node package managers.

Yes you read that right! It's an experimental feature of Node but it works pretty well.

So to start using it, you first need to enable it by entering the following command from your terminal, which has the effect of installing pnpm (and also yarn) on your system:

corepack enable

It's that simple. Now if you run pnpm --version you will see the version you have just installed. But this might not be the latest version of pnpm. If this is the case, you can install the latest version of pnpm using this command:

corepack prepare pnpm@latest --activate

Keep in mind that there are many ways to install pnpm on your system, and you can read about all of them in the installation docs. My favorite is the corepack formula I've shown above.

How to Configure your Shell for Efficiency (Optional)

Well, you now have pnpm installed. But the default command line experience can be improved to save you some effort.

Note that this section is optional. If you want you can skip to the next section. But if you are serious about setting it up so that the CLI experience is pleasant, let's learn how to do it.

pnpm is Hard to Type – So Set Up an Alias

If you find pnpm hard to type like I do, you can set up an alias to to save you some effort. If you're on Linux or MacOS, just put the following in your shell config (.bashrc, .zshrc, or config.fish):

alias pn=pnpm

If you want to set up your alias in Powershell (Windows) you can see this doc.

How to Setup Tab-Completion

There are two ways you can do this in pnpm. Both have their pros and cons.

First I will share with you my favorite method. It's a shell plugin called pnpm-shell-completion and is available for zsh, fish shell, and Powershell core. It only covers the most commonly used commands. If you are Arch Linux and zsh user, you can install it with any AUR helper. For example, if you use yay, run the following command to install it:

yay -S pnpm-shell-completion

Then add the following line in your .zshrc file to load it:

source /usr/share/zsh/plugins/pnpm-shell-completion/pnpm-shell-completion.zsh

Now it should work. If you use any other supported shell, follow the plugin's doc to learn how to install it.

The second method comes built-in with pnpm. To setup this style of auto-completion, run the following command:

pnpm install-completion

And then follow the steps it gives you. This method covers more commands than the first approach. But it has some limitations – for example it can't auto-complete your package.json scripts. It also, for example, can't auto complete any dependency name that you want to uninstall.

How to Use pnpm

Now, you should have pnpm installed with an alias and tab-completion. No more delay – let's see how to use pnpm.

How to Initialize a New Project using pnpm

To get the default package.json in the current directory, run the following command:

pnpm init

Unlike npm, it will not create it interactively and you don't need to specify the -y flag for this.

How to Install a Package

To install a package as a dependency, the syntax is:

pnpm add <pkg>

To install a package as a dev dependency, you have pass the -D (or --save-dev) flag:

pnpm add -D <pkg>

To install a package globally, use the -g flag:

pnpm add -g <pkg>

How to Install All Dependencies

Suppose you cloned a project from GitHub. It does have a package.json file but no node_modules (you should not track node_modules with Git). Now to install all the dependencies in that package.json, the command is very similar to npm:

pnpm install

or

pnpm i

How to Run a package.json Script

This process is also very similar to npm. The explicit way to do it is to use the run command. If you have a script named build, you can execute it with this command:

pnpm run build

You can also use pnpm build to do the same thing. This is a shorthand format that can do other things as well. We'll learn more about shorthand very soon in this article.

How to Run Commands that Come with Dependencies

You can run commands that come with dependencies using pnpm exec.

When you install a package, if it has commands specified by the bin field in its package.json, you will get an executable of the same name in your node_modules/.bin directory. Its purpose to execute the corresponding file.

pnpm exec prepends ./node_modules/.bin to the PATH (that is, PATH=./node_modules/.bin:$PATH) and then executes the given command.

The following is an example that shows installing typescript as a dev dependency and then running the tsc command to create a tsconfig.json file:

pnpm add -D typescript
pnpm exec tsc --init

Similar to the pnpm run command, you can also omit exec and just use pnpm tsc to do the same thing. This works when you don't have a conflicting tsc script in your package.json. In the next section we will take a close look at this shorthand syntax.

Note that since pnpm exec has access to all commands resolved by the paths specified in PATH, you may have access to many system commands for example rm, ls, and so on.

How pnpm <command> Works

pnpm <command> works like this:

• If <command> is a pnpm command (that is add, install and so on), execute that command.
• Else if <command> is a script found in package.json, execute pnpm run <command>.
• Else execute pnpm exec <command>.

So pnpm <command> serves as a convenient shortcut where pnpm exec and pnpm run are explicit commands without fallback.

How to Update Packages

To update packages to their latest versions based on the specified range in package.json, run this command:

pnpm up

To update all dependencies to their latest versions, ignoring ranges specified in package.json, run this:

pnpm up --latest

How to Remove a Package

To remove a package from both node_modules and your package.json, you can use pnpm rm. For example if you installed express, you can remove it using:

pnpm rm express

To remove a globally installed package, use the -g flag. Below is an example of removing the globally installed package nodemon:

pnpm rm -g nodemon

Is There an npx Alternative?

Yes – it's the pnpm dlx command. It's very similar to npx. It downloads the specified package from the registry without installing it as a dependency and then runs whatever default command binary it exposes.

For example, you can run the command that cowsay package exposes like below to print ASCII art of a cow saying a string that you pass:

pnpm dlx cowsay hi freeCodeCamp

Now you might be wondering, if a package exposes multiple command binaries, what command pnpm dlx chooses as the default? Or how can you explicitly specify a command binary?

Let's see how the default command binary is determined first:

• If the bin field of package.json has only one entry, then that is used.
• Else if there is a command name in the bin field of package.json that matches the package name, ignoring the scope part if any, then that command is used.
• Else pnpm can't determine the default command and throws an error with a helpful error message that most likely will answer the second question.

To explicitly specify a particular command, you will need to install the package first using the --package option and specify that command after dlx.

For example the package typescript exposes to command binaries tsc and tsserver. Now if you want to run tsc --init to create a tsconfig.json file without having typescript in your node_modules or package.json, you can use pnpm dlx like below:

pnpm --package=typescript dlx tsc --init

Conclusion

In this tutorial, you've learned what pnpm is and how to install it. We've also covered several common pnpm commands that you will most likely need on a daily basis.

I hope this article helped you get up and running with pnpm. Check out the documentation of pnpm to learn more about it.

If you want you can follow me on LinkedIn and Twitter where I share useful coding related things.

Happy coding!

In this handbook, I'm going to introduce you to the MERN stack, a widely-used technology stack embraced by many leading companies. I'll guide you through 7 essential steps to start learning these technologies on your own.

By the time you finish reading, you'll have a solid understanding of what the MERN stack entails, its component technologies, and various resources for learning. You'll also have 10 project ideas that you can develop and showcase in your portfolio.

Table of Contents

1. What is the MERN stack?
2. The MERN Stack Roadmap
   • STEP 1: Learn the right amount of HTML, JavaScript, and CSS
   • STEP 2: Get familiar with React
   • STEP 3: Understand REST API's and how a backend server works using Express/Node
   • STEP 4: Storing data with MongoDB and Mongoose
   • STEP 5: Writing tests
   • STEP 6: Using Git
   • STEP 7: Deployments
3. Top Resources to learn the MERN Stack
4. 10 project ideas you can try today
5. Wrapping up the MERN stack journey

What is the MERN stack?

The MERN stack, comprising MongoDB, Express.js, React, and Node.js, is a cohesive set of technologies used for building efficient and scalable web applications.

Its popularity stems from the seamless integration of each component: MongoDB's flexible data handling, Express.js's efficient server-side networking, React's dynamic user interfaces, and Node.js's powerful back-end runtime environment.

For beginners, the MERN stack is a smart choice because it uses JavaScript across all layers, simplifying the learning curve. This uniformity, coupled with a strong community and ample learning resources, makes it an accessible and practical toolkit for anyone looking to dive into full-stack development.

The MERN stack is also heavily utilized in the industry, favored by startups and large enterprises alike for its efficiency and the robust, modern web applications it can produce. This industry adoption not only validates its effectiveness but also opens up numerous career opportunities for those skilled in these technologies.

Let's look at a brief overview of what each part of the MERN Stack looks like:

Frontend (React)

The frontend of a website is like the dining area of a restaurant. It's everything you see and interact with directly on a website – the layout, design, buttons, and text.

Example: When you visit a website and see a beautiful homepage, interact with menus, or fill out forms, you're experiencing the frontend.

Backend (Node.js)

The backend is like the kitchen in a restaurant. It's where all the behind-the-scenes work happens. It includes the server, applications, and databases that work together to process the information you see on the frontend.

Example: When you order food (submit a form on the website), the kitchen (backend) processes the order (the data) and prepares your meal (the information or service you requested).

Database (MongoDB)

A database in web development is similar to a restaurant’s pantry or storage where all the ingredients (data) are kept. It stores and manages all the information needed by the website, like user profiles, content, and other data.

Example: In an online store, the database stores product information, prices, user reviews, and customer details.

REST APIs (Express)

REST APIs are like the waiters in a restaurant. They are the messengers or go-betweens for the frontend and backend. They take requests (like orders) from the frontend (customer), fetch or update data in the backend (kitchen), and then return responses (prepared orders).

The terms POST, PUT, DELETE, and GET are types of requests used in REST APIs:

• POST: Used to create new data. Like placing a new order in the restaurant.
• PUT: Used to update existing data. Similar to changing an order you've already placed.
• DELETE: Used to remove data. Like cancelling an order.
• GET: Used to retrieve data. Comparable to asking about the menu or checking the status of your order.

The MERN Stack Roadmap

STEP 1: Learn the right amount of HTML, JavaScript, and CSS

The MERN stack uses JavaScript heavily, so its a natural first step to learn. In this section you will look at the main things you will use day-to-day when creating full-stack MERN apps.

Understanding JavaScript is like knowing the right amount of ingredients needed for a recipe. You don't need to master everything at once, just the essential ingredients that make your particular dish (or your web project) come to life.

Variables

Variables in JavaScript are like labelled jars in your kitchen. You can store things in them (like numbers, text) and use these jars later in your cooking (or coding).

Example: A variable storing the user's name, so you can use it later to say "Hello, [Name]!"

Functions

Functions are like recipes in a cookbook. They are sets of instructions that perform a specific task. You can reuse these recipes whenever you need to perform that task again.

Example: A function that calculates the total price of items in a shopping cart.

Objects & Arrays

Objects are like information cards holding details about something (like a contact card), and arrays are like lists.

Example of an Object: A card holding a user's information (name, age, email).
Example of an Array: A list of all the user's favorite book titles.

If/else Statements, Switch Statements

These are like decision-making processes. If/else statements are like choosing what to wear based on the weather, and switch statements are like a more complex decision, like choosing what to cook based on multiple ingredients you have.

Example: If it's raining (if), take an umbrella (else), take sunglasses.

Callbacks/Promises/Async Await

These are ways to handle tasks that take some time, like ordering food and waiting for it. Callbacks are like calling a friend to do something when they’re free. Promises are like your friend promising to do it. Async-await is like making a plan to do tasks one after another in an organized way.

Example: Ordering a coffee (a task) and waiting to get it before leaving the cafe (ensuring order of actions).

ECMAScript (Template Strings, Destructuring Assignment, Spread Operator, Default Parameters, and so on)

These are advanced tools and shortcuts in JavaScript to make coding easier and cleaner. It's like having a food processor in your kitchen that makes chopping and mixing faster and more efficient.

Example: Automatically creating a welcome message like "Hello, [Name]!" without manually joining words and variables.

TypeScript

TypeScript is like JavaScript but with more rules for organizing your code (like a more detailed recipe book). It helps in managing larger projects by adding types to your code, making sure you don’t mix incompatible ingredients. This guide teaches you TypeScript basics.

Example: Specifying that a function should only take a number as an input, not text or anything else.

STEP 2: Get Familiar with React

Once you've got a feel for JavaScript, its time to venture into the wonderful world of frontend development.

React.js is a popular JavaScript library used for building user interfaces, particularly known for its efficiency in rendering dynamic, interactive web pages. It enables developers to create large web applications that can change data, without reloading the page, making for a smoother user experience.

Below is a list of common things you want to know when working with React.

Components

Think of components as individual LEGO blocks in a large LEGO model. Each block is a small, reusable piece that you can use to build different parts of your web application.

Example: A 'button' component in a website that can be used in many places, like for submitting a form or closing a pop-up.

JSX (JavaScript XML)

JSX lets you write your website's design code (like HTML) inside your JavaScript code. It's like writing the recipe and the cooking instructions in one place for easier reference.

Example: Writing a piece of JSX code that includes both JavaScript and HTML-like tags to create a user login form.

Props (Properties)

Props are like instructions or settings you pass to your LEGO blocks (components) to tell them what they should look like or do.

Example: Passing a 'color' prop to a 'button' component to make it red or blue depending on the situation.

State

State is like a personal notebook for each component, where it keeps track of its own information, like whether a button is clicked or not. Here's a course all about state management in React.

Example: A 'like' button keeping track of whether it has been clicked (liked) or not.

Hooks

Hooks are special tools in React that let you add features like state to your components without needing to use complex code structures.

Example: Using the useState hook to keep track of a counter in a component.

Event Handling

This is how you tell a component to do something when a user interacts with it, like clicking a button or entering text in a form.

Example: Setting up an event handler so that when a user clicks a 'submit' button, it sends their information to the server.

Conditional Rendering

This is like having a magic painting that can change its picture based on certain conditions. In React, you can show different components based on different conditions.

Example: Showing a 'login' button if a user is not logged in, and a 'logout' button if they are.

Lists and Keys

Keys are like name tags for items in a list. They help React keep track of which items are new, changed, or removed.

Example: Displaying a list of messages in a chat app, where each message has a unique key.

Context API

The Context API a way to share information (like a theme setting or user data) across many components without passing the information through each level manually.

Example: Using Context API to share the current logged-in user's information across different components in a web app.

Fragment

Fragments let you group several components or elements together without adding extra layers to the website. It's like putting multiple LEGO pieces on a baseplate without the baseplate being part of the final model.

Example: Grouping a list of items together in a menu without adding extra wrappers or divs around them.

STEP 3: Understand REST API's and how a backend server works using Express/Node

Every UI needs a way to store and retrieve the data it needs to make the frontend work. This is where our backend comes in.

In the MERN stack, the backend is composed of 3 main bits: Express, a Node.js server, and a database. We'll cover the database shortly, for now we'll focus on the Express/Node pieces of the MERN stack, as they're closely related.

Express.js is a lightweight, flexible framework for Node.js, designed to build web applications and REST APIs with ease and efficiency. Node.js is a powerful JavaScript runtime that allows for the development of scalable server-side applications, making the duo a popular choice for backend development.

Node.js: Foundation for Building Web Applications

Think of Node.js as the foundation for constructing a modern building. It's a platform that lets you build web applications, similar to how you would use a set of basic tools and materials to start building a house. You might hear this being to referred to as the "backend".

Example: Building a chat application where multiple users can send messages in real-time.

Express: Streamlining REST API Development

Express is a helper tool for Node.js. It's like having a pre-built kit that makes building certain parts of your house easier and faster, providing templates and shortcuts so you don't have to start from scratch.

Example: Using Express to quickly set up routes for a website, like a route to a contact page or a product catalog.

Modules and Packages: Ready-Made Components

In Node.js, modules are like pre-made components or sections of a house (like a bathroom unit or a kitchen set) that you can simply choose and add to your building project.

Example: Adding a 'date-time' module to display current times and dates on your web application.

Node Package Manager (NPM): The Tool and Material Warehouse

NPM acts like a vast warehouse where you can find all sorts of tools and materials (modules) you might need. It's a central place to get additional resources for building your web applications.

Example: Installing 'body-parser' from npm to handle JSON data in your web application.

Routing: Directing Web Traffic

Routing in Express is like setting up roads and paths in a housing complex. It's about directing the flow of traffic (data and user requests) to different parts of your web application.

Example: Creating routes in an online store, like /products for the product list and /products/:id for individual product details.

Middleware: Additional Functional Layers

Middleware in Express can be seen as extra layers or services in your building, like security, plumbing, or electrical systems. They add specific functionalities to your web application.

Example: Adding 'cookie-parser' middleware to handle cookies in your web application.

Request and Response: Communication Channels

Requests and responses in Express are like sending and receiving mail or messages. They are the way your web application communicates with users, sending them data or receiving their inputs.

Example: Your application receiving a user's login request (request) and then sending back a confirmation message (response).

Environment Variables: Secure Storage Spaces

Think of environment variables as secure storage spaces or safes in your building. They're used to store sensitive information like passwords or personal settings, keeping them secure and separate from the main construction.

Example: Storing the database connection string in an environment variable to keep it secure.

Security: Building Safeguards

In the context of web applications, security is about building safeguards into your project. It's like installing locks, security systems, and fire safety measures in a building to protect it from various threats.

Example: Implementing HTTPS to secure data transmission and using JWT (JSON Web Tokens) for user authentication to protect user data.

STEP 4: Storing data with MongoDB and Mongoose

MongoDB is a NoSQL database that offers high flexibility and scalability for storing and managing data, making it ideal for handling large volumes and diverse types of data.

Mongoose is an Object Data Modeling (ODM) library for MongoDB, providing a straightforward schema-based solution to model application data, enhancing database interaction with useful features and validation.

MongoDB: A NoSQL Database

MongoDB is a type of database that stores data in a flexible, JSON-like format. This makes it different from traditional databases, which use tables and rows. It's great for handling large volumes of data and is very scalable.

Example: Using MongoDB to store user profiles where each profile can have different fields.

Collections and Documents

In MongoDB, data is stored in 'collections', which are similar to tables in a relational database. Inside these collections, data is stored in 'documents'. Think of a document as a single record in your collection, like an entry in a diary or a contact in an address book.

Example: A 'users' collection with documents each representing a user with details like name and email.

Mongoose: A MongoDB Object Modeling Tool

Mongoose is a library for Node.js that makes it easier to interact with MongoDB. It provides a straightforward, schema-based solution to model your application data. It's like having a personal assistant to help manage the communication between your application and MongoDB.

Example: Using Mongoose to easily add, retrieve, and manage user data in a MongoDB database.

Schemas

In Mongoose, a schema is a structure that defines the format of the data to be stored in MongoDB (like defining fields and data types). Think of it as a blueprint for how your data should look.

Example: Creating a Mongoose schema for a blog post with fields like title, author, and content.

Models

A model in Mongoose acts as a constructor, compiled from the Schema definitions. It represents documents in a MongoDB database. Models are responsible for creating and reading documents from the underlying MongoDB database.

Example: Defining a 'User' model based on a user schema to interact with the 'users' collection in the database.

CRUD Operations

CRUD stands for Create, Read, Update, Delete. These are the basic operations you can perform on the database. Mongoose provides easy methods to execute these operations on your data.

Example: Using Mongoose methods to add new users, find users by name, update user information, or delete users.

Connecting to MongoDB

You can use Mongoose to connect your Node.js application to a MongoDB database. This is like setting up a phone line between your application and the database so they can talk to each other.

Example: Writing a Mongoose connect function to link your Node.js application to a MongoDB database hosted on Atlas.

Querying Data

Mongoose allows you to query your MongoDB database. This means you can search for specific data, filter your data based on certain criteria, and more.

Example: Using a Mongoose query to find all blog posts written by a specific author.

Data Validation

Mongoose provides built-in validation. This is a way to make sure the data being saved to your database is in the right format, like checking if an email address looks like an email address.

Example: Defining a schema in Mongoose where the email field must match the format of an email address.

Middleware (Pre and Post Hooks)

Mongoose middleware are functions which can be executed automatically before or after certain operations, like saving a document. They're useful for complex logic like hashing passwords before saving them to the database.

Example: Using a pre-save middleware in Mongoose to hash user passwords before saving them to the database.

Indexes

Indexes in MongoDB are used to improve the performance of searches. They are similar to indexes in a book, helping the database find data faster.

Example: Creating an index on the 'email' field in a user collection to speed up the search for users by email.

Aggregation

Aggregation in MongoDB is a powerful way to process data and get computed results. It's like having a sophisticated calculator to perform complex operations on your data, such as summing up values or averaging them.

Example: Using MongoDB's aggregation framework to calculate the average number of comments on blog posts.

STEP 5: Writing Tests

Testing in software development is like a safety check to ensure everything in your application works as expected. It's a crucial step in the development process where you look for bugs or issues before your users do. Think of it like proofreading an essay or checking a car before a road trip – it helps catch and fix problems early.

There are different types of testing, each serving a unique purpose:

Unit Testing

This is the most basic form of testing. Here, you test individual parts of your code (like functions or components) in isolation. It's like checking each light bulb in a string of Christmas lights.

In the MERN stack, tools like Jest or Mocha are commonly used for this. They let you write small tests to check if a specific part of your application behaves as expected.

Integration Testing

This type of testing checks how different parts of your application work together. It's like making sure all the light bulbs light up when connected and the string is plugged in.

For the MERN stack, you might still use Jest or Mocha, combined with other tools like Chai for assertions, to ensure that different components or services in your application interact correctly.

End-to-End (E2E) Testing

Here, you test your application's workflow from start to finish. It's like checking if the entire Christmas tree lights up and twinkles as expected.

For a MERN stack application, Cypress or Selenium are popular choices. They simulate real user scenarios, ensuring the entire application, from the front end in React to the back end with Express and Node.js, functions smoothly together.

Performance Testing

This checks if your application can handle stress, like heavy traffic or data processing. It's akin to ensuring the Christmas tree lights don't blow a fuse when they're all on. Tools like Loader.io or Apache JMeter can be used here.

Each type of testing serves to ensure a different aspect of your application is working correctly, and together, they contribute to building a robust, reliable, and user-friendly application.

By employing these tests in the MERN stack, you not only catch bugs early but also maintain a high standard of quality for your application.

STEP 6: Using Git

Git is a version control system, a tool that tracks changes in your code over time. Think of it like a detailed diary for your coding project. Every time you make changes to your code, Git keeps a record of what was changed, when, and by whom. This becomes incredibly useful when you’re working on complex projects, like those involving the MERN stack.

Why is Git so important when building with the MERN stack, or any software for that matter?

Collaboration

Git is like a team playbook. It allows multiple developers to work on the same project without stepping on each other's toes.

Everyone can work on different features or parts of the application simultaneously (like MongoDB database schemas, React components, or Express.js routes). Git helps manage these contributions, ensuring changes can be merged smoothly into the main project.

Tracking Changes

Imagine you've made changes to your React component, and suddenly things aren't working as they used to. Git allows you to go back in time and see what changes were made and by whom.

This historical data is invaluable for understanding how your project evolved and for fixing issues without having to start from scratch.

Branching and Merging

Git’s branching feature lets you diverge from the main line of development and experiment with new features or ideas in a controlled way.

You can create a branch, make your changes, and then merge those changes back into the main project when they're ready. This ensures the main project (often referred to as the 'master' or 'main' branch) remains stable.

Backup and Restore

With Git, your project's entire history is stored in the repository. If anything goes wrong, you can roll back to a previous state. It’s like having a fail-safe backup system.

Documentation

Commit messages in Git provide a narrative for your project. They allow you to document what changes were made and why, which is extremely helpful for both your future self and other developers who might work on the project.

When building applications with the MERN stack, Git offers a safety net and a collaborative workspace. It keeps your project organized, tracks every change, and allows multiple developers to work together more efficiently.

Using Git is essential for managing complex development projects in today's software development world.

STEP 7: Deployments

Code deployment is the process of taking code written by developers and making it operational on a live environment where users can interact with it.

In the context of the MERN stack, which involves technologies like MongoDB, Express.js, React, and Node.js, deployment is the final step in the journey of bringing a full-stack application to life.

Imagine you've built a house (your web application). Deployment is like moving it from the construction site (development environment) to its actual location where people can live in it (production environment). This process involves several key steps to ensure that everything works as expected when users access your app.

Preparing for Deployment

Before deploying, you need to prepare your application. This involves ensuring your code is complete and tested, dependencies are properly managed, and your app is configured for the production environment.

For MERN stack applications, this might mean setting up environment variables, configuring your database (MongoDB) connection for production, and optimizing your React front-end for performance.

Hosting and Servers

Choosing where to host your application is crucial. For MERN stack apps, you can use cloud-based hosting services like AWS, Heroku, or DigitalOcean. These platforms offer services to host both your Node.js backend and MongoDB database, and they often provide additional features like scaling, monitoring, and security.

Continuous Integration/Continuous Deployment (CI/CD)

CI/CD is a methodology that automates the deployment process. Whenever you make changes to your codebase (like fixing a bug or adding a new feature), CI/CD tools automatically test your code and deploy it if the tests pass. This ensures that your application is always up-to-date with the latest changes.

Tools like Jenkins, Travis CI, or GitHub Actions are commonly used for this purpose.

Monitoring and Maintenance

After deployment, it’s important to monitor your application for any issues and perform regular maintenance. This could involve checking server logs, updating dependencies, or rolling out fixes for any bugs that might arise.

Rollbacks

A good deployment strategy also includes plans for rollbacks. If something goes wrong after deployment, you need to be able to revert to the previous version quickly to minimize downtime.

In the MERN stack, each component (MongoDB, Express.js, React, and Node.js) might require specific considerations for deployment. For instance, you might use Docker containers to package your Node.js and Express.js backend, ensuring it runs consistently across different environments. React apps might be minified and bundled for optimal performance.

In essence, deployment in the MERN stack is about getting your application from your local machine to a server where it can serve users reliably and efficiently. It involves careful planning, choosing the right hosting solutions, automating the process as much as possible, and being prepared to address issues post-deployment.

Top Resources to Learn the MERN Stack

Now that you have a pretty good idea of what you need to learn to master the MERN stack, lets look at a list of free resources you can start using today to begin your journey!

• The freeCodeCamp curriculum is one of the best places to learn how to code for free. The curriculum is super in depth and covers alot of the topics mentioned in this post.
• The freeCodeCamp YouTube channel has a ton of free courses you can try as well. This MERN stack book project is an excellent one for beginners.
• Full Stack open is another good resource. It doesn't teach MongoDB but you learn about SQL instead which is equally as useful.
• My MERN stack hotel booking app project on YouTube is a free 15 hour course where you will learn everything talked about in this post.

10 Project Ideas You Can Try Today

One of my favourite ways to learn new technologies is to build projects. If you're struggling with coming up with ideas, here's 10 projects you can start building today. I'll be building some of these on my YouTube channel so feel free to subscribe to stay up to date!

Personal Blog Website

• Functionality: Users can read posts, and the admin can create, edit, or delete posts.
• Learning Focus: Basic CRUD operations, user authentication, integrating React with Express API, and using MongoDB for data storage.

Task Manager (To-Do List)

• Functionality: Users can add, view, edit, and delete tasks. Tasks can have deadlines and priority levels.
• Learning Focus: React state management, Express route handling, MongoDB operations, and basic UI development.

Simple E-commerce Site

• Functionality: Display products, add to cart, and checkout functionality. Admin features for adding or removing products.
• Learning Focus: React components for product listing, cart management, Express.js for product APIs, MongoDB for product data, and handling user inputs.

Recipe Sharing Application

• Functionality: Users can post, view, edit, and delete recipes. Implement a rating or comment system for interaction.
• Learning Focus: File upload for images, user-generated content management, and MongoDB schema design.

Budget Tracker

• Functionality: Users can input their expenses and income, categorize them, and view a summary of their finances.
• Learning Focus: Handling forms in React, creating RESTful services with Express, and effective data structuring in MongoDB.

Event Planning Application:

• Functionality: Users can create and manage events, send invitations, and track RSVPs.
• Learning Focus: Date handling in JavaScript, complex MongoDB documents, and Express middleware for authentication.

Fitness Tracker:

• Functionality: Log workouts, track progress over time, set fitness goals.
• Learning Focus: Data visualization with React, creating API endpoints for varied data types, and MongoDB for storing time-series data.

Chat Application:

• Functionality: Real-time chat rooms, private messaging, and user profiles.
• Learning Focus: WebSockets with Node.js for real-time communication, MongoDB for message storage, and React for dynamic UI updates.

Book Review Platform:

• Functionality: Users can post book reviews, rate books, and browse reviews by other users.
• Learning Focus: Integrating external APIs for book data, user-generated content moderation, and complex querying in MongoDB.

Local Business Directory:

• Functionality: Listing of local businesses with categories, user reviews, and contact information.
• Learning Focus: Geolocation, advanced querying and indexing in MongoDB, and creating a responsive layout with React.

Wrapping Up the MERN Stack Journey

As we reach the end of our exploration of the MERN stack, I hope this guide has illuminated the path for your journey into the world of full-stack development.

We've covered the fundamental components – MongoDB, Express.js, React, and Node.js – and delved into the practical aspects of building, testing, and deploying applications using this versatile stack.

Remember, the road to mastering the MERN stack is both challenging and rewarding. Each project you undertake will add to your skills and confidence. Use the resources and project ideas provided as stepping stones to build your portfolio and deepen your understanding.

The beauty of the MERN stack lies in its community-driven nature, so don't hesitate to seek help, collaborate, and share your experiences.

If you want to get in touch, you can reach me over at LinkedIn, or drop a comment on my YouTube channel. Good luck and happy coding!

Amazon is the most popular e-commerce website for web scrapers, with billions of product pages being scraped every month.

It is also home to a vast database of product reviews, which can be very useful for market research and competitor monitoring.

You can extract relevant data from the Amazon website and save it in a spreadsheet or JSON format. And you can even automate the process to update the data regularly.

Scraping Amazon product reviews is not always straightforward, especially when a login is required. In this guide, you'll learn how to scrape Amazon product reviews behind a login. You’ll learn the process of logging in, parsing review data, and exporting reviews to CSV.

Important Disclaimer: This tutorial is for educational purposes only. Scraping data from behind logins on websites may violate their terms and conditions (T&Cs). It's crucial to always check the T&Cs of any website before scraping data.

Without further ado, let's get started.

Prerequisites and Project Setup

We’ll use the Node.js Puppeteer library to scrape Amazon reviews. Make sure Node.js is installed on your system. If it is not, go to the official Node.js website and install it.

After Node.js is installed, install Puppeteer. Puppeteer is a Node.js library that provides a high-level, user-friendly API for automating tasks and interacting with dynamic web pages.

Now, let's install and configure Puppeteer.

Open a terminal and create a new folder with any name. (In my case, it is _amazonreviews).

mkdir amazon_reviews

Change your current directory to the folder created above.

cd amazon_reviews

Cool, you're now in the correct directory. Execute the following command to initialize the package.json file:

npm init -y

Finally, install Puppeteer using the following command:

npm install puppeteer

This is what the process looks like:

Now, open the folder in any code editor, and create a new JavaScript file (index.js). Make sure that the hierarchy looks like this:

_Hierarchy showing node_modules, index.js, package-lock.json, and package.json_

All set up successfully. We’re now ready to code the scraper.

Note: Ensure that you have an account on Amazon so you can progress through the rest of this tutorial.

Step 1: Get Access to the Public Page

You're going to scrape the reviews of the product shown below. You’ll extract the author's name, review title, and date.

Here's the product URL: https://www.amazon.com/ENHANCE-Headphone-Customizable-Lighting-Flexible/dp/B07DR59JLP/

The product we're using in the example - headphones

First, you’ll log in to Amazon, and then redirect to the product URL to scrape the reviews.

Step 2: Scrape Behind the Login

Amazon's multi-stage login process requires users to enter their username or email, click a Continue button to enter their password, and then finally submit it. Both the username and password fields are typically on different pages.

To enter the email ID, use the selector input[name=email].

HTML of the sign-in field

Now, click on the Continue button using the selector input[id=continue].

HTML of the continue button

Now you should be on the next page. To enter the password, use the selector input[name=password].

HTML of the password field

Finally, click on the Sign In button using the selector input[id=signInSubmit].

HTML of the sign-in button

Here’s the code for the login process:

const selectors = {
  emailid: 'input[name=email]',
  password: 'input[name=password]',
  continue: 'input[id=continue]',
  singin: 'input[id=signInSubmit]',
};


    await page.goto(signinURL);
    await page.waitForSelector(selectors.emailid);
    await page.type(selectors.emailid, "satyam@gmail.com", { delay: 100 });
    await page.click(selectors.continue);
    await page.waitForSelector(selectors.password);
    await page.type(selectors.password, "mypassword", { delay: 100 });
    await page.click(selectors.singin);
    await page.waitForNavigation();

We're following the same steps as discussed above. First, go to the sign-in URL, enter the email ID, and click on the Continue button. Then enter the password, click on the Sign In button, and wait for a moment for the sign-in process to complete.

After the sign-in process is completed, you’ll be redirected to the product page to scrape the reviews.

Product page

Step 3: Parse the Review Data

You've successfully logged in and are now on the product page that you want to scrape. Let's now parse the review data.

On the page, you'll find various reviews. These reviews are contained within a parent div with the ID cm-cr-dp-review-list, which holds all the reviews on the current page. If you want to access more reviews, you'll need to navigate to the next page using the pagination process.

This parent div has multiple child divs, and each child div holds one review. To extract the reviews, you can use the selector #cm-cr-dp-review-list div.review.

const selectors = {
  allReviews: '#cm-cr-dp-review-list div.review',
  authorName: 'div[data-hook="genome-widget"] span.a-profile-name',
  reviewTitle: '[data-hook=review-title]>span:not([class])',
  reviewDate: 'span[data-hook=review-date]',
};

This selector shows that you first go to the element with the ID cm-cr-dp-review-list, then search for all div elements with the data-hook review.

Review data with Author name, Review Title, Description, etc.

The following code snippet shows that you should first go to the product URL, wait for the selector to load, and then scrape all the reviews and store them in the reviewElements variable.

await page.goto(productURL);
await page.waitForSelector(selectors.allReviews);
const reviewElements = await page.$$(selectors.allReviews);

Now, let's extract the author's name, review title, and date.

Targetting Author name, Review Title, and Date

To parse the author name, you can use the selector div[data-hook="genome-widget"] span.a-profile-name. This selector tells us to first search for the div element with the data-hook attribute set to genome-widget, because the names are inside this div element. Then, search for the span element with the class name a-profile-name. This is the element that contains the author's name.

const author = await reviewElement.$eval(selectors.authorName, (element) => element.textContent);

To parse the review title, you can use the CSS selector [data-hook="review-title"] > span:not([class]). This selector tells us to search for the span element that is a direct child of the [data-hook="review-title"] element and that does not have a class attribute.

const title = await reviewElement.$eval(selectors.reviewTitle, (element) => element.textContent);

To parse the date, you can use the CSS selector span[data-hook="review-date"]. This selector tells us to search for the span element that has the data-hook attribute set to review-date. This is the element that contains the review date.

const rawReviewDate = await reviewElement.$eval(selectors.reviewDate, (element) => element.textContent);

Note that you’ll get the entire text, including the location, instead of just the full date. Therefore, you must use a regular expression pattern to extract the date from the text.

After that, combine all of the data into the reviewData and then push it to the final list reviewsData.

const datePattern = /(\w+\s\d{1,2},\s\d{4})/;
      const match = rawReviewDate.match(datePattern);
      const reviewDate = match ? match[0].replace(',', '') : "Date not found";

      const reviewData = {
        author,
        title,
        reviewDate,
      };

      reviewsData.push(reviewData);
    }

The above process will run until it has parsed all of the reviews on the current page. Here’s the code snippet to parse the data:

for (const reviewElement of reviewElements) {
      const author = await reviewElement.$eval(selectors.authorName, (element) => element.textContent);
      const title = await reviewElement.$eval(selectors.reviewTitle, (element) => element.textContent);
      const rawReviewDate = await reviewElement.$eval(selectors.reviewDate, (element) => element.textContent);

      const datePattern = /(\w+\s\d{1,2},\s\d{4})/;
      const match = rawReviewDate.match(datePattern);
      const reviewDate = match ? match[0].replace(',', '') : "Date not found";

      const reviewData = {
        author,
        title,
        reviewDate,
      };

      reviewsData.push(reviewData);
    }

Great! You’ve successfully parsed the relevant data, which is now in JSON format, as shown below:

Scraped the data in JSON format

Step 4: Export Reviews to a CSV

You've parsed the reviews in JSON format, which is a bit human-readable. You can convert this data to CSV format to make it more readable and easier for other purposes.

There are many ways to convert JSON data to CSV, but we'll use a simple and effective approach. Here is a simple code snippet to convert JSON to CSV:

let csvContent = "Author,Title,Date\n
for (const review of reviewsData) {
      const { author, title, reviewDate } = review;
      csvContent += `${author},"${title}",${reviewDate}\n`;
    }

const csvFileName = "amazon_reviews.csv";
await fs.writeFileSync(csvFileName, csvContent, "utf8");

Here’s the output of the CSV file.

Converted JSON data into CSV format

And there you have it!

You can find the full Code uploaded on GitHub here.

Conclusion

In this guide, you learned how to scrape Amazon product reviews behind a login using Puppeteer. You learned how to log in, parse relevant data, and save it to a CSV file.

To practice more, you can extract all the reviews of all the pages using pagination.

We'll start from scratch and end up with a fully functioning app where you can create, edit, and delete notes. Plus, we're adding validation on both the UI and the backend to keep things in check!

This guide is all about giving you the real-deal experience of building a web app. You'll get to learn how each piece of the puzzle fits together – from the front-end magic with React, to the server-side wonders with Node.js, and storing all the good stuff in a PostgreSQL database. And hey, we're making sure it looks good and works well on mobile screens too!

By the end of this, you’ll have a good grip on full stack development with React and Node, which you can carry with you into future projects. It’s all about learning by doing, and getting the skills to make your ideas come to life. So, grab a cup of coffee, roll up your sleeves, and let’s get coding!

Prerequisites

Since we will be focusing on how to build a project, there are a few prerequisites that will be needed to get the most out of this tutorial:

• Some knowledge about web development concepts (frontend, backend, databases, API's, REST).
• Some knowledge of JavaScript (variables, functions, objects, arrays, and so on).
• Basic understanding on React (how to create components, add styles, work with state).
• Basic understanding on Node.js/Express (working with APIs).

Table of Contents

• What We'll Build
• Challenge: Try it Yourself First!
• Video Tutorial
• PART 1 – Create the UI
• PART 2 - Create the Backend
• PART 3 - Connect UI to Backend
• The End - Why not try the bonus challenges?

What We'll Build

In this tutorial, we'll build a full stack notes app from scratch, using React, Node.js and PostgreSQL, with the following features:

• Create/Edit/Delete Notes
• Validation on the UI and Backend
• Responsive on mobile screens

Challenge: Try it Yourself First!

If you would like to attempt this project yourself first without looking at the tutorial, here's a few hints:

• Tackle one little piece at a time. For example, you would focus on getting the UI working on the UI first, and tackle the APIs later.
• Think about your data – What do you need to store? What data structure (for example, arrays) will you use to return the data via the API? How will you render this data on the UI?
• Don't forget about validation and error handling. What will happen if the user tries to save a note without a title field? How will you prevent this? (Hint: Forms and the required field will be your friend here)
• Remember there is no perfect way to complete a project. The tutorial below is one way to tackle the problem. You can choose to go a different way, putting your own unique style on things. The main thing is that you start!

If you need more help on getting started yourself, you can find more hints and tips, starter code, and completed code you can reference over at codecoyotes.com.

Video Tutorial

PART 1 – Create the UI

We'll start this tutorial by creating the UI using some mock data. This lets us focus on our styling, and how things to look, without having to worry about creating a backend right away.

Create a New React App

Okay, first things first: let's set up our project structure. Open your terminal and navigate to your desktop. Our next step is to create a new folder that will hold both our UI and backend code. Let's name it notes-app:

mkdir notes-app

Once that's done, navigate into the newly created notes-app directory:

cd notes-app

Next, we'll create a new React app using TypeScript as our template. We'll use the npx create-react-app command for this, specifying TypeScript as the template:

npx create-react-app notes-app --template typescript

After you hit Enter, the process may take a few minutes to install all necessary packages. Once it's completed, open the notes-app folder in Visual Studio Code or your preferred IDE.

In Visual Studio Code, you should see that the notes-app is at the top level of your directory. As the course progresses, we will add a notes-app-server directory as well to keep all the code together in one place.

Now open a new terminal within your IDE and navigate to your React app's directory (let's assume you named it notes-app):

cd notes-app

Then, run the following command to start the front-end development server:

npm start

If all goes well, your browser will automatically open and display your new React app. You should see a spinning React logo, among other default assets.

Finally, let's clear out the boilerplate code to have a clean starting point for our app. Open src/App.tsx in your IDE and delete its content. This will be our new starting point for building the app.

Add UI Elements

Okay, the first thing we'll do is put some of our UI components in place. This will consist of the general markup and CSS, without involving any JavaScript. This gives us a glimpse of how we envision the layout, without having to concern ourselves with API calls or database interactions at this stage.

We'll navigate to .App.tsx and create a new component. Make sure to import our stylesheet from App.css. The first thing to add is a div with a class name of AppContainer. This will help position our form and the CSS grid for our notes.

Within this div, we'll include our form tags. Here, we'll add an input field for the title — this is where the user can enter the note title. We'll also include a textarea for the note content. Both of these fields will be set to required, enabling browser-native validation messages if the user tries to submit an incomplete form.

At the bottom of the form, we'll include a button of type submit, which will handle form submissions when clicked.

Off to the right, we'll add a div to contain our notes. This will be structured as a CSS grid. Initially, we'll populate this grid with a single note to see how it looks.

For our note, we'll have a header containing a delete button situated on the right-hand side. We'll display the user-entered title – for demonstration purposes, we're using a dummy title. We'll also include the content that the user entered.

Finally, we'll export our component at the very bottom.

Completed code for this section

import "./App.css";

const App = () => {
  return (
    <div className="app-container">
      <form className="note-form">
        <input placeholder="Title" required />
        <textarea placeholder="Content" rows={10} required />

        <button type="submit">Add Note</button>
      </form>
      <div className="notes-grid">
        <div className="note-item">
          <div className="notes-header">
            <button>x</button>
          </div>
          <h2>Note Title</h2>
          <p>Note content</p>
        </div>
      </div>
    </div>
  );
};

export default App;

Add CSS

Start the App

First, let's open a terminal and type npm start. This will launch the app in the browser. As you can see on the right-hand side, the appearance isn't great yet. This is because we haven't applied any styles. To fix this, we'll navigate to App.css and style the classes we added earlier.

You can download the styles from the link in the description if you prefer to copy and paste. Alternatively, feel free to follow along with the video and pause as needed. Remember, these styles are just examples for learning – they don't have to be perfect.

Style the Body and App Container

In App.css, the first thing we'll do is add some styles to the body. We'll give it a gray background and a margin to prevent the app from touching the browser window edges. Next, we'll style our App Container.

We're designing this mobile-first, meaning the default styles will target mobile screens. We'll use media queries for larger displays. This approach is optional, but often it's easier to start with mobile designs.

For mobile screens, we want our App Container to default to a single column layout, stacking our form and notes grid on top of each other.

Use Media Queries

We'll add a media query specifying that for screens larger than 600 pixels, we'll use a two-column layout. We'll define this using grid-template-columns.

The first column will be 200 pixels wide, accommodating the form. The second column will use 1fr, filling the remaining space. A 20-pixel gap will separate the two columns.

Style the Notes Grid

Next, let's style our notes grid. We'll use CSS grid and define grid-template-columns.

Each grid item will have a minimum width of 250 pixels and can expand to fill available space. Don't worry if this sounds confusing – it will become clear soon.

We'll also set grid-auto-rows to ensure each row is a minimum of 250 pixels tall, accommodating notes of different sizes while maintaining a consistent row height.

Style Individual Notes

For each note, we'll use Flexbox and set flex-direction to column, stacking the header, title, and content vertically. We'll also add some basic styles like border, padding, and background color. A box shadow will provide a finishing touch.

Style the Header and Delete Button

The header will also use Flexbox, and we'll set justify-content to flex-end to align the delete button to the right. The button will receive custom styles for a polished look.

Style the Form

Lastly, we'll style the form on the left column. Again, we'll use Flexbox with a column layout and a 20-pixel gap between elements. The text area and input fields will get borders, padding, and resized fonts. We'll also style the submit button and add hover effects.

Completed code for this section

body {
  margin: 20px;
  background-color: lightgrey;
}

.app-container {
  grid-template-columns: 1fr;
}

@media (min-width: 600px) {
  .app-container {
    display: grid;
    grid-template-columns: 200px 1fr;
    gap: 20px;
  }
}

.notes-grid {
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(250px, 1fr));
  grid-auto-rows: minmax(250px, auto);
  gap: 20px;
}

.note-item {
  display: flex;
  flex-direction: column;
  border: 1px solid #ccc;
  padding: 10px;
  border-radius: 5px;
  background-color: #f9f9f9;
  box-shadow: 0px 0px 10px rgba(0, 0, 0, 0.1);
  cursor: pointer;
}

.notes-header {
  display: flex;
  justify-content: flex-end;
}

.notes-header button {
  font-size: 16px;
  background: transparent;
  border: none;
  cursor: pointer;
  max-width: fit-content;
}

h2 {
  margin: 0;
}

.note-form {
  display: flex;
  flex-direction: column;
  gap: 20px;
}

textarea,
input {
  border-radius: 5px;
  border: 1px solid black;
  padding: 10px;
  font-size: 16px;
}

.note-form button {
  border-radius: 5px;
  background-color: rgb(64, 154, 184);
  border: none;
  padding: 10px;
  font-size: 16px;
  color: white;
}

.note-form button:hover {
  background-color: rgb(106, 175, 198);
  cursor: pointer;
}

.edit-buttons {
  display: flex;
  justify-content: space-evenly;
  gap: 5px;
}

.edit-buttons button {
  flex: 1;
}

.edit-buttons button:last-of-type {
  background-color: rgb(220, 89, 89);
  color: white;
}

Add Dummy Notes

Add Dummy Notes for CSS Grid Testing

Now that we have our CSS in place, the next step is to add dummy notes to our App component to test the responsiveness of our CSS grid. To achieve this, we'll navigate to App.tsx and import the useState hook from React.

Inside our App component, we'll store the notes within the useState hook. Whenever you have UI elements that can change, it's a good idea to manage them in the state. We'll initialize this with a dummy array of notes, where each note has an id, a title, and content:

const [notes, setNotes] = useState<Note[]>([
{
  id: 1,
  title: "test note 1",
  content: "bla bla note1",
},
{
  id: 2,
  title: "test note 2 ",
  content: "bla bla note2",
},
{
  id: 3,
  title: "test note 3",
  content: "bla bla note3",
},
{
  id: 4,
  title: "test note 4 ",
  content: "bla bla note4",
},
{
  id: 5,
  title: "test note 5",
  content: "bla bla note5",
},
{
  id: 6,
  title: "test note 6",
  content: "bla bla note6",
},
]);

Think of this as simulating an API call and storing the returned data in the state. The structure of this data will be similar to what we'll receive from our API requests when we eventually build out our backend.

Map Notes to Components

With our array of notes in state, we can now use the map function within our notes grid to display the markup for each note. The map function will run as many times as there are notes in the array. Instead of hardcoding the title and content, we'll pull these values from each note object:

<div className="notes-grid">
  {notes.map((note) => (
    <div className="note-item">
      <div className="notes-header">
        <button>x</button>
      </div>
      <h2>{note.title}</h2>
      <p>{note.content}</p>
    </div>
  ))}
</div>

Verify Responsiveness

After these steps, you should see four notes displayed in the browser, populated with the values from the objects in the array.

To verify that our layout is responsive, you can change the window size. You'll see that the notes adjust based on the window size. When the window is at its smallest—simulating a mobile screen—the form will stack vertically above the notes grid.

Save Note Form

Now that we have our UI set up, let's focus on adding functionality to the form that allows us to create a new note. Initially, we'll implement this for the UI. Later, we'll make the data persistent by linking it to the backend, which we'll build separately.

Use State for Form Inputs in React

In React, when working with forms, it's a best practice to maintain a state variable for each form input. This enables React to control those inputs, making it easier to capture their values and use them programmatically.

In our code, we have two form inputs: one for the title and another for the content. For these, we'll set up two state variables called title and content:

const [title, setTitle] = useState("");
const [content, setContent] = useState("");

For the title input, we bind its value to the title state variable and update this state whenever the user types into the field:

<input
  value={title}
  onChange={(event) => setTitle(event.target.value)}
  placeholder="Title"
  required
></input>

Similarly, we'll handle the textarea for content:

<textarea
  value={content}
  onChange={(event) => setContent(event.target.value)}
  placeholder="Content"
  rows={10}
  required
></textarea>

Handle Form Submission

After binding our form inputs to state variables, the next step is to add a function that handles the form submission. We'll name this function handleAddNote:

const handleAddNote = (event: React.FormEvent) => {
  event.preventDefault();
  console.log("title: ", title);
  console.log("content: ", content);
};

In this function, we specify the parameter type as React.FormEvent to satisfy TypeScript's typing requirement. We also call event.preventDefault() to prevent the form from submitting and refreshing the page, which is its default behavior. Following that, we log the title and content state variables to the console.

Finally, we'll connect this function to the onSubmit event in our form:

<form onSubmit={handleAddNote}>{/* ...form inputs here... */}</form>

Test the Form

To test this setup, open the browser console, input a title and some content, and click the "Add Note" button. You should see the title and content values logged in the console, confirming that our form is capturing input as expected.

Handle the "Add Note" Functionality

Now that we've set up our state variables for the title and content, we can proceed to implement the function that handles adding a new note. This function will create a new note object and add it to our notes array, thereby updating the UI.

Create a New Note Object

First, let's create a new note object and specify its type as Note, taking advantage of TypeScript's type system:

const newNote: Note = {
  id: notes.length + 1,
  title: title,
  content: content,
};

Because we've explicitly typed our object, TypeScript's IntelliSense will assist us in populating the object, ensuring that we don't miss any required properties. For now, we'll set the id to the length of the current notes array plus one, although this id will eventually be generated by our backend database.

Update State with New Note

Once we have our new note object, we need to update our notes state array. We'll use the setNotes function for this purpose:

setNotes([newNote, ...notes]);

The new note object will be the first item in the new notes array, followed by the existing notes, which we'll spread into the new array using the spread operator. This effectively makes a copy of the old notes array and inserts it into the new one.

Clear the Form Inputs

Lastly, let's reset the title and content state variables to empty strings, improving the user experience by clearing the form once a note is added:

setTitle("");
setContent("");

Test the Functionality

And that's it! If you now go to the browser, input a title and some content, and then click "Add Note," you'll see your new note appear at the top of the list, and the form fields will be cleared, ready for a new entry.

Handle the "Update Note" Functionality

In this section, we'll focus on implementing the feature that allows users to update an existing note. When a user clicks on a note, we want to populate the title and content fields in our form with the note's existing values. We'll also add a "Save" and "Cancel" button.

Cleanup and Initial Setup

First, let's clean up our code by removing any console.log statements – they are no longer needed:

const [selectedNote, setSelectedNote] = useState<Note | null>(null);

Track the Selected Note

To track which note the user has clicked on, we'll create a new state variable called selectedNote. This state variable will have a type of Note or null to account for the possibility that no note is selected. We'll initialize this state to null.

Create the Click Handler

Next, let's create a function named handleNoteClick to handle the user's click event on a note. This function will take a note object as its argument:

const handleNoteClick = (note: Note) => {
  setSelectedNote(note);
  setTitle(note.title);
  setContent(note.content);
};

Within this function, we'll use setSelectedNote to save the clicked note to our selectedNote state. Additionally, we'll populate the title and content state variables with the values from the clicked note.

Update the UI

In the JSX for rendering each note, add an onClick event to the top-level div element for each note. Call the handleNoteClick function and pass the note object to it:

<div key={note.id} className="note-item" onClick={() => handleNoteClick(note)}>
  <div className="notes-header">
    <button>x</button>
  </div>
  <h2>{note.title}</h2>
  <p>{note.content}</p>
</div>

Since we are iterating over notes using the map function, this onClick handler will be added to each note automatically.

Save user changes

Now that we have the capability for the user to edit a note, we'll implement the functionality to save the changes they make to both the title and content of a note into our state.

The handleUpdateNote Function

Let's create a new function called handleUpdateNote:

const handleUpdateNote = (event: React.FormEvent) => {
  event.preventDefault();

  if (!selectedNote) {
    return;
  }

  const updatedNote: Note = {
    id: selectedNote.id,
    title: title,
    content: content,
  };

  const updatedNotesList = notes.map((note) => (note.id === selectedNote.id ? updatedNote : note));

  setNotes(updatedNotesList);
  setTitle("");
  setContent("");
  setSelectedNote(null);
};

Within this function, we use event.preventDefault() to prevent the form from automatically submitting when the "Save" button is clicked. We also validate if a note is selected. If not, we exit the function early to prevent potential errors.

Next, we form an updated note object based on the selected note's id and the updated title and content. After that, we utilize the map function to generate a new array of notes, replacing the selected note with our updated note where the id matches. The updated array is then set to our state using the setNotes function. Finally, we reset our title, content, and selectedNote state values to their initial states.

The handleCancel Function

We'll also implement a simple handleCancel function to reset our form and selected note when the user decides not to proceed with an update:

const handleCancel = () => {
  setTitle("");
  setContent("");
  setSelectedNote(null);
};

Update the JSX

Let's introduce conditional rendering in our JSX to display the appropriate buttons based on whether a note is selected for editing or not:

<form
  className="note-form"
  onSubmit={(event) => (selectedNote ? handleUpdateNote(event) : handleAddNote(event))}
>
  {/* ... other form elements ... */}
  {selectedNote ? (
    <div className="edit-buttons">
      <button type="submit">Save</button>
      <button onClick={handleCancel}>Cancel</button>
    </div>
  ) : (
    <button type="submit">Add Note</button>
  )}
</form>

Within our form's onSubmit event, we've added a conditional. If a note is selected, we'll trigger the handleUpdateNote function. Otherwise, the handleAddNote function will be executed.

Test the Implementation

After incorporating these changes, run your application. When you select a note, make modifications, and click on "Save", you'll observe the note gets updated.

Delete Notes from the UI

The last piece of functionality we need on the frontend before moving to the backend development is the ability to delete notes. You'll recall that we added a small "X" button to each note for this purpose. Clicking this button should remove the note from the UI. Let's jump back into the App.tsx file and implement this.

The deleteNote Function

First, create a function named deleteNote as follows:

const deleteNote = (event: React.MouseEvent, noteId: number) => {
  event.stopPropagation();

  const updatedNotes = notes.filter((note) => note.id !== noteId);

  setNotes(updatedNotes);
};

This function takes in two parameters: the event object and the noteId. The event.stopPropagation() line is crucial here because the delete button is nested within a clickable note. It prevents the deleteNote event from interfering with the click event on the note itself. This is especially important when dealing with nested onClick events.

The Filtering Logic

The core of the delete functionality lies in the filter method applied to the notes array. This method loops through the array and applies a function to each element, much like the map method. It will only return the notes whose IDs do not match the noteId provided, effectively removing the selected note.

We save this newly filtered array into a variable called updatedNotes and then update our state with it by calling setNotes(updatedNotes).

Add the onClick Event

After defining the deleteNote function, attach it to the delete button within the note. Pass in the event and the note ID, like so:

<button onClick={(event) => deleteNote(event, note.id)}>x</button>

Test the Functionality

Now, if you run your app and click the delete button on a given note, you will observe that the note disappears from the UI.

PART 2 - Create the Backend

After implementing UI functionalities, it's time to set up a backend that allows us to persist notes when the user adds, edits, or deletes them. For this, create a new folder in your project at the top level and name it notes-app-server. Even though it might seem like the server code is in the same directory as the UI, they are entirely separate and will run independently.

Initial Setup

1. Open your terminal and navigate to the notes-app-server folder you just created.
2. Run the following commands:

npm init
npm i ts-node typescript nodemon @types/cors @types/express @types/node --save-dev
npm i @prisma/client cors express prisma
npx tsc --init

• npm init: Initializes a new npm module and gives you access to npm packages.
• npm i ... --save-dev: Installs development dependencies like TypeScript and type definitions.
• npm i ...: Installs production dependencies like Express and Prisma.

Modify package.json

After running the above commands, navigate to your package.json and update the scripts section with:

"start": "npx nodemon"

This script uses nodemon for hot-reloading.

Implement the Server

Now, within the notes-app-server directory, create a src folder and within it, an index.ts file. Insert the following code:

import express from "express";
import cors from "cors";

const app = express();

app.use(express.json());
app.use(cors());

app.get("/api/notes", async (req, res) => {
  res.json({ message: "success!" });
});

app.listen(5000, () => {
  console.log("server running on localhost:5000");
});

1. import express and cors: We import the required libraries for our server.
2. const app = express();: Initializes a new Express application.
3. app.use(express.json());: Parses the JSON body from incoming API requests.
4. app.use(cors());: Adds CORS support.
5. app.listen(5000, ...): This starts the server listening on port 5000.

Test

Finally, you can test the server by navigating to the notes-app-server directory in your terminal and running:

npm start

You should see the console log: server running on localhost:5000. To further test, you can use a curl command to hit the /api/notes endpoint. If everything is set up correctly, you'll get a JSON object back.

Create a Postgres Database

ElephantSQL is a PostgreSQL database hosting service that makes it easy to set up, maintain, and scale your PostgreSQL database. Here's how to get started with creating a database using ElephantSQL.

Step 1: Sign Up / Log In

1. Navigate to the ElephantSQL website.
2. If you don't have an account, you can sign up for free. If you already have one, go ahead and log in.

Step 2: Create a New Instance

1. Once logged in, you'll find yourself on the "Dashboard" page.
2. Click on the "Create New Instance" button.
3. You'll be taken to a page where you can set the details for your new PostgreSQL database instance.

Step 3: Choose a Plan

1. You can start with a free "Tiny Turtle" plan, which is perfect for small projects and testing.
2. Select the plan that best suits your needs and click "Select".

Step 4: Configure Your Instance

1. You'll be asked to name your instance. Choose a name that you'll remember and that describes the purpose of the database.
2. You can also select the data center that is geographically closest to you or your users for better performance.
3. Click on "Review" and then "Create instance" to finalize the creation.

Step 5: Access Your Database

1. Once the instance is created, click on it in the Dashboard.
2. Here, you'll see the "Details" tab which includes all the information you need to connect to your database: URL, User & Default database, Password, and more.

Populate the DB

Step 1: Login to ElephantSQL

Open your web browser and navigate to the ElephantSQL website. Log
in to your account.

Step 2: Open your Instance

Once logged in, click on the name of the database instance you've set up.

Step 3: Navigate to the SQL Browser

In the left sidebar, find and click on "SQL Browser" or something similar (it might say "Browser").

Step 4: Run SQL Query

In the SQL Query editor that appears, you can type or paste in your SQL command:

INSERT INTO "public"."Note" (title, content)
VALUES ('test title', 'test content bla bla');

After entering the SQL, click on the "Execute" or "Run" button.

That should insert a new row into your Note table with the title 'test title' and content 'test content bla bla'.

Optional: Verify the Insert

You may also want to verify if the data has been inserted correctly. For that, you could use:

SELECT * FROM "public"."Note";

Run this SQL query in the same SQL browser, and it should return all rows from the Note table, including the one you've just inserted.

And that's it! You've inserted a new row into your table through the ElephantSQL web console.

Connect to DB from Node.js backend using Prisma

Step 1: Copy the ElephantSQL Connection URL

Once you've set up your ElephantSQL database, make sure to copy the connection URL that appears on your dashboard. This URL includes your username and password to the database, so keep it secure.

Step 2: Create an .env File

Navigate to your notes-app-server directory and create a new .env file:

touch .env

Open this file and add the following line to specify the database connection URL:

DATABASE_URL="your_connection_url_here"

Make sure not to commit this .env file to your Git repository to keep your credentials secure.

Step 3: Initialize Prisma

If you haven't installed Prisma yet, install it first:

npm install prisma --save-dev

Now, initialize Prisma in the notes-app-server directory:

npx prisma init

This command will create a new prisma folder containing a schema.prisma file.

Step 4: Configure schema.prisma

Open schema.prisma in your text editor. You'll see that Prisma has already generated some configurations for you. Update the datasource block to use the environment variable:

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

Step 5: Create the Note Model

Below the datasource block, add a new model block to represent a Note:

model Note {
  id      Int     @id @default(autoincrement())
  title   String
  content String
}

Step 6: Generate Prisma Client and Database Table

Run the following command to generate your Prisma client and create the database tables:

npx prisma db push

Step 7: Add Prisma to Your Application

First, import Prisma at the top of your index.ts:

import { PrismaClient } from "@prisma/client";

Then, initialize the Prisma client:

const prisma = new PrismaClient();

Step 8: Query Your Database

Now you can use Prisma in your application to query the database. For example, in a GET endpoint:

app.get("/notes", async (req, res) => {
  const notes = await prisma.note.findMany();
  res.json(notes);
});

Optional: Install Thunder Client in VS Code

(Feel free to skip this step if you already have a preferred API client)

Using curl is useful for quickly testing APIs, but it becomes cumbersome when you need to build out more complex requests. For instance, handling POST requests with custom bodies and headers can be more complicated.

To make API requests more straightforward, we'll install a client designed for this purpose.

While there are several options like Postman, we're going to focus on installing Thunder Client within VS Code, which makes it simple to execute requests right from your IDE.

To install Thunder Client, navigate to the Extensions section in VS Code and type "Thunder Client" in the search bar. You'll find it in the list of available extensions, identifiable by its purple logo. Click "Install," and upon completion, you'll see a Thunder Client option appear on the left-hand taskbar of your IDE.

Once you've clicked on Thunder Client, a list of your past requests will display. To initiate a new request, click the "New Request" button at the top. This action opens a new tab within Visual Studio Code.

Before proceeding, ensure that your server is running. Open the terminal and verify this. We will use Thunder Client to test our GET endpoint and get familiar with the request-making process. In the URL bar, enter the address of your 'notes' endpoint and specify that it's a GET request.

Click "Send," and you'll see a small window displaying the response. If the status code is 200 and you see an array containing your note, you've successfully made a GET request. Thunder Client will be our tool of choice for testing subsequent create, update, and delete requests. Of course, feel free to use any other tool you're comfortable with for this purpose.

Create POST Endpoint

In this section, we'll add an endpoint to our Express application that allows us to create a new note. Locate the index.ts file and insert the following code below your existing GET endpoint:

app.post("/api/notes", async (req, res) => {
  const { title, content } = req.body;

  if (!title || !content) {
    return res.status(400).send("title and content fields required");
  }

  try {
    const note = await prisma.note.create({
      data: { title, content },
    });
    res.json(note);
  } catch (error) {
    res.status(500).send("Oops, something went wrong");
  }
});

The structure is similar to the GET endpoint, but we're using app.post this time. We specify the URL for this POST endpoint and then define our function.

Inside the function, the first task is to extract title and content from the req.body. This is what the UI will send when a user submits the "Add Note" form.

After obtaining title and content, we utilize the Prisma client that we set up earlier to create a new note. We pass the title and content to the prisma.note.create() method, which returns a new note object complete with an ID. This object is then sent back as a JSON response.

To test the endpoint, go to the Thunder Client tab in VS Code. Switch the HTTP method from GET to POST while keeping the URL the same. Click the "Body" tab, which should default to JSON, and input some test values for title and content. After hitting "Send," you should receive a 200 OK status along with the created note, containing an ID, title, and content.

For robustness, we've added validation and error-handling. If either title or content is missing, the server returns a 400 Bad Request status with an appropriate error message. To test this, remove either title or content from the request body and resend it. You should now see a 400 status code along with your error message.

Additionally, we use a try-catch block to handle any errors thrown by the Prisma client. This helps in cases of database connection issues or other unforeseen errors, preventing the backend from crashing.

Finally, you can test the GET endpoint again. It should now return two notes: the first one manually added to the database, and the second one just created through Thunder Client. Switch the method back to GET in Thunder Client and hit "Send"; you should see two notes in the response.

Create PUT Endpoint

In this segment of the tutorial, we'll focus on adding the ability to update a note. Add the following code snippet below the code for your previous POST endpoint:

app.put("/api/notes/:id", async (req, res) => {
  const { title, content } = req.body;
  const id = parseInt(req.params.id);

  if (!title || !content) {
    return res.status(400).send("title and content fields required");
  }

  if (!id || isNaN(id)) {
    return res.status(400).send("ID must be a valid number");
  }

  try {
    const updatedNote = await prisma.note.update({
      where: { id },
      data: { title, content },
    });
    res.json(updatedNote);
  } catch (error) {
    res.status(500).send("Oops, something went wrong");
  }
});

The structure of this app.put function is similar to the GET and POST endpoints you've already created. The major difference is the :id parameter in the URL. This acts as a placeholder, allowing you to specify the ID of the note you wish to update.

Inside the function, you'll notice we extract title and content from req.body, just like before. Additionally, we retrieve the ID from req.params and convert it to an integer using parseInt(), as our database stores IDs as integers.

We've added validation checks to ensure that the id exists and is a valid number. If either id, title, or content is missing or invalid, the API returns a 400 status code along with an error message.

Next, we use a try-catch block to attempt the update operation. Within the try section, we call the prisma.note.update() function. We specify the id in a where object and provide the new title and content via a data object. If the operation succeeds, the updated note is sent back in the response. In case of an error, the catch block will return a 500 status and an error message.

To test this, switch to your Thunder Client tab in VS Code. Update the method to PUT and set the URL to include the ID of the note you want to update, for example, /api/notes/3. In the request body, send JSON data with the new title and content. Upon hitting "Send", a 200 status should confirm the update. The returned note should reflect your changes.

To double-check, perform a GET request on the /api/notes endpoint. You should see the updated note in the list.

Lastly, test the validation by supplying an invalid ID, like a random string. The API should return an error message stating that the ID must be a valid number.

Create DELETE Endpoint

In addition to our existing endpoints, it's crucial to add validation for empty title or content fields in our app.put function, since these fields are required by our database. Revisit your app.put function in index.ts and add similar validation to what we added for the POST request. Specifically, if either title or content is empty, return a 400 status code along with an error message.

With that in place, let's move on to the DELETE endpoint. Add the following code just after your PUT endpoint:

app.delete("/api/notes/:id", async (req, res) => {
  const id = parseInt(req.params.id);

  if (!id || isNaN(id)) {
    return res.status(400).send("ID field required");
  }

  try {
    await prisma.note.delete({
      where: { id },
    });
    res.status(204).send();
  } catch (error) {
    res.status(500).send("Oops, something went wrong");
  }
});

This app.delete function works similarly to the update (app.put) endpoint. It also accepts an ID as part of the URL parameters (query params should be URL parameters or route parameters).

First, we validate that the provided ID is a valid number. If it isn't, we return a 400 status code and an accompanying error message.

Once the ID is validated, we proceed to delete the note using Prisma's delete method. In the try block, we specify which note to delete by its ID in the where object. Upon successful deletion, we return a 204 status code, which indicates 'No Content.' This is a standard way to signal to the frontend or API consumers that the deletion was successful.

If an error occurs during the deletion, the catch block returns a 500 status code along with a generic error message.

To test the new DELETE endpoint, switch your HTTP method to DELETE in your testing tool (like Thunder Client or Postman). Use the ID of the note you wish to delete, such as /api/notes/3, and hit 'Send'. You should receive a 204 status code, indicating the operation was successful. To confirm, perform a GET request on your /api/notes endpoint and observe that the note with the specified ID has indeed been removed.

PART 3 - Connect UI to Backend

Now that we have our backend and UI ready, it's time to connect them. We will do this by using the built in fetch function to call our backend from our UI.

Get and Display Notes

Let's dive back into our frontend code. Just below our state declarations at the top of our component, we're going to introduce a useEffect hook:

useEffect(() => {
  // ...
}, []);

Inside this useEffect, we'll define an asynchronous function named fetchNotes. We need to put this in a separate function because React does not support making the useEffect hook asynchronous directly:

const fetchNotes = async () => {
  // ...
};

To handle any potential errors from the API, we'll wrap our API logic inside a try-catch block:

try {
  // ...
} catch (e) {
  console.log(e);
}

Inside the try block, we use the native fetch function to make an API call. Our API is running at http://localhost:5000/api/notes. By default, fetch performs a GET request, which is what we need:

const response = await fetch("http://localhost:5000/api/notes");

After making the request, we'll process the response and convert it to JSON. The API returns an array of notes, which we'll capture in a variable named notes of type Note[]:

const notes: Note[] = await response.json();

If everything goes smoothly, the next step is to update our state with the notes fetched from the API:

setNotes(notes);

In the catch block, we'll log any errors that may occur:

console.log(e);

We've defined fetchNotes, but haven't called it yet. To invoke this function, add a call to fetchNotes() at the end of the useEffect block:

fetchNotes();

Lastly, add an empty dependency array to ensure that this code only runs once when the component is first mounted:

}, []);

After saving your changes, you should see the notes from your database displayed in the browser. If you've added or deleted notes directly through the database, those changes should be reflected here.

To wrap things up, you can remove any hardcoded array that you initially added to your notes state variable. Instead, populate it with the data fetched from the API:

const [notes, setNotes] = useState<Note[]>([]);

This ensures that the notes state is initially empty, then populated by the useEffect through the fetchNotes function.

Completed Code for this Section

  const [notes, setNotes] = useState<Note[]>([]);


  useEffect(() => {
    const fetchNotes = async () => {
      try {
        const response = await fetch(
          "http://localhost:5000/api/notes"
        );

        const notes: Note[] =
          await response.json();

        setNotes(notes);
      } catch (e) {
        console.log(e);
      }
    };

    fetchNotes();
  }, []);

Save New Note

Next, let's explore how to save a note to our backend. We already have a function called handleAddNote that deals with adding a note to the UI:

const handleAddNote = async (
  event: React.FormEvent
) => {
  // ...
};

To start, remove any code that manually creates a new note object on the frontend. This is because our backend will return this object with all its properties once the note has been saved to the database.

As in our previous example, we'll use a try-catch block to handle the API logic and error handling:

try {
  // API logic here
} catch (e) {
  console.log(e);
}

Place your existing state-changing function calls (setNotes, setTitle, and setContent) inside the try block. These will be executed after the API successfully saves the note:

setNotes([newNote, ...notes]);
setTitle("");
setContent("");

To call the API, we'll use the fetch function, similar to how we fetched notes. The difference is that this time, we need to pass a second argument to fetch to specify the HTTP method and payload:

const response = await fetch(
  "http://localhost:5000/api/notes",
  {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      title,
      content,
    }),
  }
);

Don't forget to add the async keyword to the handleAddNote function signature if you haven't already, as we are using the await keyword inside the function.

The server will respond with the newly created note object, which we can then add to our UI. Convert the response to JSON and store it in a variable named newNote:

const newNote = await response.json();

Finally, in the catch block, we log any errors that might occur:

console.log(e);

Also, make sure to add headers to specify the content type of the data we are sending:

headers: {
  "Content-Type": "application/json",
}

Save your changes and test the functionality in the browser. Use the form to add a new note and click "Add Note." If everything is set up correctly, your new note should appear in the list.

Completed Code for this Section

  const handleAddNote = async (
    event: React.FormEvent
  ) => {
    event.preventDefault();
    try {
      const response = await fetch(
        "http://localhost:5000/api/notes",
        {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            title,
            content,
          }),
        }
      );

      const newNote = await response.json();

      setNotes([newNote, ...notes]);
      setTitle("");
      setContent("");
    } catch (e) {
      console.log(e);
    }
  };

Save Updated Note

Next, let's explore how to save a note to our backend. We already have a function called handleAddNote that deals with adding a note to the UI:

const handleAddNote = async (
  event: React.FormEvent
) => {
  // ...
};

To start, remove any code that manually creates a new note object on the frontend. This is because our backend will return this object with all its properties once the note has been saved to the database.

As in our previous example, we'll use a try-catch block to handle the API logic and error handling:

try {
  // API logic here
} catch (e) {
  console.log(e);
}

Place your existing state-changing function calls (setNotes, setTitle, and setContent) inside the try block. These will be executed after the API successfully saves the note:

setNotes([newNote, ...notes]);
setTitle("");
setContent("");

To call the API, we'll use the fetch function, similar to how we fetched notes. The difference is that this time, we need to pass a second argument to fetch to specify the HTTP method and payload:

const response = await fetch(
  "http://localhost:5000/api/notes",
  {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      title,
      content,
    }),
  }
);

Don't forget to add the async keyword to the handleAddNote function signature if you haven't already, as we are using the await keyword inside the function.

The server will respond with the newly created note object, which we can then add to our UI. Convert the response to JSON and store it in a variable named newNote:

const newNote = await response.json();

Finally, in the catch block, we log any errors that might occur:

console.log(e);

Also, make sure to add headers to specify the content type of the data we are sending:

headers: {
  "Content-Type": "application/json",
}

Save your changes and test the functionality in the browser. Use the form to add a new note and click "Add Note." If everything is set up correctly, your new note should appear in the list.

Completed Code for this Section

  const handleAddNote = async (
    event: React.FormEvent
  ) => {
    event.preventDefault();
    try {
      const response = await fetch(
        "http://localhost:5000/api/notes",
        {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            title,
            content,
          }),
        }
      );

      const newNote = await response.json();

      setNotes([newNote, ...notes]);
      setTitle("");
      setContent("");
    } catch (e) {
      console.log(e);
    }
  };

Save Deleted Note

In this section, we'll discuss how to delete a note by invoking an API endpoint. We'll focus on the deleteNote function for this functionality:

const deleteNote = async (
  event: React.MouseEvent,
  noteId: number
) => {
  // ...
};

First, we need to make our function asynchronous to handle API calls. So, add the async keyword to the function declaration like this:

const deleteNote = async (
  event: React.MouseEvent,
  noteId: number
) => {
  // ...
};

Next, let's add a try-catch block to manage the API call. The catch block is essential for logging errors, which prevents the application from crashing unexpectedly:

try {
  // API logic here
} catch (e) {
  console.log(e);
}

Copy the existing UI-update logic you have and paste it into the try block, right after the API call. This ensures that the UI only updates if the API call is successful.

Now, let's get to the main part—making the API call to delete the note. To do so, we'll use the fetch API:

await fetch(
  `http://localhost:5000/api/notes/${noteId}`,
  {
    method: "DELETE",
  }
);

Note that the URL is a template string. It allows us to inject the ID of the note (noteId) that we want to delete. This noteId is passed into our deleteNote function when the user clicks the delete button corresponding to a specific note.

We specify the HTTP method as "DELETE" to indicate that we're requesting to delete a note:

method: "DELETE",

Unlike the 'add' or 'update' operations, there's no need to assign the API response to a variable, as we're not expecting any data to be returned:

await fetch(
  `http://localhost:5000/api/notes/${noteId}`,
  {
    method: "DELETE",
  }
);

After successfully deleting the note, we filter out the deleted note from our local notes state:

const updatedNotes = notes.filter(
  (note) => note.id !== noteId
);
setNotes(updatedNotes);

Finally, if everything goes smoothly and you save your changes, try running the application in the browser. Click the delete button for a specific note, and then refresh the page. You'll see that the note has been removed successfully.

Completed Code for this Section

  const deleteNote = async (
    event: React.MouseEvent,
    noteId: number
  ) => {
    event.stopPropagation();

    try {
      await fetch(
        `http://localhost:5000/api/notes/${noteId}`,
        {
          method: "DELETE",
        }
      );
      const updatedNotes = notes.filter(
        (note) => note.id !== noteId
      );

      setNotes(updatedNotes);
    } catch (e) {
      console.log(e);
    }
  };

The End - Why not try the bonus challenges?

Congratulations on making it to the end! If you enjoyed this project, I have created a list of additional challenges to try over at codecoyotes.com.

If you have any questions or suggestions feel free to drop me a message here. See you in the next one!

Then, Node.js came to free JavaScript from this confinement. It made JavaScript ubiquitous (meaning it could now be run everywhere).

Node.js is a JavaScript runtime environment that allows JavaScript developers to write command line tools and server side scripts outside of a browser.

Learning Node.js enables JavaScript developers to write server side code and code for embedded systems. This opens up opportunities in backend development and hardware programming.

But, before diving into Node.js, a JavaScript developer has to learn and understand some JavaScript concepts.

In this article, you are going to learn the JavaScript concepts you need to understand before learning Node.js.

Before you continue with the article, check the prerequisites.

Prerequisites

To follow along with this article, you should have some basic knowledge of:

• JavaScript

• The browser console (this is where you will run your code).

Now that we have those out of our way, let's dive into the article, starting with expressions.

Expressions

An expression is a unit or block of code that evaluates to a value.

There are five basic categories of expressions in JavaScript: primary expressions, arithmetic expressions, string expressions, logical expressions, and left-hand side expressions.

Primary expressions

Primary expressions consist of basic keywords in JavaScript. A common example is the this keyword:

this['item'];

this.item;

Later on in this article you will learn more about the this keyword.

Arithmetic expressions

An arithmetic operator takes numerical values as its operands and returns a numerical value. The operands and the operator make up the arithmetic expression:

2 + 3;

2 * 3;

2 ** 3;

String expressions

When strings are concatenated they form string expressions:

console.log('My name is' + 'Peter');

Logical expressions

Logical expressions are expressions in which various values are compared:

10 > 2;
2 < 10;
c === 2 || d === 10;

Left-hand side expressions

Left-hand side expressions are expressions where values are assigned to a variable:

// variables
a = 2;

// object properties
obj = {};
obj.name = 'Paul';

Data Types

There are 8 data types in JavaScript: String, Number, Boolean, Object, Null, Undefined, Symbol, and BigInt.

String

The string type represents textual data. A string is surrounded by a single quote or a double quote.

Each element in a string occupies a certain position in the string. The first element is at index 0, and the second and third are at index 1 and 2 respectively.

Here is an example of a string:

let name = 'Yusuf';
let newName = "Joseph";

Number

The number types are stored in 64-bit format, also known as double-precision floating point numbers.

They consist of numbers that are within the range of -(2⁵³ - 1) and (2⁵³ – 1), with both of these numbers inclusive. These two numbers are known as the MIN_SAFE_INTEGER and the MAX_SAFE_INTEGER respectively.

Numbers that exceed this range, belong to another data type called BigInt.

Here is an example of a positive integer, a float and a negative integer:

let number = 15;
let anotherNumber = 1.5;
let lastNumber = -3;

Boolean

The boolean type is logical, and it has only two values: true or false. It is commonly used in loops and conditional statements.

In the example below, I declared variables and assigned them a value of true and false respectively.

Then, I created a conditional statement that returns 1 if the bool variable is true or -1 if it is false. It returns zero if it is neither true nor false.

let positive = true;
let negative = false;
let bool = false;

// conditional statement
if (bool) {
    return 1;
} else if (bool) {
    return -1;
} else {
    return 0;
}

Object

An object type allows you to store collections of data. The data are stored in a pair of curly braces and a key-value pair format. Object keys must be textual (for example a string).

An object can store any of the other data types, like string, number, arrays, booleans, and so on.

In the example below I created an object named myObject, and gave it three keys with their corresponding values:

let myObject = {
    name: "Gabriel",
    number: 15,
    developer: [true, "Daniel", 1]
}

Null

Null data types are special data types. They have the value null which means that the value is unknown or empty:

let unknown = null;

Undefined

Unlike null, undefined means that a variable is declared and not assigned a value. A variable can also be assigned undefined specifically:

let name = undefined;

let newName;
console.log(newName);

Symbol

Symbols are used to create unique values. They are created by calling the Symbol() function. Every time the Symbol() function is invoked it creates a new unique value.

Symbols are kept hidden or private and they can only be used internally. For instance, they can be used as keys in an object. When you try to get the list of keys in an object where a symbol is part of its keys the symbol key will not be displayed.

You can pass a parameter to the symbol function to serve as a description for the symbol when debugging it in the console:

let firstSymbol = Symbol();

let anotherSymbol = Symbol(anotherSymbol);

BigInt

BigInt is a special type of number that provides support for numbers with a length that a normal number type can not hold (such as numbers that exceed the safe integer limit).

It can be created by appending n to the end of an integer or passing a number into the BigInt function:

let bigNumber = 175337823472347884278247898427427427642n;
let newBigNumber = BigInt(1624743724724724898718247248744774227422n);

let anotherBigNumber = BigInt(14);

Classes

A JavaScript class is a template for creating objects. It contains data and functions that manipulate data.

Classes were introduced to JavaScript in the ECMAScript 2015 (ES6) version of JavaScript.

The functions used in classes are called methods.

There is a basic syntax for declaring classes looks something similar to the following:

class TemplateClass {
    constructor() {...};
    method() {...};
    method() {...};
}

From the syntax you can create a class named Visitor:

class Visitor {
    constructor(name) {
        this.name = name;
    }
    introduce() {
        console.log(`My name is ${this.name} and I am a visitor`)
    }
}

You can create a new class from this class by using the new class syntax. The newly created class can access any method from its parent class:

let visitor = new Visitor("Jeff");

// visitor can call the introduce method in its parent class.
visitor.introduce();

Variables

A variable is a named storage for JavaScript data. JavaScript variables can be declared in three ways:

• Using the var keyword

• Using the let keyword

• Using the const keyword

How to declare variables using the var keyword

Variables declared with the var keyword are function scoped and they can be redeclared:

var num = 1;

// num can be redeclared
var num = 2;

How to declare variables using the let keyword

Variables declared with the let keyword are block scoped and they cannot be redeclared – they can only be reassigned:

let fruit = "banana";

// fruit can only be reassigned
fruit = "orange";

How to declare variables using the const keyword

Variables declared with the const keyword are block scoped and they cannot not be redeclared or reassigned (meaning they are constant):

const bestStudent = "Daniel";

Functions

A function is a block of code that performs a specific task. It can be declared using the function keyword:

function doSomething() {
    return "Does Something";
}

A function takes inputs called arguments and outputs results.

To execute a function you will invoke the function by calling the function name with enclosed brackets:

function addNumbers(a, b) {
    return a + b;
}

addNumbers();

You can assign a function to a variable and call the variable when you want to invoke the function:

function addNumbers(a, b) {
    return a + b;
}

let numberAddition = addNumbers(2, 3)
numberAddition();

Arrow Functions

Arrow functions are a concise and compact way to write a function. They have some deliberate limitations in their usage:

• They cannot be used as a method.

• They cannot be used as a constructor.

• Arrow functions cannot use yield within their own bodies.

Below is the syntax for an arrow function:

const doSomething = () => {
    return "Do something";
}

An arrow function can also take an argument.

const multiplyNumbers = (a, b) => {
    return a * b;
}

this keyword

The this keyword in JavaScript refers to an object that is executing a function or a code.

The this keyword can be used in different contexts – the context in which the this keyword is used determines what it refers to.

The this keyword can be used:

• In an object method.

• Alone.

• In object method bindings.

How to use the this keyword in an object method

The this keyword refers to the object whenever it is used as an object method:

intro : function() {
    return "My name is" + this.name "and, I am a" + this.occupation;
}

How to use the this keyword alone

When this is used alone it refers to the global object:

let wes = this;

How to use the this keyword in object method bindings

When this is used in an object method, it refers to the object:

let student = {
  firstName  : "Juliana",
  lastName   : "Carpe",
  myFunction : function() {
    return this;
  }
};

Loops

Loops are useful for executing a block of code for a specified number of times based on some specified conditions. There are different types of loops in JavaScript:

• for loops

• for ... in loops

• for ... of loops

• while loops

• do ... while loops

How to use for loops

for loops are used to execute a block of code a number of times:

for (let i = 0; i < 5; i++) {
    return i;
}

How to use for ... in loops

for ... in loops are used to loop through the properties of an object:

for (let prop in obj) {
    return obj.prop;
}

How to use for ... of loops

for ... of loops are used to loop through the values of iterable objects like arrays, strings, maps, and so on:

let numArr = [2, 4, 6, 8]
for (let val of numArr) {
    return val ** 2
}

How to use while loops

while loops are used to execute a block of code while a certain condition still holds true:

while (i < 20) {
    return i;
    i++;
}

How to use do ... while loops

do ... while loops execute a block of code first without any conditions. For as long as a certain condition still holds true it continues to execute the block of code:

let i = 3;
do {
    return i;
    i++;
} 
while (i < 4)

Scopes

The scope is the current context of execution. It is where variables and expressions can be accessed.

There is an hierarchical arrangement of scope in JavaScript. This makes it possible for lower scopes to access higher scopes.

The scopes in JavaScript are:

• Global scope

• Module scope

• Function scope

• Block scope

The global scope is the default scope for all codes running in script mode while the module scope is the default scope for all codes running in module mode.

The function scope is created by functions while the block scope is created by variables.

Here is an example of function and block scope:

// function scope
function introduce(name) {
    let age = 12;
    console.log(`My name is ${name}`);
    console.log(`I am ${age} years old`);
}

let firstAge = 13

// block scope
if (firstAge === 13) {
    let secondAge = 20;
    console.log(`I am ${secondAge} years old`);
}

Arrays

An array is a special type of an object that stores data in an ordered form.

Arrays can be declared in two ways:

• Using square brackets.

• Using the the Array() constructor.

How to declare arrays using square brackets

This is the common way of creating an array. It is done by wrapping the array's items in a pair of square brackets:

  let array = [1, 3, "a", "c"];

How to declare arrays using the the Array() constructor

The Array() constructor does the same thing as the bracket notation. It can be called with or without the new keyword:

  let anotherArray = Array(1, 2, 3, "go");

How to access array elements

Array elements can be accessed in three ways:

• using their index.

• using the array's length property.

• using a loop

How to access an array element using its index

You call the name of the array with a square bracket containing the index you want to access:

 let newArray = ["Idris", "Daniel", "Joseph"];

  // To access first element
  let firstElement = newArray[0]; 
  console.log(firstElement);  // Idris

  // To access second element 
  let secondElement = newArray[1];  
  console.log(secondElement);  // Joseph

How to access an array element using the array's length property

You can get the length of the array using the length property. Then, you will subtract any number from it to get the index of the element you want to access:

let newArray = ["Idris", "Daniel", "Joseph"];

  let length = newArray.length;

  let firstElement = newArray[length - 3]; 
  console.log(firstElement);  // Idris

  let secondElement = newArray[length - 2];
  console.log(secondElement);  // Joseph

How to access an array element using a loop

Array elements can be accessed using loops. You can make use of a for loop, while loop, or a for ... of loop:

  let newArray = ["Idris", "Daniel", "Joseph"];
  for (let i = 0; i < newArray.length; i++) {
      return newArray[i]
  }

Important Array methods

There are over fifteen array methods in JavaScript. In this article, you will learn the ones that are most useful in Node.js:

• push() and pop()

• shift() and unshift()

• map()

• sort()

• forEach()

How to use the push() and pop() methods

The push() method is used to add an item to the end of an array, while the pop() method is used to remove an item from the end of an array:

let arr = [1, 2, 3, 9]

arr.push(21);

console.log(arr)    // [1, 2, 3, 9, 21]

arr.pop()

console.log(arr)    // [1, 2, 3, 9]

How to use the unshift() and shift() methods

The unshift() method is used to add an item to the beginning of an array, while the shift() method is used to remove an item from the beginning of an array:

let arr = [1, 2, 3];

arr.unshift(0);

console.log(arr);    // [0, 1, 2, 3]

arr.shift();

connsole.log(arr);    // [1, 2, 3]

How to use the map() method

The map() method iterates through the elements of an array and calls a function on each element of the array. It returns a new array that contains the result of each function call:

let fruits = ["Apple", "Grape", "Cashew"];

let mappedFruits = fruits.map(item => item + "s");
console.log(mappedFruits); // ["Apples", "Grapes", "Cashews"]

How to use the sort() method

The sort() method sorts an array in place and returns the same array in a sorted form.

The default order of the sort() method is ascending order. Strings are sorted in alphabetical order:

let numbers = [8, 7, 5];
let fruits = ["Apple", "Grape", "Cashew"];

let sortedNumbers = numbers.sort();
let sortedFruits = fruits.sort()

console.log(sortedNumbers);  // [5, 7, 8]
console.log(sortedFruits);  // ["Apple", "Cashew", "Grape"]

How to use the forEach() method

The forEach() method loops through the array and calls a function for every element of the array:

let fruits = ["Apple", "Grape", "Cashew"];

fruits.forEach(fruit => console.log(fruit));
//  "Apple"
//  "Grape"
//  "Cashew"

Template Literals

Template literals are enclosed in backticks, just like strings are enclosed in quotes. They allow you to store multiline strings and also interpolate strings with embedded expressions.

The example below shows a basic template literal:

let basic = `I write codes`

You can write a template literal that stores multiline strings like this:

let multiLine = `I write codes                     
        I debug codes`;

You can use the dollar sign and curly braces to embed expressions in template literals.

In the example below, the function myName is embedded in the display variable with a template literal:

function myName(Musab, Habeeb) {        
    alert("Musab Habeeb");    
}    

let display = `This displays my name ${myName()}`

Strict mode

JavaScript is a sloppy language in the sense that it allows you to write code that is not allowed in other languages. Some of the code you write in JavaScript has errors but, JavaScript does not throw an error.

Strict mode does the following:

• It throws errors for JavaScript silent errors.

• It fixes mistakes that makes it difficult for JavaScript engines to perform optimizations.

• It prohibits some syntax likely to be defined in future versions of the ECMAScript.

Strict mode works in an entire script file and functions, but it does not work in block scopes.

To invoke script mode you will add the "use strict"; syntax to the top of the code you want to apply it to. You can apply strict mode to a script like this:

"use strict";

let name = "Musab";

console.log(name);

Also, you can apply strict mode to a function like this:

function sayHi(name) {
    "use strict";
    console.log(`Hi ${name}`);
}

Conclusion

Finally, you are done with learning the JavaScript concepts you need to understand before learning Node.js.

All these concepts are important concepts that a JavaScript Developer aspiring to learn Node.js should understand. Understanding these concepts will make learning Node.js easier.

But, to understand these concepts in depth, you can do more research on each of them and read other articles. Keep learning, keep building.

A wise man once told me, "When you start eating Bun, Node.js will feel bland".

But why is that relevant? JavaScript got way faster with a new JavaScript runtime called Bun, which is now production-ready with its version 1.0 release.

But how and why is it faster than Node.js? A lot of questions come to mind.

I'll answer some of those questions in this article. And I'll do that quickly as I am now faster just like my pal, JavaScript, cooking in the oven with Bun 1.0.

Bun is a fast, all-in-one toolkit for running, building, testing, and debugging JavaScript and TypeScript, from a single file to a full-stack application.

Here are some things we can do with Bun.

Run your Code Faster with Bun

Now, we don’t need tools like npm, pnpm, or yarn because Bun is 17 times faster. Take a look at the data below:

Bun takes only 0.36 seconds to compile your code, whereas it takes about 6.44 seconds in the case of pnpm, 10.58 seconds with npm, and 12.08 seconds with yarn.

Bun Supports Hot Reloading

Bun supports hot reloading out of the box so you don’t need tools like Nodemon. It will automatically refresh the server when running JavaScript or TypeScript Code.

You can replace npm run with bun run to save over 150ms milliseconds every time you run a command.

Here is the full chart:

From the chart above, using npm takes about 176ms to run, yarn takes about 131ms. In the case of pnpm, it takes 259ms. However, it takes about 7ms in the case of Bun. That's fast, isn't it?

Bun as a JavaScript Bundler

Bun is also a JavaScript bundler with best-in-class performance and an ESBuild-compatible plugin API, so we don’t need things like:

• ESBuild
• Webpack
• Parcel, .parcelrc
• Rollup, rollup.config.js

Bun now supports Next.js, Remix, Nuxt, Astro, SvelteKit, Nest, SolidStart, and Vite.

Bun has both ESM and CommonJS Compatibility

Another great feature about Bun is that we can use ES Modules and CommonJs together in the same file, which was not possible in Node.js.

You can use import and require() in the same file:

import lodash from "lodash";
const _ = require("underscore");

Apart from that, Bun has built-in support for the Web standard APIs that are available in browsers, such as fetch, along with extra Bun APIs like Bun.file() to lazy read a file and Bun.Write() to write a file to the local file system which is a lot simpler than Node.js.

Bun.file() Example

const file = Bun.file("package.json");
const contents = await file.text();

The code above will read the contents of a package.json file and transfer its content to a new variable called contents.

Bun.write() Example

await Bun.write("index.html", "<html/>");
await Bun.write("index.html", Buffer.from("<html/>"));
await Bun.write("index.html", Bun.file("home.html"));
await Bun.write("index.html", 
await fetch("https://example.com/"));

In the code above, Bun.write() will write the string "<html/>", or copy the contents of home.html file into the index.html file. If we have to fetch data, it will fetch the results from an external web API and write the contents to a index.html file.

Why is Bun so fast?

Bun is fast because it uses the JavaScriptCore engine, while Node.js uses the JavaScript V8 engine. The former has been optimized for faster startup time.

If you want to get things done faster, you should consider replacing Node.js with Bun.

How to Get Started With Bun

You cam install Bun on MacOS and Linux systems using npm:

npm install -g bun

Now you are all set. To install a npm package, do this:

bun install <package-name>

To start a Next.js app, do this:

bun run dev

All you need to do is replace npm with bun.

However, Bun is only ready for production in MacOS and Linux operating systems. The Windows version is still experimental. At the moment, only the JavaScript runtime is supported for Windows, and not the package manager, the bundler, or the test runner. You can read more about that here.

Conclusion

This article shows how you can use Bun as a Node.js alternative and speed up your development time.

You can also check out my video on The Node.js killer is here — Bun 1.0 First Look.

Thanks for reading!

Hello everyone 👋 In this tutorial, you'll learn how to make a simple Task Manager CLI (Command Line Interface) tool. This means you can use commands to Create, View, Update, or Delete your todos.

We will be building this CLI tool using NodeJS. We'll also use MongoDB as the database to store all our to-dos. Finally, we'll use a few helpful packages from npm:

• commander: This helps us to build the CLI tool.
• chalk: This makes messages in the terminal colorful and easy to read.
• inquirer: This lets us ask the user for input.
• ora: This makes the terminal show nice spinning animations.

Before we dive in, I want you to know that you can find the complete code for this project on GitHub. If you're ever unsure about something in the code, you can always refer to the final version there.

Here is the link to the Repository: Task Manager CLI Tool Repo.

Table Of Contents:

• Project Setup
• How to create the package.json file
• How to install dependencies
• How to convert CommonJS modules to ES modules
• How to create the folder structure
• How to Connect to the Database
• How to obtain a MongoDB connection string
• Code for connecting to the database
• How to Create a Mongoose Model
• Working on CRUD Operations
• How to create Todos
• How to reading Todos
• How to deleting Todos
• How to update Todos
• How to Write the CLI Entry Point using Commander
• How to Test the CLI tool
• Conclusion

Project Setup

Welcome to the first section of this handbook! Here, we will be setting up our project.

This involves a few simple steps: creating a new directory, setting up the package.json file, and installing necessary npm packages like chalk, inquirer, commander, and others we'll talk about soon. We'll also organize the project by creating folders.

Before we dive in, let's ensure you have NodeJS installed on your system. You can get the latest LTS version from this website: https://nodejs.org/en.

To check if Node is properly installed, type this command: node --version. If you see a version number, you're all set! If not, you need to troubleshoot the errors.

Once NodeJS is up and running, create a new folder named "todo." You can use your favorite code editor (I prefer Visual Studio Code) or follow these steps in your terminal:

1. Make a new folder: mkdir todo
2. Go inside the folder: cd todo
3. Open it in your code editor: code .

How to Create the package.json File

The first and foremost step is setting up the package.json file. But don't worry about doing it manually. You can save time by using this command:

npm init --yes

Once this step is done, let's move on to the next step and get all the necessary things for our project.

How to Install Dependencies

To build this project, we'll need some packages. Just run this simple command to get them all:

npm i commander inquirer chalk ora mongoose nanoid dotenv

How to Convert CommonJS Modules to ES Modules

Before you continue, let's make a little change in the package.json file. Remove this line: "main": "index.js", and add these two lines instead:

"exports": "./index.js",
 "type": "module",

With these changes, we're converting our project from CommonJS Modules to ES Modules. This means we'll use import instead of require() to bring in modules, and export instead of module.exports to share things between files.

If you want to dive deeper into different types of modules in JavaScript and how they work, check out this tutorial on FreeCodeCamp: Modules in JavaScript - CommonJS and ESmodules Explained.

How to Create the Folder Structure

Now, let's organize our project by setting up a smart folder structure. This means we'll make folders to neatly hold our JavaScript files. This step's really important. It makes things easy to manage and develops smoothly.

We're creating 3 folders and 2 files in the main folder:

First folder: commands. Inside this folder, you'll create 4 files. The names of the files and the description of the code they will contain is mentioned below:

• addTask.js: Code for Creating a new todo.
• deleteTask.js: Code for Deleting a todo.
• readTask.js: Code for Displaying all the todos.
• updateTask.js: Code for Updating a todo.

Second folder: db. Inside this folder, add a file named connectDB.js. This file will contain the code for connecting to the MongoDB database and disconnecting when needed.

Third folder: schema. Inside it, make a file named TodoSchema.js. This file stores the Mongoose Schema and Model. Basically, a blue print for our tasks, that is how our tasks will look.

First file: .env. Create this file inside the root directory / main folder of the project. This is where you'll put your MongoDB Connection string.

Second file: Create the index.js file in the root directory itself which will serve as the entry point of our project. It's like the project's front – where everything starts.

Once we are done, your project's folders should look something like this:

Project folder structure

How to Connect to the Database

Now that you've successfully set up the project, it's time to dive into the exciting part.

How to Obtain a MongoDB Connection String

To keep track of all our todos, we need a place to store them. That's where MongoDB Atlas comes in. It's like a special service that handles databases for us. The best part? You can start using it for free (no credit card needed).

To connect to it, all you need is something called a connection string. If MongoDB Atlas is new to you, don't worry. Check out this easy-to-follow article: MongoDB Atlas Tutorial - How to Get Started. It gives you just enough info to start using Atlas. By the time you're done, you'll know how to get what you need, including the connection string.

Once you have that connection string, create a new thing called an "environment variable." It's like a secret code your project uses. Open the .env file and make a line like this: MONGO_URI=. After the =, put in your connection string.

Remember: Replace <password> with your actual password and <username> with the username of your database admin in the connection string. Also, add todos between /? in the string. When you're done, your .env file should look something like this:

MONGO_URI=mongodb+srv://<username>:<password>@cluster0.k5tmsld.mongodb.net/todos?retryWrites=true&w=majority

Code for Connecting to the Database

Now, let's dive into the code that connects our tool to the MongoDB database. Open up the ./db/connectDB.js file and let's write some code to make this connection happen.

First things first, we need to bring in the dotenv package that we grabbed earlier when we were setting up the project and invoke the config() method on dotenv. This helps us load environment variables from the .env file. Here's how you do it:

import dotenv from 'dotenv'
dotenv.config()

Next up, we want to import a few more packages that we'll use here. These are mongoose, ora, and chalk:

import mongoose from 'mongoose'
import ora from 'ora'
import chalk from 'chalk'

Note: mongoose is an Object Data Modeling (ODM) library for MongoDB. It provides a higher-level abstraction making it easier to do things like adding, reading, updating, and deleting stuff from the MongoDB database.

Now, let's get into the real action. We'll define two functions here: connectDB() and disconnectDB().

The connectDB() function will contain the code to help connect our NodeJS Application to the M0ngoDB Database using mongoose. It's like a phone call connecting them. If we don't establish a connection first, our app won't be able to interact with the database and perform the various CRUD operations.

The disconnectDB() function does the opposite. It's like hanging up the phone after our app is done talking to the database. If we don't disconnect, it's like keeping the call going even after we're done.

Failing to disconnect from the database after we are done interacting with it could cause resource leaks. These may cause your app to slow down or potentially crash over time.

Let me show you the code for both the functions:

export async function connectDB(){
    try {
        const spinner = ora('Connecting to the database...').start()
        await mongoose.connect(process.env.MONGO_URI)
        spinner.stop()
        console.log(chalk.greenBright('Successfully connected to database!!!'))   
    } catch (error) {
        console.log(chalk.redBright('Error: '), error);
        process.exit(1) 
    }
}

export async function disconnectDB(){
    try {
        await mongoose.disconnect()
        console.log(chalk.greenBright('Disconnected from the database.'))
    } catch(err) {
        console.log(chalk.redBright('Error: '), error);
        process.exit(1) 
    }
}

This is a lot of code to digest at one time, so let me explain this for you:

In the connectDB() function, the line mongoose.connect(process.env.MONGO_URI) helps us actually connect to the database using the connection string.

Remember the .env file? We're using its info here. To load the MONGO_URI variable, we use the dotenv package and call the config() function and then we can access it using process.env.MONGO_URI.

Since mongoose.connect() returns a promise, we use the await keyword before it to make sure we proceed only when this returned promise gets resolved.

It is possible to encounter some errors while running this code, so we have wrapped the entire code in a try...catch() block to make sure any errors which pop up are handled properly in the catch() block.

The ora package helps us show a spinner while we connect to the database. Once successfully connected, we stop the spinner and show a happy message in green using chalk.

If you notice, we are doing the same thing in the disconnectDB() function. But, instead of connecting, we disconnect from the database using mongoose.disconnect(). We wrap it in a similar try-catch block, and again we show colorful messages using chalk.

We use export before these functions to let other parts of the project use them. Don't forget to add these two temporary lines at the end of the file for now:

connectDB()
disconnectDB()

Now, you can run the connectDB.js file using the command: node ./db/connectDB.js and expect to see this in the console:

Output seen on the terminal when connectDB.js file is executed. It shows how our code successfully connects to the database and disconnects from it showing appropriate console messages when we invoke the connectDB() and disconnectDB() methods.

Connecting to the database is a big step, but you're making great progress! Before moving ahead, make sure you remove those 2 lines you added at the end because they were added just to check if our connection and disconnection functions are working as expected.

How to Create a Mongoose Model

A Mongoose model is like a tool that helps us talk to the database. With it, we can easily do things like add, read, update, and delete tasks. It's like a helpful assistant that understands how to communicate with the database.

To make this model, we need something called a Schema. It basically defines what each task should look like. Think of it as a blueprint or a set of instructions that guides how each task is created, what information it should have, and how that information is organized. It's like setting rules for how our tasks are stored in the database.

We're going to build this Schema in the ./schema/TodoSchema.js file. Open it up, and let's dive in. First, we need two special tools: mongoose and nanoid. We'll use nanoid to make short and unique IDs for each task.

Type these lines to import the tools:

import mongoose from 'mongoose'
import {nanoid} from 'nanoid'

Now, we use the mongoose.Schema() method to create our Schema. Here's the code for it:

const TodoSchema = new mongoose.Schema({
    name: {
        type: String,
        required: true,
        trim: true
    },
    detail: {
        type: String,
        required: true,
        trim: true
    },
    status: {
        type: String,
        required: true,
        enum: ['completed', 'pending'],
        default: 'pending',
        trim: true
    },
    code: {
        type: String,
        required: true,
        default: 'code',
        trim: true
    }
}, {timestamps: true})

Any task created using this Schema will have the following properties:

• name: This is a short title for the task. The type: String emphasizes that it can only be text (a String). The required: true specifies that we have to provide this when creating a task and the trim: true specifies that any extra spaces at the beginning or at the end of the task's name will be removed before saving it in the database.
• detail: This is a description of the task. It has exactly the same properties as name.
• status: This shows if the task is done or not. The enum: ['completed', 'pending'] property specifies that it can only be completed or pending. The default: 'pending' property specifies that if you do not set the status property while creating the task, it is assumed to be pending.
• code: This is a short and unique ID for the task. We are giving it a default value of code. This value is just a placeholder and doesn't have any real significance in terms of identifying the task. Don't worry, we'll change it soon.
• The {timestamps: true} is a configuration option that automatically adds timestamp fields like createdAt and updatedAt to the tasks when they are created or modified.

We have successfully defined our Schema – but you may wonder if the code property was supposed to be unique for every task. Currently it stores the same value, that is "code", for every single task. Don't worry, we'll fix that. Add this code at the end:

TodoSchema.pre('save', function(next){
    this.code = nanoid(10)
    next()
})

Here, TodoSchema.pre('save', function(){....}) helps us defining a pre-save hook/function which runs every time before a task gets saved in the database.

Inside the function, we use nanoid(10) to create a unique, 10 character long ID for the task and put this generated id in the code field of the task (we can actually access any property/field of the task using the this keyword).

The last line of code: next() basically tells the computer that we are done and it can finally save the document now. With this, we generate a unique ID for every single task created using the nanoid package.

Lastly, we'll make a Todos model using this TodoSchema blueprint and export it. This is how:

const Todos = mongoose.model('Todos', TodoSchema)
export default Todos

And there you go! We've built our Schema and Model. Now let's proceed to the next section of this tutorial.

How to Work on CRUD Operations

Congrats for having successfully followed up till here. So far, we have done 3 things:

1. We've set up the project
2. We've connected to the MongoDB database, and
3. We've created the Mongoose Model

Up next, we will be working on the various CRUD operations like Creating, Reading, Updating, and Deleting the tasks from our database.

How to Create Todos

Now, let's start with creating tasks in our project. At first, I planned for a simple process where you add one task to the database at a time. This means when you tell the tool to create a task, it asks you once for the task's details—like the name and description—and then it saves the task.

But then I realized, what if someone wants to add many tasks quickly? Doing it one by one isn't cool. There are two issues:

1. If you have, say, 5 tasks in your mind, you'd have to type the create command 5 times—one for each task.
2. After entering task details, you wait a bit because saving stuff to the database can take time, especially if the internet is slow.

These problems aren't fun at all! To fix this, we need a way to add multiple tasks in one go. Here's how we'll do it:

After you put in the task's name and description, we'll ask if you want to add more tasks. If you enter yes, we continue the process from the start (asking you to again enter the name and description of the next task). But if you enter no, the question prompting process will stop and all the entered tasks get saved to the database together. This way, you can create many tasks without the hassle of doing it one by one. It's all about making things smooth for you.

We'll be writing some code in the ./commands/addTask.js file. This is where the magic happens. Let's break it down step by step:

First, we import the necessary packages and functions we've created earlier. You can add these lines of code to do that:

import inquirer from "inquirer";
import { connectDB, disconnectDB } from '../db/connectDB.js'
import Todos from "../schema/TodoSchema.js";
import ora from "ora";
import chalk from "chalk";

Now, we create an asynchronous function called input() to gather the task's name and details from the user. Here's how it goes:

async function input(){
    const answers = await inquirer.prompt([
        { name: 'name', message: 'Enter name of the task:', type: 'input' },
        { name: 'detail', message: 'Enter the details of the task:', type: 'input' },
    ])

    return answers
}

In simple terms, input() uses inquirer to ask the user for the task's name and details. The answers are then returned as an object.

But wait, you might wonder what inquirer.prompt() is doing. It's a method in the inquirer package that asks questions and waits for responses. You provide an array of question objects, each containing details like the message to display to the user and the type of question. The function returns a Promise, so we use await to wait for the user's answers which get's returned as an object.

Here, { name: 'name', message: 'Enter name of the task:', type: 'input' } is the first question that will be asked to the user. The message property contains the question that will be displayed to the user. In our case, it is: Enter the name of the task. The user will be prompted to input some text (a String), since this question is of type: 'input'. The name: 'name' signifies that the user's answer to this question will be assigned to a property named – name in the answer's object.

The next object is the second question that will be asked to the user. In this case, a message will be displayed in the terminal: Enter the details of the task and the user's response will be assigned to a property called detail in the answer's object.

To see how the above code works, you can add these 2 lines of code at the end of the file:

const output = await input()
console.log(output)

Now, save the file and run the code using the command: node ./commands/addTask.js. This is what you will see when you run the code:

Output seen on terminal when we just invoke the input() method and execute the code. It shows how inquirer.js returns user's answers after the question prompting process.

We can now proceed with the rest of the code and you can remove the last 2 lines which you just added.

Now, let's create a function named askQuestions() to gather multiple tasks. This is how it looks:

const askQuestions = async() => {

    const todoArray = []
    let loop = false

    do{
        const userRes = await input()
        todoArray.push(userRes)
        const confirmQ = await inquirer.prompt([{ name: 'confirm', message: 'Do you want to add more tasks?', type: 'confirm' }])
        if(confirmQ.confirm){
            loop = true
        } else {
            loop = false
        }
    } while(loop)

    return todoArray
}

In askQuestions(), we set up a loop that keeps asking for tasks until the user decides to stop. We gather each task from the user by calling the input() function, and the returned user's response gets pushed to the todoArray.

Then, we ask if the user wants to add more tasks using a confirmation question. If they say yes, we set loop to true and the loop continues – otherwise, loop becomes false, and the loop ends. Finally, we return the array of tasks, that is todoArray.

You can test this by adding these lines of code at the end of the file:

const output = await askQuestions()
console.log(output)

When you run the file using node ./commands/addTask.js, you'll see a similar result to what you see here:

Output seen on terminal when we invoke the askQuestions() method and execute the code. It shows the array of tasks returned by the method when the user does not want to continue adding more tasks.

We are almost there! Before proceeding, do not forget to remove the last 2 lines you added just now. After you've done that, let's move ahead.

Up to this point, we have successfully managed to collect all the tasks the user wants to create.

Now, let's define the last piece of the puzzle: the addTask() function. This function brings everything together and completes the task creation process. Here's the full code:

export default async function addTask() {
    try {
        // calling askQuestions() to get array of todo's
        const userResponse = await askQuestions()

        // connecting to the database
        await connectDB()

        // Displaying a spinner with the following text message using ora 
        let spinner = ora('Creating the todos...').start()

        // looping over every todo in the userResponse array
        // and saving each todo in the database
        for(let i=0; i<userResponse.length; i++){
            const response = userResponse[i]
            await Todos.create(response)
        }

        // Stopping the spinner and displaying the success message
        spinner.stop()
        console.log(
            chalk.greenBright('Created the todos!')
        )

        // disconnecting the database
        await disconnectDB()
    } catch (error) {
        // Error Handling
        console.log('Something went wrong, Error: ', error)
        process.exit(1)
    }
}

The addTask() function starts by calling the askQuestions() function to gather the array of tasks and assigning it to the userResponse variable. Then, it connects to the database using connectDB(), displays a spinner using ora to show the task creation process, loops through each task in the array, and saves it to the database using Todos.create(response).

Once all tasks are saved, the spinner stops, a success message is shown, and then it disconnects from the database using disconnectDB().

The entire code is wrapped in a try...catch block to handle any potential errors gracefully.

With this code, you've completed the task creation process. Nice work! This was probably the most complex piece of code in the entire project. Future operations such as reading, deleting and updating tasks are going to be fairly simple and easy by comparison. With that said, let's move to performing the Read operation.

How to Read Todos

Now we'll explore how to read tasks from the MongoDB database. The process is straightforward, and I'll guide you through the entire code in the ./commands/readTask.js file:

First, let's import the necessary packages and functions at the beginning of the file:

// Importing packages and functions
import { connectDB, disconnectDB } from '../db/connectDB.js'
import Todos from '../schema/TodoSchema.js'
import chalk from 'chalk'
import ora from 'ora'

Now, let's define an asynchronous function named readTask() which encapsulates the logic for reading tasks. The whole function is wrapped in a try...catch block for handling any potential errors:

export default async function readTask(){
    try {
        // connecting to the database
        await connectDB()

        // starting the spinner
        const spinner = ora('Fetching all todos...').start()

        // fetching all the todos from the database 
        const todos = await Todos.find({})

        // stopping the spinner
        spinner.stop()

        // check if todos exist or not
        if(todos.length === 0){
            console.log(chalk.blueBright('You do not have any tasks yet!'))
        } else {
            todos.forEach(todo => {
                console.log(
                    chalk.cyanBright('Todo Code: ') + todo.code + '\n' + 
                    chalk.blueBright('Name: ') + todo.name + '\n' + 
                    chalk.yellowBright('Description: ') + todo.detail + '\n'
                )
            })
        }

        // disconnect from the database
        await disconnectDB()
    } catch (error) {
        // Error Handling
        console.log('Something went wrong, Error: ', error)
        process.exit(1)
    }
}

readTask()

Now, let's break down the code step by step:

1. We establish a connection to the MongoDB database using await connectDB().
2. We start a spinner using ora to indicate that we are fetching all the todos.
3. We fetch all the todos from the database using Todos.find({}). Once the process is complete, the todos variable will contain either an empty array (if no tasks exists in the database) or an array of tasks.
4. After fetching is complete, we stop the spinner using spinner.stop().
5. We check whether there are any todos by checking if todos.length is equal to 0. If it is, we display a message in blue saying "You do not have any tasks yet!". If there are todos in the array (which means that the length of the array is not equal to 0), we loop through each todo in the array and print its code, name, and description using chalk for color formatting.
6. Finally, we disconnect from the database using await disconnectDB().

In the last line of code, we call the readTask() function. This is only for testing purposes, and you can remove this line as instructed.

To run the code, use the command: node ./commands/readTask.js. When you execute this, you'll see something similar to the output shown here:

Note: I had created some random tasks before, so when I run the readTask.js file, I get to see this in my terminal:

Output seen on terminal when readTask.js file is executed. It shows how the code successfully reads all the tasks from the database and prints it out in the terminal.

Before we proceed, don't forget to remove the last line of code in the readTask.js file because we won't be needing it in the future.

With this code, you've successfully implemented the read functionality for your task manager CLI tool. Great job! In the upcoming sections, we'll explore how to delete and update tasks.

How to Delete Todos

This section of the tutorial covers the simple process of deleting todos from the database. The logic is straightforward: users enter the Todo code of the todo they want to delete, and we remove that todo from the database.

Let's delve into the code to make this happen in the ./commands/deleteTask.js file.

The first step is to import necessary packages and functions at the beginning of the file, including inquirer, Todos model, connectDB(), disconnectDB(), ora, and chalk.

// Importing packages and functions
import inquirer from "inquirer";
import Todos from '../schema/TodoSchema.js'
import {connectDB, disconnectDB} from '../db/connectDB.js'
import ora from "ora";
import chalk from "chalk";

Up next, we will define an asynchronous function called getTaskCode(). The role of this function is to prompt the user to enter the code of the todo they want to delete using inquirer. The function then trims the code entered by the user using the trim() method and returns the trimmed code. The trimming process is necessary to remove the leading or trailing whitespace which the code might contain.

Here is the code for the getTaskCode() function:

export async function getTaskCode(){
    try {
        // Prompting the user to enter the todo code
        const answers = await inquirer.prompt([
            {name: 'code', 'message': 'Enter the code of the todo: ', type: 'input'},
        ])

        // Trimming user's response so that the todo code does not contain any starting or trailing white spaces
        answers.code = answers.code.trim()

        return answers
    } catch (error) {
        console.log('Something went wrong...\n', error)
    }
}

Now we will define the main function named deleteTask(). The entire code is below:

export default async function deleteTask(){
    try {
        // Obtaining the todo code provided by user
        const userCode = await getTaskCode()

        // Connecting to the database
        await connectDB()

        // Starting the spinner
        const spinner = ora('Finding and Deleting the todo...').start()

        // Deleting the task
        const response = await Todos.deleteOne({code: userCode.code})

        // Stopping the spinner
        spinner.stop()

        // Checking the delete operation
        if(response.deletedCount === 0){
            console.log(chalk.redBright('Could not find any todo matching the provided name. Deletion failed.'))
        } else {
            console.log(chalk.greenBright('Deleted Task Successfully'))
        }

        // Disconnecting from the database
        await disconnectDB()
    } catch (error) {
        // Error Handling
        console.log('Something went wrong, Error: ', error)
        process.exit(1)
    }
}

Let's break down this code step-by-step:

1. We obtain the response object which includes the todo code entered by the user by calling the getTaskCode() function defined above. We then assign this object to the userCode variable.
2. We connect to the database using await connectDB().
3. We start a spinner using ora to indicate that we're finding and deleting the todo.
4. We use Todos.deleteOne({ code: userCode.code }) to search for and delete the todo with a matching code. The response will indicate if any document was deleted or not.
5. After the operation is complete, we stop the spinner using spinner.stop().
6. We use an if...else condition to check the deletedCount property in the response. If it's 0, we print a message indicating that the task with the provided code wasn't found and deletion failed. If deletedCount is greater than 0, we print a success message.
7. We disconnect from the database using await disconnectDB().

If I call the function: deleteTask() and then proceed to run the code using node /commands/deleteTask.js command, I get to see this in my console:

Output seen on terminal whendeleteTask.js file is executed. It shows how the code successfully deletes a single task from the database.

As you can see in the above GIF, the code will prompt you to enter a Todo code for the task you want to delete. Upon deletion, you'll receive a confirmation message in the console. When we read all our tasks after the deletion process, we don't get to see the deleted task. This implies that our code is successful in doing what it is supposed to do!

How to Update Todos

In this section, we will be looking at the code to update a specific todo. Updating a todo is a bit more involved compared to the previous operations. The process unfolds as follows:

1. Prompt the user to input the code of the todo to be updated.
2. Connect to the database.
3. Find the task whose code property matches the user's input.
4. If the task doesn't exist, display a message indicating the failure to find a matching todo.
5. If the task exists, prompt the user to update the name, description and status of the task.
6. If the user sets the status property of a task to "completed," then that task is deleted. If set to "pending," the task's name and description are updated in the database.
7. Display a success message in the console after the update operation.

Let's start coding! The first thing you need to do is to import all the packages and functions we will need to perform this job.

// Importing packages and functions
import {connectDB, disconnectDB} from '../db/connectDB.js'
import { getTaskCode } from './deleteTask.js'
import inquirer from 'inquirer'
import Todos from '../schema/TodoSchema.js'
import ora from 'ora'
import chalk from 'chalk'

Before we start working on our updateTask() function, we will create a small function in the same file named askUpdateQ(). The role of this function is to prompt the user to enter the updated values of the task like the task name, description, and status. At the end, this function will return the response object.

Here is the code for it:

async function askUpdateQ(todo){
    try {
        // Prompting the user to update the todo data
        const update = await inquirer.prompt([
            {name: 'name', message: 'Update the name?', type: 'input', default: todo.name},
            {name: 'detail', message: 'Update the Description?', type: 'input', default: todo.detail},
            {name: 'status', message: 'Update the status', type: 'list', choices: ['pending', 'completed'], default: todo.status}
        ])

        return update
    } catch (error) {
        console.log('Something went wrong... \n', error)
    }
}

Two things are to be noted here:

1. todo is the original task object (the task which the user wants to update). This will be passed to the askUpdateQ() function by the updateTask() function.
2. Each question object within the array passed to inquirer.prompt() contains a default property set to the original values of the task. This ensures that if the user skips a question, the default value remains unchanged.

With that said, now let's look at the code for the updateTask() function:

export default async function updateTask(){
    try {
        // Obtaining the task code entered by user by calling getTaskCode() method
        const userCode = await getTaskCode()

        // Connecting to the database
        await connectDB()

        // Starting the spinner
        const spinner = ora('Finding the todo...').start()

        // Finding the todo which the user wants to update
        const todo = await Todos.findOne({code: userCode.code})

        // Stopping the spinner
        spinner.stop()

        // Checking if the todo exists or not
        if(!todo){
            console.log(chalk.redBright('Could not find a Todo with the code you provided.'))
        } else{
            console.log(chalk.blueBright('Type the updated properties. Press Enter if you don\'t want to update the data.'))

            // Get the user's response of the updated data by calling askUpdateQ() method
            const update = await askUpdateQ(todo)

            // If user marked status as completed, we delete the todo else we update the data
            if(update.status === 'completed'){
                // Changing spinner text and starting it again
                spinner.text = 'Deleting the todo...'
                spinner.start()

                // Deleting the todo
                await Todos.deleteOne({_id : todo._id})

                // Stopping the spinner and display the success message
                spinner.stop()
                console.log(chalk.greenBright('Deleted the todo.'))
            } else {
                // Update the todo
                spinner.text = 'Updating the todo'
                spinner.start()
                await Todos.updateOne({_id: todo._id}, update, {runValidators: true})
                spinner.stop()
                console.log(chalk.greenBright('Updated the todo.'))
            }
        }
        // Disconnecting from the database
        await disconnectDB()
    } catch (error) {
        // Error Handling
        console.log('Something went wrong, Error: ', error)
        process.exit(1)
    }
}

Here's a breakdown of the above code:

1. Obtain the code of the task which the user wants to update. For this, we are utilizing the getTaskCode() function defined in the ./commands/deleteTask.js file. We simply call the function and assign the returned response object to the userCode variable.
2. Connect to the database using await connectDB().
3. Start a spinner to indicate that the code is finding the todo.
4. Use Todos.findOne({ code: userCode.code }) to find the task the user wants to update and assign it to the todo variable. We are doing this because we will need the original values of the task.
5. Stop the spinner.
6. If no matching task is found, display a message using chalk indicating that the task wasn't found.
7. If the task is found, prompt the user to input updated properties by calling the askUpdateQ() function and pass the todo object (original task) in the function. Assign the returned object to update variable.
8. If the user marks the status as "completed," the task is deleted from the database using deleteOne(). If marked as "pending," the task's name and description are updated using updateOne().

updateOne() method takes in 3 parameters – Query Object, Update Object, and the Options object. Here, {_id: todo._id} is the Query Object. Mongoose searches the entire collection for a task whose id property matches with todo_.id. On finding the task, it replaces the task with the update object, that is update in our case. The third parameter, { runValidators: true }, ensures that Mongoose validates the update object against the schema's rules before executing it. If the validation fails, the update will be rejected, and you'll receive an error. If the validation is successful, the document will be updated successfully in the database.

Both in case of the Delete and Update Operation, we change the text of the spinner using spinner.text and start it before performing the operation and once the operation is completed, we stop the spinner.

9. Display appropriate success messages in the console based on the operation performed.
10. Disconnect from the database using await disconnectDB().

If I call the updateTask() function and run the code using the command: node ./commands/updateTask.js, I get to see something like this in my console:

Output seen on terminal when updateTask.js file is executed. It shows how the code successfully fetches the original task and successfully replaces it with the updated values provided by the user.

With this, you've successfully implemented all CRUD operations. Now, let's use the commander library to bring everything together and create a fully functional CLI tool.

How to Write the CLI Entry Point using Commander

In the final stages of our project, we're going to leverage the power of the commander library to craft a user-friendly CLI interface. With commander, we can neatly define different commands – such as read, add, update, and delete – in an organized and intuitive manner.

Our code will reside in the index.js file, which serves as the entry point of our application. Below is the complete code:

#!/usr/bin/env node

// Importing the required functions for each command
import addTask from './commands/addTask.js'
import deleteTask from './commands/deleteTask.js'
import readTask from './commands/readTask.js'
import updateTask from './commands/updateTask.js'

// Importing the Command class from Commander.js library
import { Command } from 'commander'

// Creating an instance of the Command class
const program = new Command()

// Setting the name and description of the CLI tool
program
.name('todo')
.description('Your terminal task manager!')
.version('1.0.0')

// Defining a command called 'add'
program
.command('add')
.description('Create a new todo.')
.action(addTask)

// Defining a command called 'read'
program
.command('read')
.description('Reads all the todos.')
.action(readTask)

// Defining a command called 'update'
program
.command('update')
.description('Updates a todo.')
.action(updateTask)

// Defining a command called 'delete'
program
.command('delete')
.description('Deletes a todo.')
.action(deleteTask)

// Parsing the command-line arguments and executing the corresponding actions
program.parse()

1. The very first line, #!/usr/bin/env node, is a "shebang." It informs the system to execute the script using the Node.js interpreter. This enables us to run the script directly from the command line without explicitly typing node before the script filename.
2. Next, we import all the required functions that contain the logic for each command.
3. The line import { Command } from 'commander' imports the Command class from the Commander.js library. The subsequent line, const program = new Command(), creates an instance of the Command class. This instance is essential for defining and managing commands for our CLI tool.
4. We then set the CLI tool's information. The .name(), .description(), and .version() methods set the name, description, and version of our CLI tool. These details are displayed when users invoke the tool with specific flags such as --help or --version.
5. Next, we define the various commands. Each program.command() block defines a command for our CLI tool. Within each block, the command's name is set using a string argument (for example, 'add', 'read'). The .description() method provides the command's description, while the .action() method associates a function (for example, addTask, readTask) with a specific command. When a user enters a command in the terminal, this associated function is executed.
6. The program.parse() line is essential for parsing the command-line arguments provided by the user. Based on the command entered, Commander.js will execute the associated action function.

How to Test the CLI Tool

We're now nearly at the finish line of creating our Task Manager CLI tool! Before we can install and utilize the tool, we just need to make a small adjustment to the package.json file. Simply add the following entry to the JSON file:

"bin": {
  "todo": "index.js"
}

If you've ever built a CLI tool and wish for users to access it from the command line in a manner similar to commands like node or npm, then the "bin" property comes into play.

The "bin" property in the package.json file enables you to specify commands that become globally accessible once your package is installed. In simpler terms, it lets you create shortcuts for running specific scripts or functions from the command line.

The provided code instructs Node.js to execute the script defined in index.js whenever someone enters todo in the terminal. In essence, this transforms your script into a globally accessible command-line tool.

The final step before you can start using the tool is to install it globally on your system! Run the following command to do so:

npm i -g .

Conclusion

Congratulations if you've followed along this far! You're now fully equipped to begin using your Task Manager CLI tool.

Here are the commands you can use to operate the tool:

• todo add – Create a new task
• todo read – Read all your pending tasks
• todo update – Update a specific task
• todo delete – Delete a task

You can also use these 2 options using the tool:

• todo --version or todo -V – To know the version number of this tool
• todo --help or todo -h – To display help for command

And that wraps up this handbook. I hope you've found it enjoyable and informative.

I encourage you to share your learning journey on Twitter and LinkedIn using the hashtag #LearnInPublic. Also, be sure to follow freeCodeCamp for more informative coding tutorials.

If you are facing any issues while following along with this tutorial, you can always refer to the entire code available on GitHub - Task Manager CLI GitHub Repo. You can also connect with me on Twitter (X) - @Krish4856. My DM's are open!

See you next time! 👋 ❤️ ✨