To use STOMP over WebSocket, you can use @stomp/stompjs, but that has tricky callbacks and a complicated API that caters to more specialized use cases. Luckily, there’s also the lesser-known @stompjs/rx-stomp which provides a nice interface via RxJS observables. Observables aren't exclusive to Angular, and they fit quite well with how React works. It's a neat interface when composing complex workflows and pipelines with many different message sources.

The tutorial follows a somewhat similar path as the initial version in Angular, but the component structure and code style are tuned towards the functional style of React.

Note: This tutorial is written with strict TypeScript, but the JavaScript code is almost identical since we only have 5 type declarations. For the JS version, you can skip the type imports and definitions.

Table of Contents

• Goals

• Prerequisites

• Starter STOMP Server with RabbitMQ

• Starter React Template

• How to Install RxStomp

• How to Manage Connection and Disconnection with the STOMP Server

• How to Monitor the Connection Status

• How to Send Messages

• How to Receive Messages

• Summary

Goals

Here, we’ll build a simplified chatroom application that shows various aspects of RxStomp across different components. Overall, we want to have:

• A React frontend connected with RxStomp to a STOMP server.

• A live connection status display based on our connection to the STOMP server.

• Pub/Sub logic for any configurable topic.

• Splitting RxStomp logic across multiple components to show how to separate logic and responsibility.

• Aligning RxStomp connection/subscription lifecycles with React component lifecycles to ensure that there are no leaks or unclosed watchers.

Prerequisites

• You should have a STOMP server running so that the React application can connect to it. Here, we’ll use RabbitMQ with the rabbitmq_web_stomp extension.

• Latest React version. This tutorial will use v18, although older versions will probably work as well.

• Some familiarity with observables will also help.

Starter STOMP Server with RabbitMQ

If you’d like to use RabbitMQ too (not strictly required), here’s are installation guides for different operating systems. To add the extension, you’ll need to run:

$ rabbitmq-plugins enable rabbitmq_web_stomp

If you’re able to use Docker, a Docker file similar to this will set everything needed for the tutorial:

FROM rabbitmq:3.8.8-alpine

run rabbitmq-plugins enable --offline rabbitmq_web_stomp

EXPOSE 15674

Starter React Template

For this tutorial, we'll use Vite's react-ts template. The central part of our application will be in the App component, and we'll create child components for other specific STOMP functionality.

How to Install RxStomp

We’ll use the @stomp/rx-stomp npm package:

$ npm i @stomp/rx-stomp rxjs

This will install version 2.0.0

Note: This tutorial still works without explicitly specifying rxjs since it's a sister dependency, but it's good practice to be explicit about it.

How to Manage Connection and Disconnection with the STOMP Server

Now, let's open App.tsx and initialize our RxStomp client. Since the client isn't a state that will change for rendering, we’ll wrap it in the useRef Hook.

// src/App.tsx
import { useRef } from 'react'
import { RxStomp } from '@stomp/rx-stomp'

import './App.css'

function App() {
  const rxStompRef = useRef(new RxStomp())
  const rxStomp = rxStompRef.current

  return (
    <>
      <h1>Hello RxStomp!</h1>
    </>
  )
}

export default App

Assuming the default ports and authentication details, we’ll define some configuration for our connection next.

// src/App.tsx

import { RxStomp } from '@stomp/rx-stomp'
import type { RxStompConfig } from '@stomp/rx-stomp'
...
const rxStompConfig: RxStompConfig = {
  brokerURL: 'ws://localhost:15674/ws',
  connectHeaders: {
    login: 'guest',
    passcode: 'guest',
  },
  debug: (msg) => {
    console.log(new Date(), msg)
  },
  heartbeatIncoming: 0,
  heartbeatOutgoing: 20000,
  reconnectDelay: 200,
}

function App() {
  ...

For a better dev experience, we logged all messages with timestamps to a local console and set low timer frequencies. Your configuration should be quite different for your production application, so check out the RxStompConfig docs for all the options available.

Next, we’ll pass the configuration to rxStomp inside a useEffect Hook. This manages the connection's activation alongside the component lifecycle.

// src/App.tsx
...
function App() {
  const rxStompRef = useRef(new RxStomp())
  const rxStomp = rxStompRef.current

  useEffect(() => {
    rxStomp.configure(rxStompConfig)
    rxStomp.activate()

    return () => { 
      rxStomp.deactivate() 
    }
  })
  ...

While there's no visual change in our app, checking the logs should show connection and ping logs. Here's an example of what that should look like:

Date ... >>> CONNECT
login:guest
passcode:guest
accept-version:1.2,1.1,1.0
heart-beat:20000,0

Date ... Received data 
Date ... <<< CONNECTED
version:1.2
heart-beat:0,20000
session:session-EJqaGQijDXqlfc0eZomOqQ
server:RabbitMQ/4.0.2
content-length:0

Date ... connected to server RabbitMQ/4.0.2 
Date ... send PING every 20000ms 
Date ... <<< PONG 
Date ... >>> PING

Note: Generally, if you see duplicate logs, it may be a sign that a deactivation or unsubscribe functionality wasn't implemented correctly. React renders each component twice in a dev environment to help people catch these bugs via React.StrictMode

How to Monitor the Connection Status

RxStomp has a RxStompState enum that represents possible connection states with our broker. Our next goal is to display the connection status in our UI.

Let's create a new component for this called Status.tsx:

// src/Status.tsx
import { useState } from 'react'

export default function Status() {
  const [connectionStatus, setConnectionStatus] = useState('')

  return (
    <>
      <h2>Connection Status: {connectionStatus}</h2>
    </>
  )
}

We can use the rxStomp.connectionState$ observable to bind to our connectionStatus string. Similar to how we used useEffect, we’ll use the unmount action to unsubscribe().

// src/Status.tsx
import { RxStompState } from '@stomp/rx-stomp'
import { useEffect, useState } from 'react'
import type { RxStomp } from '@stomp/rx-stomp'


export default function Status(props: { rxStomp: RxStomp }) {
  const [connectionStatus, setConnectionStatus] = useState('')

  useEffect(() => {
    const statusSubscription = props.rxStomp.connectionState$.subscribe((state) => {
      setConnectionStatus(RxStompState[state])
    })

    return () => {
      statusSubscription.unsubscribe()
    }
  }, [])

  return (
    <>
      <h2>Connection Status: {connectionStatus}</h2>
    </>
  )
}

To view it, we include it in our app:

// src/App.tsx
import Status from './Status'
...
  return (
    <>
      <h1>Hello RxStomp!</h1>

      <Status rxStomp={rxStomp}/>
    </>
  )

At this point, you should have a working visual indicator on the screen. Try playing around by taking the STOMP server down and see if the logs work as expected.

How to Send Messages

Let's create a simple chatroom to show a simplified end-to-end messaging flow with the broker.

We can place the functionality in a new Chatroom component. First, we can create the component with a custom username and message field that's bound to inputs.

// src/Chatroom.tsx
import { useState } from 'react'
import type { RxStomp } from '@stomp/rx-stomp'

export default function Chatroom(props: {rxStomp: RxStomp}) {
  const [message, setMessage] = useState('')
  const [userName, setUserName] = useState(`user${Math.floor(Math.random() * 1000)}`)

  return (
    <>
      <h2>Chatroom</h2>

      <label htmlFor='username'>Username: </label>
      <input
        type='text'
        name='username'
        value={userName}
        onChange={(e) => setUserName(e.target.value)}
      />

      <label htmlFor='message'>Message: </label>

      <input
        type='text'
        value={message}
        onChange={(e) => setMessage(e.target.value)}
        name='message'
      />
    </>
  )    
}

Let’s include this within our App with a toggle to join the chatroom:

// src/App.tsx
import { useEffect, useState, useRef } from 'react'
import Chatroom from './Chatroom'
...
function App() {
  const [joinedChatroom, setJoinedChatroom] = useState(false)
  ...
  return (
    <>
      <h1>Hello RxStomp!</h1>

      <Status rxStomp={rxStomp}/>

      {!joinedChatroom && (
        <button onClick={() => setJoinedChatroom(true)}>
          Join chatroom!
        </button>
      )}

      {joinedChatroom && (
        <>
          <button onClick={() => setJoinedChatroom(false)}>
            Leave chatroom!
          </button>

          <Chatroom rxStomp={rxStomp}/>
        </>
      )}

    </>
  )

Time to actually send messages. STOMP is best for sending text-based messages (binary data is also possible). We’ll define the structure of the data we're sending in a new types file:

// types.ts
interface ChatMessage {
  userName: string,
  message: string
}

Note: If you're not using TypeScript, you can skip adding this type definition.

Next, let's use JSON to serialize the message and send messages to our STOMP server using .publish with a destination topic and our JSON body.

// src/Chatroom.tsx
import type { ChatMessage } from './types'
...
const CHATROOM_NAME = '/topic/test'

export default function Chatroom(props: {rxStomp: RxStomp}) {
  ...
  function sendMessage(chatMessage: ChatMessage) {
    const body = JSON.stringify({ ...chatMessage })
    props.rxStomp.publish({ destination: CHATROOM_NAME, body })
    console.log(`Sent ${body}`)
    setMessage('')
  }

  return (
    <>
      <h2>Chatroom</h2>

      <label htmlFor="username">Username: </label>
      <input
        type="text"
        name="username"
        value={userName}
        onChange={(e) => setUserName(e.target.value)}
      />

      <label htmlFor="message">Message: </label>

      <input
        type="text"
        value={message}
        onChange={(e) => setMessage(e.target.value)}
        name="message"
      />

      <button onClick={() => sendMessage({userName, message})}>Send Message</button>
    </>
  )
}

To test it out, try clicking the Send Message button a few times and see if the serialization works fine. While you won't be able to see any visual changes yet, the console logs should show it:

Date ... >>> SEND
destination:/topic/test
content-length:45

Sent {"userName":"user722","message":"1234567890"}

How to Receive Messages

We’ll create a new component to show the list of messages from all the users. For now, we'll use the same type, pass the topic name as a prop, and display everything as a list. All this goes into a new component called MessageList.

// src/MessageDisplay.tsx
import { useEffect, useState } from 'react'
import type { RxStomp } from '@stomp/rx-stomp'
import type { ChatMessage } from './types'

export default function MessageDisplay(props: {rxStomp: RxStomp, topic: string}) {
  const [chatMessages, setChatMessages] = useState<ChatMessage[]>([
    {userName: 'admin', message: `Welcome to ${props.topic} room!`}
  ])

  return(
  <>
  <h2>Chat Messages</h2>
  <ul>
    {chatMessages.map((chatMessage, index) => 
      <li key={index}>
        <strong>{chatMessage.userName}</strong>: {chatMessage.message}
      </li>
    )}
  </ul>
  </>
  )
}

Time to bring everything together!

We can display our messages to display within our Chatroom component by adding it to the bottom.

// src/Chatroom.tsx
import { useState } from 'react'
import type { ChatMessage } from './types'
import type { RxStomp } from '@stomp/rx-stomp'

import MessageDisplay from './MessageDisplay'

export const CHATROOM_NAME = '/topic/test'

export default function Chatroom(props: {rxStomp: RxStomp}) {
  const [message, setMessage] = useState('')
  const [userName, setUserName] = useState(`user${Math.floor(Math.random() * 1000)}`)

  function sendMessage(chatMessage: ChatMessage) {
    const body = JSON.stringify({ ...chatMessage })
    props.rxStomp.publish({ destination: CHATROOM_NAME, body })
    console.log(`Sent ${body}`)
    setMessage('')
  }

  return (
    <>
      <h2>Chatroom</h2>

      <label htmlFor='username'>Username: </label>
      <input
        type='text'
        name='username'
        value={userName}
        onChange={(e) => setUserName(e.target.value)}
      />

      <label htmlFor='message'>Message: </label>

      <input
        type='text'
        value={message}
        onChange={(e) => setMessage(e.target.value)}
        name='message'
      />

      <button onClick={() => sendMessage({userName, message})}>Send Message</button>

      <MessageDisplay rxStomp={props.rxStomp} topic={CHATROOM_NAME} />
    </>
  )
}

And once you've verified the static display working locally, we can make this display dynamic using an RxJS Observable to receive our chat messages.

Similar to managing the subscription with the Status component, we set up the subscription on mount, and unsubscribe on unmount.

Using RxJS pipe and map, we can deserialize our JSON back to our ChatMessage. The modular design can let you set up a more complicated pipeline as needed using RxJS operators.

// src/MessageDisplay.tsx
...
import { map } from 'rxjs'

export default function MessageDisplay(props: {rxStomp: RxStomp, topic: string}) {
  const [chatMessages, setChatMessages] = useState<ChatMessage[]>([
    {userName: 'admin', message: `Welcome to ${props.topic} room!`}
  ])

  useEffect(() => {
    const subscription = props.rxStomp
      .watch(props.topic)
      .pipe(map((message) => JSON.parse(message.body)))
      .subscribe((message) => setChatMessages((chatMessages) => [...chatMessages, message]))

    return () => {
      subscription.unsubscribe()
    }
  }, [])

  ...

At this point, the chat GUI should show messages correctly, and you can experiment with opening multiple tabs as different users.

Another thing to try here is turning off the STOMP server, sending a few messages, and turning it back on. The messages should get queued locally and dispatched once the server is ready to go. Neat!

Summary

In this tutorial, we:

• Installed @stomp/rx-stomp for a nice dev experience.

• Set up RxStompConfig to configure our client with the connection details, debugger logging and timer settings.

• Used rxStomp.activate and rxStomp.deactivate to manage the client’s main lifecycle.

• Monitored the subscription state using rxStomp.connectionState$ observable.

• Published messages using rxStomp.publish with configurable destinations and message bodies.

• Created an observable for a given topic using rxStomp.watch.

• Used both console logs and React components to see the library in action, and verify functionality and fault tolerance.

You can find the final code on Gitlab: https://gitlab.com/harsh183/rxstomp-react-tutorial. Feel free to use it as a starter template too and report any issues that may come up.

To illustrate just how innovative real time technologies are, we will build a chart application which automatically updates with new dynamic online data.

This is going to be a really good example of how we can implement user data into useful commercial products such as sports leaderboards, social media analytics, financial money trackers, medical instruments, games and many more.

As you can see, there are numerous use cases for this technology. So let's get under way and first learn about the main platform that we will be using to create our realtime data: Pusher.

You can find the codebase online here.

Prerequisites

• An IDE/code editor installed
• Basic knowledge of JavaScript and React
• An understanding of Node.js and npm
• An account on Pusher

Table of Contents

1. What are WebSockets?
2. What is Pusher?
3. What we will be building
4. How to Create an Account on Pusher
5. How to Build the Company Annual Income Application
6. Conclusion

What are WebSockets?

WebSockets are a communications protocol which allows for data to travel bidirectionally between a client and a server. This essentially means that it's possible for a client and server to simultaneously send data back and forth independently of each other.

Being able to send data this way brings many advantages, such as preventing blocking. This can occur if the system is only capable of either sending or receiving and not both at the same time.

Another advantage is the fact that the Transmission Control Protocol (TCP) connection remains long-lived and connected. This is a drawback of traditional Hypertext Transfer Protocol (HTTP) connections, as they are regularly opening and closing each time there is a request or a response. This prevents communication from being instant and realtime.

Not having to worry about the opening and closing of connections results in dramatically less network traffic – so in terms of speed and resources, this is a welcome bonus.

Now that you know why WebSockets are helpful, let's take a look at Pusher and see what the platform is capable of.

What is Pusher?

Pusher is an online platform you can use to build and develop applications that require real-time communication. These communications typically take place between web browsers, mobile phones, and other internet connected devices.

The platform is designed to make it easy to implement real-time communication systems which are nowhere near as complicated as raw WebSocket connections in terms of setup and management. This allows for better scaling and even a way to handle fallbacks for legacy environments that do not support WebSockets.

Pusher is capable of using numerous technologies such as WebSockets, HTTP fallbacks, and even its own propriety protocol called ws-longpolling. This enables the platform to work in places where WebSockets is not supported.

Pusher also has a lot of other useful features. For example, it fully supports WebSockets, so bidirectional communication works as you would expect. The platform also follows the publish-subscribe (Pub/Sub) pattern which is a popular messaging pattern used in the industry. It pretty much guarantees that anyone subscribed to any of the channels can receive the messages in realtime.

A good example of this would be everyone getting the latest news notification who is subscribed to that social media channel. Subscriptions can be public and private, which requires authentication to gain access to those messages.

There's also wide platform support, as the Pusher SDK is available in different programming languages like JavaScript, Python, iOS, Android and many others. It's available as a service online, so all of the scaling and communications infrastructure is already handled. This means one less system for developers to handle when building an application.

Pusher is a great option, as I hope you can tell. But there are other tools out there that do similar things:

• PubNub
• deepstreamHub
• AWS IoT
• AWS SNS
• Google Cloud Pub/Sub
• Fanout

Ok, you've learned quite a lot about WebSockets and Pusher. Now, we'll take a look at the application that we are going to build.

What We Will Be Building

To help you learn how Pusher works, we are going to build a Company Annual Income dashboard application. The application has a graph, a 3D Mode toggle, and three buttons.

We'll build our application using Next.js, Pusher, and HighCharts (the tool that we'll use to create our graph component). This will be a full stack application with one API endpoint for our Pusher REST API. We just need one GET endpoint which will connect to the Pusher API.

The graph data will come directly from Pusher, and the 3D Mode toggle essentially lets us switch our chart between 2D and 3D mode. The buttons let us change the type of chart that is displayed on screen, and all 3 charts can work in 2D and 3D mode. The chart has some starting data, and new data is automatically added to the end for each new year.

Here you can see our Company Annual Income Application in 2D Graph Mode.

Company Annual Income Application 2D Mode

And here you can see our Company Annual Income Application in 3D Graph Mode.

Company Annual Income Application 3D Mode

As you can see, the HighCharts library is very good when it comes to building applications that require data visualisation. If you are interested in other libraries which work just as well, I have put together a list of some of them here. Each one has its own unique benefits and it's possible to use more than one library if you want to:

• Chart.js
• D3
• Recharts
• Google Charts
• Apex Charts
• dyagraphs
• Victory

Alright, we know what we are going to build. In the next section, we will quickly go through how to create an account on Pusher followed by actually building our application afterwards.

How to Create an Account on Pusher

Creating an account on Pusher only requires a few steps. Depending on when you follow this tutorial, the website might look a bit different – but the sign up process should be the same.

Start by going to the Pusher website as shown below:

Pusher website home page

Click on either the Get your free account button in the middle of the page, or the Sign up button in the top right hand corner.

You should now be on the Sign up page as shown here:

Pusher website sign up for account page

Either use your GitHub, Google, or email to create an account.

The next page to load should be your main dashboard after you have completed the sign up as shown in the image below:

Pusher website main dashboard page

Click on the Manage button in the Channels section in the image shown above to get to the Channels page.

The Channels page can be seen below which displays all of the data for our app.

Pusher website Channels page

We need to create an app, so click on the Create app button in the top right hand corner.

Clicking on the button should give you this Create your Channels app box shown here:

Pusher website Create your channels app page

Give your app a name and then choose React for the front end and Node.js for the back end. Then click the Create app button at the bottom of the form.

That's it – you should now see the main screen for your app as shown in the example image below (with data for connections and messages shown in the middle). The sidebar on the left has the features and App keys which we need to use later.

Pusher website app page

Great! Up next we will start building our app...so lets get to it.

How to Build the Company Annual Income Application

The first thing we have to do is setup our project architecture and the folder structure so let's start with that. Create a folder on your computer called realtime-chart-pusher and then cd into it.

Create a Next.js project by running the usual install and setup command here:

npx create-next-app .

On the configuration screen, our project needs to use Tailwind CSS for styling and its recommended that you use the App Router. The other defaults should be fine.

Now we have to install our dependencies for the project. Go ahead and do that with this command:

npm i axios highcharts-react-official pusher pusher-js

Here is a quick breakdown of the purpose of each package we are installing:

• Axios: for making API requests to our backend API which is connected to the Pusher API
• highcharts-react-official: for building our realtime chart
• pusher and pusher-js: for connecting to our Pusher account and the Pusher API

Right, with that done, we just have to setup our project files. You can use this run script in your terminal while still in the project root folder:

touch .env.local
cd src/app
mkdir -p api/pusher components/ChartButton components/CompanyIncomeChart components/Toggle3DButton
touch api/pusher/route.js
touch components/ChartButton/ChartButton.js components/CompanyIncomeChart/CompanyIncomeChart.js components/Toggle3DButton/Toggle3DButton.js

This script creates a .env.local file for our Pusher App keys. It also creates files and folders for our backend and frontend.

Before we begin adding the code to our codebase, we need to get our Pusher App Keys. You can find them on your Pusher account as shown in this example below:

Pusher website App keys page

Add your app keys to the .env.local file like in this code example here:

NEXT_SECRET_PUSHER_APP_ID = "yourid"
NEXT_SECRET_PUSHER_KEY = "yourkey"
NEXT_SECRET_PUSHER_SECRET = "yoursecret"
NEXT_SECRET_PUSHER_CLUSTER = "yourlocation"

We just added NEXT_SECRET at the start of all of the variables because it is a Next.js convention and ensures that the variables work properly.

Now we can start with the bulk of our codebase. First we will do the route.js file inside of the api/route folder. This is the code needed for the file:

import Pusher from 'pusher';

const pusher = new Pusher({
  appId: process.env.NEXT_SECRET_PUSHER_APP_ID,
  key: process.env.NEXT_SECRET_PUSHER_KEY,
  secret: process.env.NEXT_SECRET_PUSHER_SECRET,
  cluster: process.env.NEXT_SECRET_PUSHER_CLUSTER,
  useTLS: true,
});

export async function GET() {
  // Define the initial value
  const value = Math.random() * 800000 + 200000;

  setInterval(() => {
    pusher.trigger('company-income', 'new-price', {
      value: value,
    });

    return Response.json({ value: value }, { status: 200 });
    // Every ten seconds, the setInterval method will get data from the API because its value is set to ten seconds. Pusher's free plans are limited to 200,000 messages a day, so be careful while reducing the interval or you risk exceeding your limit too soon.
  }, 10000);
  return Response.json({ value: value }, { status: 200 });
}

The code in this file lets us connect to Pusher using our App Keys. The value is a randomly generated number that gets sent as JSON to our Pusher channel. This number is what we use to show the company income in each new year which is automatically generated in our chart.

This update occurs every 10 seconds and can be changed in the setInterval function. Pusher's free plans are limited to 200,000 messages a day, so be careful while reducing the interval or you risk exceeding your limit too soon.

The next file we will work on is the ChartButton.js file which you can find inside its folder ChartButton.

Add this code to the file:

export default function ChartButton({ chartRef, type, name }) {
  const switchToLineChart = () => {
    chartRef.current.update({
      chart: {
        type: type,
      },
    });
  };

  return (
    <button
      onClick={() => switchToLineChart()}
      className="bg-indigo-500 hover:bg-indigo-700 p-2 rounded text-white shadow-md"
    >
      {name}
    </button>
  );
}

Our chart application has three buttons for Area Chart, Bar Chart, and Line Chart. This is basically just the component that creates the buttons.

Good, up next will be our CompanyIncomeChart.js file which you can find in the CompanyIncomeChart folder. This file will be taking this code here:

'use client';

import { useEffect, useRef, useState } from 'react';
import axios from 'axios';
import ChartButton from '../ChartButton/ChartButton';
import Toggle3DButton from '../Toggle3DButton/Toggle3DButton';

export default function CompanyIncome() {
  const initialData = [
    [1965, 360202],
    [1966, 400123],
    [1967, 460331],
    [1968, 460346],
    [1969, 460339],
    [1970, 460370],
  ];

  const chartRef = useRef(null);
  const [chartData, setChartData] = useState([...initialData]);
  const [toggle3D, setToggle3D] = useState(false);

  useEffect(() => {
    // Dynamically import Highcharts and its 3D module
    import('highcharts/highcharts-3d').then((Highcharts3D) => {
      import('highcharts').then((Highcharts) => {
        Highcharts3D.default(Highcharts.default);

        chartRef.current = Highcharts.default.chart('chart-container', {
          colors: ['#F3F7FB', '#F3F7FB'],
          chart: {
            style: {
              fontFamily: ['Prompt', 'sans-serif'],
              fontSize: '16px',
            },
            type: 'column',
            options3d: {
              enabled: toggle3D,
              alpha: 10,
              beta: 25,
              depth: 70,
              viewDistance: 25,
            },
            backgroundColor: {
              linearGradient: [0, 0, 500, 500],
              stops: [
                [0, 'rgb(128, 130, 221)'],
                [1, 'rgb(128, 130, 221)'],
              ],
            },
          },
          title: {
            text: 'COMPANY ANNUAL INCOME',
            style: {
              fontSize: '27px',
            },
          },
          xAxis: {
            title: {
              text: 'Year',
              style: {
                color: 'white',
              },
            },
            labels: {
              style: {
                color: 'white',
              },
            },
            type: 'category',
          },
          yAxis: {
            title: {
              text: 'Income',
              style: {
                color: 'white',
              },
            },
            labels: {
              style: {
                color: 'white',
              },
            },
            min: 0,
          },
          credits: {
            text: '',
          },
          series: [
            {
              name: 'Income',
              data: chartData,
            },
          ],
          animation: {
            duration: 100,
          },
        });
      });
    });

    const interval = setInterval(async () => {
      try {
        const response = await axios.get('/api/pusher');
        const newDataPoint = [
          chartData[chartData.length - 1][0] + 1,
          response.data.value,
        ];
        chartRef.current.series[0].addPoint(newDataPoint, true, true);
        setChartData((prevData) => [...prevData, newDataPoint]);
      } catch (error) {
        console.error('Error fetching data:', error);
      }
      // Every ten seconds, the setInterval method will get data from the API because its value is set to ten seconds. Pusher's free plans are limited to 200,000 messages a day, so be careful while reducing the interval or you risk exceeding your limit too soon.
    }, 10000);
    return () => {
      clearInterval(interval);
    };
  }, [toggle3D]);

  return (
    <div>
      <div id="chart-container" className="w-full h-96"></div>
      <div className="border-solid border-2 border-indigo-500 m-10">
        <div className="grid justify-center mt-5 text-center">
          <p>3D Mode</p>
          <div className="w-20">
            <Toggle3DButton toggle3D={toggle3D} setToggle3D={setToggle3D} />
          </div>
        </div>
        <div className="grid gap-4 justify-center mt-5 xl:grid-cols-3 p-4">
          <ChartButton chartRef={chartRef} type={'area'} name={'Area Chart'} />
          <ChartButton chartRef={chartRef} type={'bar'} name={'Bar Chart'} />
          <ChartButton chartRef={chartRef} type={'line'} name={'Line Chart'} />
        </div>
      </div>
    </div>
  );
}

This component creates our Company Income Chart and is already set with some initial loading data so that the chart loads some data when it first starts.

The file contains our chart configuration settings. There is a function that does a GET request to our Next.js backend which then connects to our Pusher account. This is how the data is retrieved for our chart file. Like on the backend, there is a setInterval function that runs every 10 seconds to get the latest data from our backend (we we can increase or decrease).

Next is the Toggle3DButton.js file in the Toggle3DButton folder. Here is the code for the file:

export default function Toggle3DButton({ toggle3D, setToggle3D }) {
  const toggle3DMode = () => {
    setToggle3D(!toggle3D);
  };

  return (
    <label className="flex items-center cursor-pointer border border-gray-400 rounded-full p-1 relative">
      <input
        type="checkbox"
        className="hidden"
        checked={toggle3D}
        onChange={toggle3DMode}
      />
      <span
        className={`toggle__line w-full h-4 bg-gray-400 rounded-full shadow-inner ${
          toggle3D ? 'bg-green-500' : 'bg-gray-400'
        }`}
      ></span>
      <span
        className={`toggle__dot absolute w-6 h-6 bg-white rounded-full shadow inset-y-0 ${
          toggle3D ? 'right-0' : 'left-0'
        }`}
      ></span>
    </label>
  );
}

Our application has a 3D Mode toggle and this component is used to create it.

The main files are completed – we just have three left and then we're done. Next is the globals.css file which needs this code to replace the existing code:

@import url('https://fonts.googleapis.com/css2?family=Bebas+Neue&family=Prompt:ital,wght@0,100;0,200;0,300;0,400;0,500;0,600;0,700;0,800;0,900;1,100;1,200;1,300;1,400;1,500;1,600;1,700;1,800;1,900&display=swap');

@tailwind base;
@tailwind components;
@tailwind utilities;

body {
  font-family: 'Prompt', sans-serif;
  font-weight: 400;
  font-style: normal;
}

We just imported the Prompt font to use in our application.

Continuing on from before, now we have to update the layout.js file so that it also uses the Prompt font. Add this code:

import { Prompt } from 'next/font/google';
import './globals.css';

const prompt = Prompt({ subsets: ['latin'], weight: '400' });

export const metadata = {
  title: 'Create Next App',
  description: 'Generated by create next app',
};

export default function RootLayout({ children }) {
  return (
    <html lang="en">
      <body className={prompt.className}>{children}</body>
    </html>
  );
}

Now we can access the Prompt font throughout our application with this import and update.

All that remains is our page.js file which is the main entry point for all of our components. Replace all the code in that file with this final code shown here:

import CompanyIncomeChart from './components/CompanyIncomeChart/CompanyIncomeChart';

export default function Home() {
  return <CompanyIncomeChart />;
}

Alright all done – our project codebase is complete!

Just return to the root project folder and run the command below. Your application should be up and running in a web browser.

npm run dev

Everything should be working, and now you should have a realtime chart with live random dates.

Conclusion

Realtime communication platforms are essential for all of the tech that we use. Today you learned about Pusher and how it can help you build realtime data applications that can integrate with a chart library like HighCharts.

There are lots of potential applications you can build with these tools. Today's introduction should open the door and give you many ideas for future projects.

We just published a course on the freeCodeCamp.org YouTube channel that is designed to help you understand the fundamentals of WebSockets and how to implement real-time, bidirectional communication in web applications using Socket.io. Nishant from Cybernatico developed this course. He has developed many courses for our channel.

What Are WebSockets?

WebSockets provide a way to open an interactive communication session between the user's browser and a server. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

Unlike the typical HTTP request/response cycle that only allows the client to initiate communications, WebSockets enable both the server and the client to push messages at any time without the need to establish multiple HTTP connections. This makes WebSockets a more efficient and lower-latency way to communicate.

Exploring Socket.io

Socket.io is a JavaScript library that simplifies the process of working with WebSockets. It provides a seamless way to handle real-time web functions, ensuring compatibility across different browsers and devices. Socket.io is not just limited to WebSocket connections; it can also use other methods like polling to maintain a constant connection between the client and server, which is great for developers looking for flexibility and reliability in their real-time applications.

Course Overview

The video course begins with a brief introduction to WebSockets, explaining how this technology differs from traditional HTTP connections and its advantages in modern web applications. Then, you will learn the following:

• How to set up a basic WebSocket server
• How to integrate Socket.io into both your front-end and back-end applications
• Building a simple application from scratch
• Implementing real-time notifications and broadcasting messages to multiple clients

This video course is packed with examples, best practices, and practical challenges to test your new skills in real-world scenarios.

Watch the full course on the freeCodeCamp.org YouTube channel (2-hour watch).

The interface is what the software presents to other humans or programs, allowing them to interact with it.

A good analogy for an interface is a remote control. Imagine you have a universal remote that can control your TV, lights, and fan.

Image showing a remote control and a TV, light fixture, and fan.

Let’s break down what a universal remote control can do:

1. The remote control has various buttons, each serving a different purpose. One button might change the channel, while another can dim the lights of the chandelier, and another can turn on the fan.

2. When you press a button, it sends a specific signal via infrared, bluetooth, or wifi to the object you are controlling, instructing it to perform a particular action.

3. The key thing about the remote is that it allows you to interact with the TV, chandelier, and the fan without understanding the internal workings of these objects. All that complexity is abstracted away from you. You simply press a button, and you get a response that you can observe straight away.

APIs work in a similar way.

1. APIs can have various endpoints, each designed to perform a specific action. One endpoint might retrieve data, while another updates or deletes it.

2. When you send a request to an endpoint, it communicates with the server using HTTP methods – GET, POST, PUT, DELETE to instruct it to perform a particular action (like retrieving, sending, updating, or deleting data).

3. The key thing about APIs, as with remote controls, is that APIs abstract away the inner workings of the server and the database behind the API. The API allows users, developers and applications to interact with a software application or platform without needing to understand its internal code or database structure. You simply send a request, the server processes it and provides a response.

This analogy only holds true for so long, as APIs are more complex than a remote control. But the basic principles of operation between an API and a universal remote are quite similar.

This article will explain API integration patterns, which can be split into two broad groups: Request-response (REST, RPC & GraphQL) and event driven APIs (Polling, WebSockets & WebHooks).

Request-Response Integration

In a request-response integration, the client initiates the action by sending a request to the server and then waits for a response.

Different patterns of the request-response integration exist, but at a high level, they all conform to the same rule of the client initiating a request and waiting for a response from the server.

1. REST

Rest stands for Representational State Transfer – the acronym is a combination of the first one or two letters from these three words. This is the simplest and most popular form of a request-response integration.

REST APIs use a stateless, client-server communication model, wherein each message contains all the information necessary to understand and process the message.

REST is all about resources. Resources are entities that the API exposes, which can be accessed and manipulated using URL paths.

To understand REST APIs, consider the following analogy. Imagine you go into a restaurant to order some food. The menu is extensive and items are categorically organised. Each item on the menu can be equated to a resource.

First, you call the waiter to get their attention, then you place an order. Each request receives a response, before you proceed with another request, like ordering a dish.

Restaurant analogy for REST API

In REST API terms, the client initiates requests to the server by specifying exactly what it wants using HTTP methods (such as GET, POST, PUT, DELETE) on specific URLs (the menu items). Each interaction is stateless, meaning that each request from the client to the server must contain all the information needed to understand and process the request.

The server then processes the request and returns the appropriate response – in our analogy, bringing the ordered item to the table.

Simple sequence diagram for REST API

2. RPC

RPC stands for Remote Procedure Call. Unlike REST APIs which are all about resources, RPC is all about actions. With RPC, the client executes a block of code on the server

Think of a restaurant without a menu. There is no dish you can request in this restaurant. Instead, you request a specific action to be performed by the restaurant.

Restaurant analogy for RPC

With a REST API, the guest would have simply asked for some fish and chips. With RPC, they have to give instructions on what they want the kitchen to prepare.

In the RPC pattern, the client calls a specific procedure on the server and waits for the result. The procedure to prepare and what gets prepared are tightly bound together. This might give the client very specific and tailored results, but lacks the flexibility and ease of use of REST.

There is a reason most restaurants use menus, instead of following the custom requests of their customers. This partly explains why RPC is a less popular integration pattern compared to REST.

3. GraphQL

With GraphQL, the client specifies exactly what data it needs, which can include specific fields from various resources. The server processes this query, retrieves the exact data, and returns it to the client.

This enables the client to have a high degree of flexibility and only retrieve exactly the data it needs. It also requires the server to be capable of handling more complex and unique queries.

In this way, GraphQL is a more customisable form of REST. You still deal with resources (unlike actions in RPC) but you can customise how you want the resource returned to you.

Think of a restaurant that allows you to customise your own dish by specifying exact quantities or ingredients you want.

Restaurant analogy for GraphQL

This may look similar to the RPC pattern, but notice that the customer is not saying how the food should be made, they're just customising their order by removing some ingredients (no salt) and reducing the number of some items (two pieces of fish instead of four).

One of the drawbacks of GraphQL is that it adds complexity to the API since the server needs to do additional processing to parse complex queries. This additional complexity would also apply to the restaurant analogy, since each order would need to be customised to the guest.

GraphQL has one clear benefit over REST and RPC. Since clients can specify exactly what they need, the response payload sizes are typically smaller, which means faster response times.

Event Driven Integration

This integration pattern is ideal for services with fast changing data.

Some of these integration patterns are also asynchronous and initiated by the server, unlike the request-response patterns which are synchronous and initiated by the client.

1. Polling

Let’s bring back the restaurant analogy. When you order food, it will take some time for it to be prepared.

You can get updates on your order by asking the waiter if it is ready yet. The more frequently you ask, the closer you will be to having real-time information about your order.

This, however, puts unnecessary strain on the waiter since they have to constantly check the status of your order and have them update you whenever you ask.

Restaurant analogy for polling

Polling is when the client continuously asks the server if there is new data available, with a set frequency. It's not efficient because many requests may return no new data, thus unnecessarily consuming resources.

The more frequently you poll (make requests) the closer the client gets to real-time communication with the server.

Simple sequence diagram showing polling in action

Most of the requests during polling are wasted, since they only return something useful to the client once there is a change on the server.

There is, however, another version of polling called long polling. With long polling, the waiter does not respond to the guest straightaway about the status of the order. Instead, the waiter only responds if there is an update.

Naturally, this only works if the guest and the waiter agree beforehand that a slow response from the waiter does not mean that the waiter is being rude and the guest is being ignored.

Restaurant analogy for long polling

With long polling, the server does not respond to the client immediately. It waits until something has changed before responding.

As long as the client and server agree that the server will hold on to the client’s request, and the connection between the client and server remains open, this pattern works and can be more efficient than simply polling.

These two assumptions for long polling may be unrealistic, though – the server can lose the client's request and/or the connection can be broken.

To address these limitations, long polling adds extra complexity to the process by requiring a directory of which server contains the connection to the client, which is used to send data to the client whenever the server is ready.

Standard polling on the other hand can remain stateless, making it more fault tolerant and scalable.

2. WebSockets

WebSockets provide a persistent, two-way communication channel between the client and server. Once a WebSocket connection is established, both parties can communicate freely, which enables real-time data flows and is more resource-efficient than polling.

Using the restaurant analogy again, a guest orders a meal and then establishes a dedicated communication channel with the waiter so they can freely communicate back and forth about updates or changes to the order until the meal is ready. This means the waiter can also initiate the communication with the guest, which is not the case for the other integration patterns mentioned so far.

Restaurant analogy for WebSockets

WebSockets are similar to long polling. They both avoid the wasteful requests of polling, but WebSockets have the added benefit of having a persistent connection between the client and the server.

WebSockets are ideal for fast, live streaming data, like real-time chat applications. The downside of WebSockets is that the persistent connection consumes bandwidth, so may not be ideal for mobile applications or in areas with poor connectivity

3. WebHooks

WebHooks allow the server to notify the client when there's new data available. The client registers a callback URL with the server and the server sends a message to that URL when there is data to send.

With WebHooks, the client sends requests as usual, but can also listen for and receive requests like a server.

Simple sequence diagram showing WebHooks in action

Using the restaurant analogy, when the guest orders a meal, they give the waiter a bell (analogous to the callback URL). The waiter goes to the kitchen and rings the bell as soon as the meal is ready. This allows the client to know, in real-time, about the progress of his order.

WebHooks are superior to polling because you get real-time updates from the server once something changes, without having to make frequent, wasteful requests to the server about that change.

They're also superior to long polling because long polling can consume more client and server resources as it involves keeping connections open, potentially resulting in many open connections.

Bringing it Together

In conclusion, APIs are crucial tools in software development, allowing users and applications to interact with software without understanding its inner workings.

They come in different integration patterns, such as REST, RPC, GraphQL, Polling, WebSockets, and WebHooks.

If you need a simple request-response integration, then REST, RPC or GraphQL could be ideal. For real-time or near-real-time applications, polling, WebScokets, or WebHooks are ideal.

As with any design problem, the right choice depends on the business case and what tradeoffs you are willing to tolerate.

We will use React with Typescript for creating the UI, Redux for managing the application state, and styled-components for applying the styling. And last, but not least, we'll use WebSockets for fetching the data feeds.

GitHub Repo

💡 If you want to skip the reading, here 💁 is the GitHub repository with a detailed README 🙌, and here you can see the live demo.

What is an Order Book?

An Order Book is an application that usually displays some kind of information related to buying and selling stuff.

💡 The most common use case is showing data for various assets, such as stocks, bonds, currencies, and even cryptocurrencies.

Why Would I Need an Order Book?

In practice, Order Books are used by traders to watch the fluctuations of the bidding price and the asking price of certain products – currencies, stocks, and so on.

This is happening real time, so the changes can be very rapid. Here is where WebSockets will come in handy, as you will see later.

In the past, people did something similar on paper, but the ‘real-time' part was impossible, of course.

A regular Order Book usually has two sides: buying (or bidding), shown in green on the left side and selling (or asking), red, on the right.

Classic Order book

The Plan for our Order Book App

Our Order Book app will consist of five parts:

• order book main view

• grouping select box

• Toggle Feed button

• Kill Feed button

• Status Message.

The app design will look as shown below. Note that the Status Message component, that you will see in the my implementation, is missing on these screenshots:

Desktop layout

Mobile layout

Application Features

Order book

The order book has two sides: the buy side and the sell side.

Both sides contain information about the number of orders opened at each price level.

Each level displays:

• Price: this is what defines the level. As orders must be placed at a price that is a multiple of the selected market's tick size (0.5), each level will be an increment of 0.5 (as long as there is an order open at that level).

• Size: the total quantity of contracts derived from open orders that have been placed at this level.

• Total: the summed amount of contracts derived from open orders that reside in the book at this level and above. To calculate the total of a given level we take the size of the current level and sum the sizes leading to this price level in the order book. The total is also used to calculate the depth visualizer (colored bars behind the levels). The depth of each level is calculated by taking that level's total as a percentage of the highest total in the book.

Grouping Select Box

By default the orders are grouped by the selected market's ticket size (0.5).

Possible toggling of the grouping is between 0.5, 1, 2.5 for XBTUSD market and 0.05, 0.1 and 0.25 for ETHUSD market.

To group levels, we combine the levels rounded down to the nearest group size – for example, if we change our grouping from 0.5 to 1 then we would combine the data from prices 1000 and 1000.5 and display it under a single level in the order book with the price 1000.

Toggle Feed Button

This button toggles the selected market between PI_XBTUSD and PI_ETHUSD. These are the two markets we will support -> Bitcoin/USD and Ethereum/USD.

It supports dynamic grouping logic and handles groupings for XBT (0.5, 1, 2.5) and groupings for ETH (0.05, 0.1, 0.25).

Kill Feed Button

Clicking this button stops the feed.

Then clicking this button a second time renews the feed.

Status Message

This message will show the currently selected market. It will also show a message saying the feed is killed.

Tech Stack for our App

Here is a list of the main technologies we will be using:

• React with TypeScript (yarn create react-app my-app --template typescript) — a UI library we will use for building our application’s user interfaces.

• Redux — a state management library we will use for managing our application’s state.

• WebSockets — The WebSocket object provides the API for creating and managing a WebSocket connection to a server, as well as for sending and receiving data on the connection. We will use it to implement the logic for consuming the live feeds as well as to be able to stop and renew.

• styled-components — a CSS-in-JS library that lets you define the CSS styles of your components using ES6 template literals. We will use it to add styles to our app and make the look and feel beautiful. It utilizes tagged template literals to style your components and removes the mapping between components and styles. This means that when you’re defining your styles, you’re actually creating a normal React component that has your styles attached to it.

• react-testing-library — The React Testing Library is a very light-weight solution for testing React components. We will use it for testing the UI components of our app.

• Jest - a JavaScript Testing Framework that has become the de facto standard when we talk about testing React applications. We will use it to write some unit tests that will cover the reducer functions we have in our app.

How to Build the App

From this point onward I will try to guide you through the process I followed when building this.

💡 I must say that what I am showing you here is just a way of creating such an app – but it's not the way in any regard. Probably folks with more experience in crypto would do it better.

Project Structure

The project structure is pretty straightforward. We are using React and styled-components, which makes this way of structuring very convenient.

Let's see first what it looks like and then I will explain the what and the why.

Project structure

As you can see on the image above, I have organized most of the components in folders. Each folder contains an index.tsx file, a styles.tsx and a .test.tsx files.

index.tsx – contains the code responsible for the component logic.

styles.tsx – contains the code responsible for styling the component. Here is where styled-components shines.

.text.tsx – these contain the component unit tests.

Let me give you a short summary of the idea behind each of the components in the components folder. Starting top to bottom:

Button renders a button with a given background color and title. It's used for the two buttons in the footer, Toggle Feed and Kill Feed / Renew Feed.

DepthVisualizer is the component responsible for drawing the red and the green backgrounds you are seeing behind the numbers. It does this by rendering a row (an HTML div element) with given width, position being left (Bids) or right (Asks).

Footer – well, there's not much to say here, it contains the two buttons used in the app.

GroupingSelectBox renders the select box we use to change the grouping value, using setGrouping reducer to amend the application state when grouping is being changed.

Header renders the title of the application as well as the GroupingSelectBox component.

Loader renders loading animation implemented by leveraging SVG.

Order Book contains the core logic of the app. Separated components are located in sub-folders, and the Redux state management logic is here also.

Spread renders the spread value, displayed in the middle of the header (in desktop view). The component itself contains short methods for calculating the amount itself and the percentage value.

StatusMessage is a small component used to display status messages. It basically shows which market is currently being displayed and whether the feed is killed.

Rendering Performance

Here is a good moment to talk about rendering performance and inline styling a bit.

Rendering is the process of React asking your components to describe what they want their section of the UI to look like based on the current combination of props and state.

This process is triggered by a change of the state in your component. This change could be caused by some of the props being changed or by some internal logic of the component.

The point here is that when re-rendering happens unnecessarily, it reduces the performance of our app. This is exactly what happened to me when I introduced the initial implementation of the DepthVisualizer component. It was using styled-components, that is JavaScript, for the drawing part.

In order to solve this, I have changed the component to use inline styles, that is pure CSS, instead of a CSS in JS approach. In other words, my bottleneck was using JavaScript animations, which is a famous reason for reduced performance.

Here is how it looks like now:

const DepthVisualizer: FunctionComponent<DepthVisualizerProps> = ({windowWidth, depth, orderType }) => {
  return <div style={{
    backgroundColor: `${orderType === OrderType.BIDS ? DepthVisualizerColors.BIDS : DepthVisualizerColors.ASKS}`,
    height: "1.250em",
    width: `${depth}%`,
    position: "relative",
    top: 21,
    left: `${orderType === OrderType.BIDS && windowWidth > MOBILE_WIDTH ? `${100 - depth}%` : 0}`,
    marginTop: -24,
    zIndex: 1,
  }} />;
};

export default DepthVisualizer;

Inline styling is when you write your CSS along with your markup, as values for the style attribute. This is something that is NOT considered a good practice, but as you can see here, there are cases when it's necessary to use it.

💡 Usually you would extract your CSS code into a separate file.

Footer a simple dummy component used to render the two buttons in the footer of the app.

Dummy components, also known as stateless or representational ones, are components that don't hold state and are usually used just to visualize data in some way. This data is being passed via the props. For example the isFeedKilled flag in the component above.

If such a component needs to execute some kind of interaction, it usually does this by accepting (again via the props, for example toggleFeedCallback) callback functions that can be executed when that interaction happens. For example clicking a button.

On the opposite side we could have smart or state-full components. They are the ones that are connected to the app state and can manipulate it directly. Usually they are the ones that read the data from the state and pass it to the stateless components via their props.

GroupingSelectBox contains the Select element you can use to switch between the groupings.

Header is the header part of the app. It takes care of setting properly the layout consisting of the title 'Order Book' on the left and the select box on the right.

Loader is used as an indicator for when the data has not yet been loaded. It leverages a SVG animation I have found online.

Order Book is where the real thing is happening. This one consists of a few smaller components:

• TableContainer – used for styling the views for both the Odds and Bets sides.

• TitleRow – this is the component responsible for displaying the titles of the columns: prize, size, and total, respectively.

How to Build the UI with React and styled-components

When we talk about component-based structure, such as the one React provides us, the styled-components library is likely one of the first choices you might make when styling is needed.

Like Josh Comeau says in his detailed article:

💡 It's a wonderful tool. In many ways, it's changed how I think about CSS architecture, and has helped me keep my codebase clean and modular, just like React!

As the name of the lib hints, we could easily style our components by using the CSS-in-JS pattern. Here is an example of how I used it to write the styles for my Button component:

import styled from "styled-components";

interface ContainerProps {
  backgroundColor: string;
}

export const Container = styled.button<ContainerProps>`
  padding: .3em .7em;
  margin: 1em;
  border-radius: 4px;
  border: none;
  color: white;
  background: ${props => props.backgroundColor};
  font-family: "Calibri", sans-serif;
  font-size: 1.2em;

  &:hover {
    cursor: pointer;
    opacity: .8;
  }
`

Notice how I am using an interface in my styles file, and also the background property being passed as an argument via props. This is part of the CSS-in-JS story.

The possibility to use CSS code in JavaScript or (as someone might say) vice versa comes very handy. For example, when we need a component to look differently depending on something, we can pass through its props a parameter to define this.

As every style is actually a component, this way of writing styles feels a lot like writing React components. I mean, in the end, everything is components, right?

Responsiveness and Page Visibility Detection

While working on this app, I read in several places that, for applications which support rapid updates, is a good practice to implement some kind of mechanism for pausing the whole thing when it is not being used by the user. For example when the user minimizes the browser window or just opens another tab.

Since our Order book is consuming a lot of new chunks of data every second via WSS, I decided to implement such a mechanism as well.

What this does is:

• it shows a loader when the data is not there yet

• it changes the meta title to signify that the app is in paused mode

• it unpauses the work once the app window is on focus

Active mode

Paused mode

You may see the whole implementation here.

The essential part is in the useEffect hook, which is triggered only once when the application renders for first time.

In there we take advantage of the Page Visibility API by attaching the necessary listeners. And then, in the handlers, we simply execute the logic we want.

Window Size Detection

In almost every app that has some level of responsiveness, you need some logic for detecting the changes in the window size and taking some actions accordingly.

In other words, you need to know when your app is being viewed in certain screen size, so you can arrange your components and adjust your styles so that everything looks nice and in place.

This is especially valid for mobile friendly applications, where responsiveness is essential.

Our implementation of the window size change detection is based on the innerWidtgh property of the browser window object and onresize event that is being triggered when it gets resized.

I am attaching a listener for this event in a useEffect hook in App.tsx file. And then, every time the window size changes, I am setting the new width to a state variable via setWindowWidth hook.

const [windowWidth, setWindowWidth] = useState(0);
...
...

// Window width detection
useEffect(() => {
  window.onresize = () => {
    setWindowWidth(window.innerWidth);
  }
  setWindowWidth(() => window.innerWidth);
}, []);

Then propagate this variable down through all interested components and use it accordingly. For example here is how I use it in Order Book/index.tsx in order to know when and where to render the TitleRow component.

{windowWidth > MOBILE_WIDTH && <TitleRow windowWidth={windowWidth} reversedFieldsOrder={false} />}

TitleRow component - desktop view

TitleRow component - mobile view

Note that it appears on different position depending on that whether you are seeing the app on desktop or mobile.

You may take a look at the component itself and see similar approach of using the window width there.

State Management with Redux

As you probably guessed already, I used Redux for managing the state of the app.

The main logic behind that is concentrated in the orderbookSlice reducer. In the following few lines I will walk you through it and see how and why I built it that way.

First we define the interface and the initial state of our order book data. The initial state contains the default values we need to have in place when starting the app.

export interface OrderbookState {
  market: string;
  rawBids: number[][];
  bids: number[][];
  maxTotalBids: number;
  rawAsks: number[][];
  asks: number[][];
  maxTotalAsks: number;
  groupingSize: number;
}

const initialState: OrderbookState = {
  market: 'PI_XBTUSD', // PI_ETHUSD
  rawBids: [],
  bids: [],
  maxTotalBids: 0,
  rawAsks: [],
  asks: [],
  maxTotalAsks: 0,
  groupingSize: 0.5
};

Then there are a few short, self-explanatory methods helping to manipulate the levels data:

const removePriceLevel = (price: number, levels: number[][]): number[][] => levels.filter(level => level[0] !== price);

const updatePriceLevel = (updatedLevel: number[], levels: number[][]): number[][] => {
  return levels.map(level => {
    if (level[0] === updatedLevel[0]) {
      level = updatedLevel;
    }
    return level;
  });
};

const levelExists = (deltaLevelPrice: number, currentLevels: number[][]): boolean => currentLevels.some(level => level[0] === deltaLevelPrice);

const addPriceLevel = (deltaLevel: number[], levels: number[][]): number[][] => {
  return [ ...levels, deltaLevel ];
};

Then the real magic is happening. If the size returned by a delta is 0 then that price level should be removed from the order book. Otherwise you can safely overwrite the state of that price level with new data returned by that delta.

/** The orders returned by the feed are in the format
 of [price, size][].
 * @param currentLevels Existing price levels - `bids` or `asks`
 * @param orders Update of a price level
 */
const applyDeltas = (currentLevels: number[][], orders: number[][]): number[][] => {
  let updatedLevels: number[][] = currentLevels;

  orders.forEach((deltaLevel) => {
    const deltaLevelPrice = deltaLevel[0];
    const deltaLevelSize = deltaLevel[1];

    // If new size is zero - delete the price level
    if (deltaLevelSize === 0 && updatedLevels.length > ORDERBOOK_LEVELS) {
      updatedLevels = removePriceLevel(deltaLevelPrice, updatedLevels);
    } else {
      // If the price level exists and the size is not zero, update it
      if (levelExists(deltaLevelPrice, currentLevels)) {
        updatedLevels = updatePriceLevel(deltaLevel, updatedLevels);
      } else {
        // If the price level doesn't exist in the orderbook and there are less than 25 levels, add it
        if (updatedLevels.length < ORDERBOOK_LEVELS) {
          updatedLevels = addPriceLevel(deltaLevel, updatedLevels);
        }
      }
    }
  });

  return updatedLevels;
}

What follows after these are few helper methods. Let me say a few words about each of them now:

• addTotalSums – with the help of this method, we iterate through the orders data, bids or asks, and calculate for each of them the total sum. The total sum value is then used for making the background visualizations.

• addDepths – we use this method to calculate the so-called depth for each order. These values will be used later by the depth meter component to display the red and green rows in the background.

• getMaxTotalSum – this one returns the max value of all total sums.

Everything below is what we use for creating the application state. As per the Redux Toolkit documentation, it’s using createSliceAPI to create the slice.

Redux state

Creating a slice requires a string name to identify the slice, an initial state value, and one or more reducer functions to define how the state can be updated.

Once a slice is created, we can export the generated Redux action creators and the reducer function for the whole slice.

The last few lines consist of the exports in question – action creators, state slices selectors and the main reducer.

export const { addBids, addAsks, addExistingState, setGrouping, clearOrdersState } = orderbookSlice.actions;

export const selectBids = (state: RootState): number[][] => state.orderbook.bids;
export const selectAsks = (state: RootState): number[][] => state.orderbook.asks;
export const selectGrouping = (state: RootState): number => state.orderbook.groupingSize;
export const selectMarket = (state: RootState): string => state.orderbook.market;

export default orderbookSlice.reducer;

With all that, our state manipulation logic is complete. 🎉

Now it’s time to take a look at the protocol we used in our app to take advantage of all these rapid changes in the data we consume.

Websocket Protocol (WSS)

As you may have noticed, we're using the Web Socket communication protocol for fetching data into our application. We also use its features, as you will see in a moment, to accomplish other things (such as toggling the feeds and subscribe/unsubscribe from the data channel).

Here is how I used it.

Instead of trying to rely on manual implementation, I used the react-use-websocket package. It gives you all you need when you want to leverage WSS in a React app. If you want to go into details about this, you may take a look at their documentation.

A Few Words About My Implementation

What we need fist is the endpoint URL where the data feeds are coming from. I am sure there are multiple options out there when we talk about cryptocurrencies. In our app I used the one provided by www.cryptofacilities.com/.

const WSS_FEED_URL: string = 'wss://www.cryptofacilities.com/ws/v1';

Then the only thing we need to do to start consuming the data is to put the useWebSocket hook to work. As you may have guessed already, this hook is provided by the package mentioned above.

import useWebSocket from ["react-use-websocket"](<https://github.com/robtaussig/react-use-websocket>);

...
...
...

const { sendJsonMessage, getWebSocket } = useWebSocket(WSS_FEED_URL, {
    onOpen: () => console.log('WebSocket connection opened.'),
    onClose: () => console.log('WebSocket connection closed.'),
    shouldReconnect: (closeEvent) => true,
    onMessage: (event: WebSocketEventMap['message']) =>  processMessages(event)
  });

We pass the endpoint as the first argument and a few callback functions after that. These help us perform certain actions when one of the following happens:

• onOpen – what to do when WebSocket connection is established.

• onClose – what to do when WebSocket connection is terminated.

• shouldReconnect – this is just a flag, saying if we want automatic reconnect when the connection drops for some reason.

• onMessage – this is the main event that brings us the chunks with the data (I call processMessage method every time when that happens. This means that every time when a new chunk of data is received, we process it and display it respectively).

Down below is the method in question. It simply does two things:

• Either calls a method called process (No pun intended 😄) – this method is called every time new data for bids or asks is received and it processes it accordingly.

• Dispatches an event that is using one of the reducer functions we have seen earlier. This function practically creates the initial state of our application.

In order to decide whether we are adding data to the current state or we should initialize it, we check for a property called numLevels. This is something that comes from the API, the very first time we establish the WebSocket connection.

Initial payload

The rest of the code you see in this file is mostly for preparing and rendering the results on the screen.

The most interesting part would be the method buildPriceLevels that is used for both halves – bids and asks. It sorts the data, makes the necessary calculations, and passes it to the relevant components for visualizing it. Those are DepthVisualizer and PriceLevelRow I mentioned earlier in this article.

Grouping

The grouping is an important part of how the order book works, as it defines by what ticket size the orders are grouped.

In our application, I have implemented a toggling functionality per each market, that allows grouping it as follows:

• Between 0.5, 1, 2.5 for XBTUSD market.

XBTUSD market grouping

• Between 0.05, 0.1 and 0.25 for ETHUSD market.

ETHUSD market grouping

There is a short gist I created when trying to figure out how to implement the grouping logic. You may find it here.

Also, aside from that gist, while developing this I have performed more than a few experiments out of the project itself. And just because these are just local files on my computer, I will publish them here for those of you who are even more curious.

It’s a small side npm project that has only one dependency. Here is the package.json file:

{
  "name": "grouping",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "lodash.groupby": "^4.6.0"
  }
}

And here is the code itself:

const bids = [
    [
        50163,
        110
    ],
    [
        50162,
        13140
    ],
    [
        50158,
        3763
    ],
    [
        50156,
        1570
    ],
    [
        50155,
        21997
    ],
    [
        50152.5,
        450
    ],
    [
        50151,
        4669
    ],
    [
        50150.5,
        10329
    ],
    [
        50150,
        2500
    ],
    [
        50149.5,
        450
    ],
    [
        50149,
        4022
    ],
    [
        50148,
        20000
    ],
    [
        50147,
        5166
    ],
    [
        50146.5,
        5274
    ],
    [
        50145,
        174609
    ],
    [
        50143,
        20000
    ],
    [
        50141,
        28000
    ],
    [
        50140.5,
        5000
    ],
    [
        50138,
        6000
    ],
    [
        50132.5,
        4529
    ],
    [
        50132,
        4755
    ],
    [
        50131,
        12483
    ],
    [
        50128.5,
        61115
    ],
    [
        50128,
        23064
    ],
    [
        50125.5,
        181363
    ]
]

/* function roundDownNearest(num, acc) {
    if (acc < 0) {
        return Math.floor(num * acc) / acc;
    } else {
        return Math.floor(num / acc) * acc;
    }
} */

/* function groupByTicketSize(ticketSize, levels) {
    const result = levels.map((element, idx) => {
        const nextLevel = levels[idx + 1];

        if (nextLevel) {
            const currentPrice = element[0];
            const currentSize = element[1];
            const nextPrice = nextLevel[0];
            const nextSize = nextLevel[1];
            console.log("current level: ", element)
            console.log("next level: ", nextLevel)

            element[0] = roundDownNearest(currentPrice, ticketSize);

            if (currentPrice - nextPrice < ticketSize) {
                element[1] = currentSize + nextSize;
            }
            console.log("==================================> Result: ", element)

            return element;
        }

    }).filter(Boolean); 


    console.log("============================================================");
    console.log(result)
} */

const test = [
    [1004.5, 1],
    [1001.5, 1],
    [1001,   1],
    [1000.5, 1],
    [1000,   1],
    [999.5,  1],
    [999,    1],
    [990,    1],
    [988,    1]
]

function groupByTicketSize(ticketSize, levels) {
    const result = [];

    for (let i = 0; i < levels.length; i++) {
        console.log(levels[i])
        const prevLevel = levels[i-1]
        const level1 = levels[i]
        const level2 = levels[i+1]

        if (prevLevel && level1 && level1[0] - ticketSize === prevLevel) return

        if (level2 && level1[0] - level2[0] < ticketSize) {
            const newLevel = [level2[0], level1[1] + level2[1]];
            console.log("newLevel", newLevel)
            result.push(newLevel);
        } else {
            result.push(level1)
        }
    }

    console.log("============================================================");
    console.log(result)
}

// groupByTicketSize(1, bids);
groupByTicketSize(1, test);

How to Perform Unit Tests on the App

For performing unit testing I used the react-testing-library.

The main idea behind it that the developer should write tests only for what the user will see and interact with. There is no much point of testing implementation details.

💡 Imagine, just to give you an example, that you have implemented a list component that is just displaying lines of textual data. Say something like a todo list.

Then imagine that this data is coming from an API call in the shape of array. A data structure that you could easily iterate through via various methods – some sort of a loop cycle, such as for() or while(). Or you could use another more functional approach, say .map() method.

Now ask yourself – for the end user, the one that will just see the listed text data, does your implementation matter? As long as everything works as expected and in a good, performant way, the answer is ‘no, it does not’.

This is what your tests should reflect.

In the context of our Order Book application, each test file is located in the same directory as the implementation file. Most of the tests are short and self-explanatory, due to the fact that these are testing mostly rendering logic and only the happy path.

For example let’s take a look at the button component tests below:

import React from 'react';
import { render, screen } from '@testing-library/react';
import Button from './index';

test('renders button with title', () => {
  render(<Button backgroundColor={'red'} callback={jest.fn} title={'Toggle'} />);
  const btnElement = screen.getByText(/Toggle/i);
  expect(btnElement).toBeInTheDocument();
});

It just verifies that the component is rendered properly and it displays what we expect the user to see. Which is the title Toggle in this case.

For testing the reducers I have used Jest, as this is the only not visual part that we'll cover. These tests are also pretty simple and self-explanatory. I use them for testing whether the initial application state is in place and to see that adding price levels to that state works correctly.

How to Deploy the App to Vercel

Finally – deployment time. 🎉

After finishing the development and testing our application, let’s put it live.

I used the Vercel platform for this purpose. They offer a pretty rich and easy to use interface as well as integrations for all famous source control platforms out there – including, of course, GitHub (where our application repo lives).

Assuming you have a GitHub account, what you need to do if you want to deploy it on your own is to login with it here.

Vercel login screen

Click on the +New Project button in the top right corner. Then import your Git repository using the provided options in the screen that opens. Here is how mine looks:

Vercel Import Git Repository screen

After importing the project, you will be able to do the actual deploy. When finished, Vercel will generate URLs for you to access your newly deployed app.

Vercel production deployment screen

And I think you will receive an email letting you know if your deployment was successful. That email also contains these URLs.

Vercel successful deployment email

Congratulations! 👏🏻

You now have your own Order Book application up and running online.

How to Add a Build Badge on GitHub

This is not order book related, but I decided to share it with you here anyway. It’s about those small details that make the big picture somehow more complete and attractive.

Maybe some of you have wondered how can you get one of these so called badges?

Here is the answer: https://shields.io/.

You go to the Other section and find the GitHub Deployments option.

Then click on it and follow the instructions.

There is one more thing you need to do in order to have this fully functioning. You go to your GitHub repository → Actions tab and create new workflow file. You may just go ahead and copy the content of mine from here. Name it main.yml.

What this will do is run the jobs defined in that file. In our case this is just the build job which is basically spinning a new build and running the tests.

After completing this, you just need add the following lines to your README file:

<!-- prettier-ignore-start -->
[![Tests](<https://github.com/mihailgaberov/orderbook/actions/workflows/main.yml/badge.svg>)](<https://github.com/mihailgaberov/orderbook/actions/workflows/main.yml>)
[![Build Status][build-badge]][build]

[build-badge]: <https://img.shields.io/github/deployments/mihailgaberov/orderbook/production?label=vercel&logoColor=vercel>
[build]: <https://github.com/mihailgaberov/orderbook/deployments>
<!-- prettier-ignore-end -->

💡 Don’t forget to put your own details in the URLs, that is your GitHub username and the name of your repository.

After pushing these changes you should see the badges displayed on your README: 🥳.

GitHub badges

Wrapping Up

If you are reading this from the beginning, I will name you a champion. 🍾

It has been a long trip, but hopefully interesting and fun to walk along with me!

Now it’s time to summarize what we have done here and try to extract some useful insights which will help us in our future development challenges.

I will layout below my opinion of what was the most challenging in building this application. And I will be even more eager to find out what is yours.

Rendering Performance

This really bit me in the beginning, when I was building the UI and was trying to implement the drawing of the price level rows.

I mentioned earlier how I have managed to solve it and I think this is going to be something I will remember for sure.

Grouping Functionality

Implementing this was also kind of challenging because there were several factors I had to take into account. Because of the market we are in and the range I had to do the calculations in.

It took me a while to polish it (remember the side mini project and the gist I shared in the previous sections) and I still think it could be improved even more. Try switching between the markets and the grouping values multiple times and observe the results.

Space for Improvement

One thing already mentioned is for sure the grouping. Which should also improve the visualizing of the red and green parts – they (almost) always should form a not ideal triangle.

If we try to look at the bigger picture, this order book application can be a part of a dashboard screen filled with other widgets as well, and they all can interact between them.

For example, changing the grouping of the order book to reflect on changing the views in the other widgets as well – say showing a market chart like this one below:

I am not even mentioning adding new markets as an improvement, as it’s kinda clear. But this should be taken into account when building the functionality for the current markets, as to do it in a way that will be easily extendable. So that adding a new market to the order book be a trivial and quick task to do.

I think that's all from me.

Thanks for reading! 🙏

References

Here are few links you might find useful to read:

The styled-components Happy Path

Blogged Answers: A (Mostly) Complete Guide to React Rendering Behavior

I think it's helpful to work with a practical use case because it is much simpler to understand the concepts when you can relate them to real life.

So in this guide, you will be learning what web workers are in JavaScript, you'll get a brief introduction to WebSockets, and you'll see how you can manage sockets in the proper way.

This article is quite application/hands-on oriented, so I would suggest trying the example out as you go along to get a much better understanding.

Let’s dive in.

Table of contents

• Prerequisites
• What are web workers in JavaScript?
• Brief introduction to web sockets
• Use case description
• Project structure
• Client and Server architecture
• Worker System
• Communication between the UI and the socket via web worker
• Summary

Prerequisites

Before you start reading this article, you should have a basic understanding of the following topics:

• Class Diagrams: We are going to use them to showcase our example. Here are a couple resources you can use to learn more about them:
  • Class diagrams
  • UML Diagram course
• Context Diagram and Container Diagrams
• React
• Web sockets
  • Introduction to sockets
  • How JavaScript works: Deep dive into WebSockets and HTTP/2 with SSE + how to pick the right path
• Difference between scope and context
• Global objects

What are web workers in JavaScript?

A web worker is a piece of browser functionality. It is the real OS threads that can be spawned in the background of your current page so that it can perform complex and resource-intensive tasks.

Imagine that you have some large data to fetch from the server, or some complex rendering needs to be done on the UI. If you do this directly on your webpage then the page might get jankier and will impact the UI.

To mitigate this, you can simply create a thread – that is a web worker – and let the web worker take care of the complex stuff.

You can communicate with the web worker in a pretty simple manner which can be used to transfer data to and fro from the worker to the UI.

Common examples of web workers would be:

• Dashboard pages that display real-time data such as stock prices, real-time active users, and so on
• Fetching huge files from the server
• Autosave functionality

You can create a web worker using the following syntax:

const worker = new Worker("<worker_file>.js");

Worker is an API interface that lets you create a thread in the background. We need to pass a parameter, that is a <worker_file>.js file. This specifies the worker file the API needs to execute.

NOTE: A thread is created once a Worker call is initiated. This thread only communicates with its creator, that is the file which created this thread.

A worker can be shared or used by multiple consumers/scripts. These are called shared workers. The syntax of the shared worker is very similar to that of the above mentioned workers.

const worker = new SharedWorker("<worker_file>.js");

You can read more about SharedWorkers in this guide.

History of web workers

Web workers execute in a different context, that is they do not execute in a global scope such as window context. Web workers have their own dedicated worker context which is called DedicatedWorkerGlobalScope.

There are some cases where you can't use web workers, though. For example, you can't use them to manipulate the DOM or the properties of the window object. This is because the worker does not have the access to the window object.

Web workers can also spawn new web workers. Web workers communicate with their creator using certain methods like postMessage, onmessage, and onerror. We will look into these methods closely in the later sections of this article.

Brief Introduction to Web Sockets

A web socket is a type of communication that happens between two parties/entities using a WebSocket protocol. It actually provides a way to communicate between the two connected entities in a persistent manner.

You can create a simple web socket like below:

const socket = new WebSocket("ws://example.com");

Over here we have created a simple socket connection. You'll notice that we have passed a parameter to the WebSocket constructor. This parameter is a URL at which the connection should be established.

You can read more about web sockets by referring to the Websockets link in the prerequisites.

Use Case Description

NOTE: Context, Container, and Class diagrams drawn in this blog post don't accurately follow the exact conventions of these diagrams. They're approximated here so that you can understand the basic concepts.

Before we start, I would suggest reading up on c4models, container diagrams, and context diagrams. You can find resources about them in the prerequisites section.

In this article, we are going to consider the following use case: data transfer using web workers via socket protocol.

We are going to build a web application which will plot the data on a line chart every 1.5 seconds. The web application will receive the data from the socket connection via web workers. Below is the context diagram of our use case:

Container Diagram

As you can see from the above diagram, there are 4 main components to our use case:

1. Person: A user who is going to use our application
2. Software system: Client App – This is the UI of our application. It consists of DOM elements and a web worker.
3. Software system: Worker system – This is a worker file that resides in the client app. It is responsible for creating a worker thread and establishing the socket connection.
4. Software system: Server application – This is a simple JavaScript file which can be executed by node to create a socket server. It consists of code which helps to read messages from the socket connection.

Now that we understand the use case, let's dive deep into each of these modules and see how the whole application works.

Project Structure

Please follow this link to get the full code for the project that I developed for this article.

Our project is divided into two folders. First is the server folder which consists of server code. The second is the client folder, which consists of the client UI, that is a React application and the web worker code.

Following is the directory structure:

├── client
│   ├── package.json
│   ├── package-lock.json
│   ├── public
│   │   ├── favicon.ico
│   │   ├── index.html
│   │   ├── logo192.png
│   │   ├── logo512.png
│   │   ├── manifest.json
│   │   └── robots.txt
│   ├── README.md
│   ├── src
│   │   ├── App.css
│   │   ├── App.jsx
│   │   ├── components
│   │   │   ├── LineChartSocket.jsx
│   │   │   └── Logger.jsx
│   │   ├── index.css
│   │   ├── index.js
│   │   ├── pages
│   │   │   └── Homepage.jsx
│   │   ├── wdyr.js
│   │   └── workers
│   │       └── main.worker.js
│   └── yarn.lock
└── server
    ├── package.json
    ├── package-lock.json
    └── server.mjs

To run the application, you first need to start the socket server. Execute the following commands one at a time to start the socket server (assuming you are in the parent directory):

cd server
node server.mjs

Then start the client app by running the following commands (assuming you are in the parent directory):

cd client
yarn run start

Open http://localhost:3000 to start the web app.

Client and Server Application

The client application is a simple React application, that is CRA app, which consists of a Homepage. This home page consists of the following elements:

• Two buttons: start connection and stop connection which will help to start and stop the socket connection as required.
• A line chart component - This component will plot the data that we receive from the socket at regular intervals.
• Logged message - This is a simple React component that will display the connection status of our web sockets.

Below is the container diagram of our client application.

Container Diagram: Client Application

Below is how the UI will look:

Actual UI

To check out the code for the client UI, go to the client folder. This is a regular create-react-app, except that I have removed some boilerplate code that we don't need for this project.

App.jsx is actually the starter code. If you check this out, we have called the <Homepage /> component in it.

Now let's have a look at the Homepage component.

const Homepage = () => {
  const [worker, setWorker] = useState(null);
  const [res, setRes] = useState([]);
  const [log, setLog] = useState([]);
  const [buttonState, setButtonState] = useState(false);

  const hanldeStartConnection = () => {
    // Send the message to the worker [postMessage]
    worker.postMessage({
      connectionStatus: "init",
    });
  };

  const handleStopConnection = () => {
    worker.postMessage({
      connectionStatus: "stop",
    });
  };

    //UseEffect1
  useEffect(() => {
    const myWorker = new Worker(
      new URL("../workers/main.worker.js", import.meta.url)
    ); //NEW SYNTAX
    setWorker(myWorker);

    return () => {
      myWorker.terminate();
    };
  }, []);

    //UseEffect2
  useEffect(() => {
    if (worker) {
      worker.onmessage = function (e) {
        if (typeof e.data === "string") {
          if(e.data.includes("[")){
            setLog((preLogs) => [...preLogs, e.data]);
          } else {
            setRes((prevRes) => [...prevRes, { stockPrice: e.data }]);
          }
        }

        if (typeof e.data === "object") {
          setButtonState(e.data.disableStartButton);
        }
      };
    }
  }, [worker]);

  return (
    <>
      <div className="stats">
        <div className="control-panel">
          <h3>WebWorker Websocket example</h3>
          <button
            id="start-connection"
            onClick={hanldeStartConnection}
            disabled={!worker || buttonState}
          >
            Start Connection
          </button>
          &nbsp;
          <button
            id="stop-connection"
            onClick={handleStopConnection}
            disabled={!buttonState}
          >
            Stop Connection
          </button>
        </div>
        <LineChartComponent data={res} />
      </div>
      <Logger logs={log}/>
    </>
  );
};

As you can see, it's just a regular functional component that renders two buttons – a line chart, and a custom component Logger.

Now that we know how our homepage component looks, let's dive into how the web worker thread is actually created. In the above component you can see there are two useEffect hooks used.

The first one is used for creating a new worker thread. It's a simple call to the Worker constructor with a new operator as we have seen in the previous section of this article.

But there are some difference over here: we have passed an URL object to the worker constructor rather than passing the path of the worker file in the string.

const myWorker = new Worker(new URL("../workers/main.worker.js", import.meta.url));

You can read more about this syntax here.

If you try to import this web worker like below, then our create-react-app won’t be able to load/bundle it properly so you will get an error since it has not found the worker file during bundling:

const myWorker = new Worker("../workers/main.worker.js");

Next, we also don’t want our application to run the worker thread even after the refresh, or don’t want to spawn multiple threads when we refresh the page. To mitigate this, we'll return a callback in the same useEffect. We use this callback to perform cleanups when the component unmounts. In this case, we are terminating the worker thread.

We use the useEffect2 to handle the messages received from the worker.

Web workers have a build-in property called onmessage which helps receive any messages sent by the worker thread. The onmessage is an event handler of the worker interface. It gets triggered whenever a message event is triggered. This message event is generally triggered whenever the postMessage handler is executed (we will look more into this in a later section).

So in order for us to send a message to the worker thread, we have created two handlers. The first is handleStartConnection and the second is handleStopConnection. Both of them use the postMessage method of the worker interface to send the message to the worker thread.

We will talk about the message {connectionStatus: init} in our next section.

You can read more about the internal workings of the onmessage and postMessage in the following resources:

• Onmessage
• Postmessage

Since we now have a basic understanding about how our client code is working, then let's move on to learn about the Worker System in our context diagram above.

Worker System

To understand the code in this section, make sure you go through the file src/workers/main.worker.js.

To help you understand what's going on here, we will divide this code into three parts:

1. A self.onmessage section
2. How the socket connection is managed using the socketManagement() function
3. Why we need the socketInstance variable at the top

How self.onmessage works

Whenever you create a web worker application, you generally write a worker file which handles all the complex scenarios that you want the worker to perform. This all happens in the main.worker.js file. This file is our worker file.

In the above section, we saw that we established a new worker thread in the useEffect. Once we created the thread, we also attached the two handlers to the respective start and stop connection buttons.

The start connection button will execute the postMessage method with message: {connectionStatus: init} . This triggers the message event, and since the message event is triggered, all the message events are captured by the onmessage property.

In our main.worker.js file, we have attached a handler to this onmessage property:

self.onmessage = function (e) {
  const workerData = e.data;
  postMessage("[WORKER] Web worker onmessage established");
  switch (workerData.connectionStatus) {
    case "init":
      socketInstance = createSocketInstance();
      socketManagement();
      break;

    case "stop":
      socketInstance.close();
      break;

    default:
      socketManagement();
  }
}

So whenever any message event is triggered in the client, it will get captured in this event handler.

The message {connectionStatus: init} that we send from the client is received in the event e. Based on the value of connectionStatus we use the switch case to handle the logic.

NOTE: We have added this switch case because we need to isolate some part of the code which we do not want to execute all the time (we will look into this in a later section).

How the socket connection is managed using the socketManagement() function

There are some reasons why I have shifted the logic of creating and managing a socket connection into a separate function. Here is the code for a better understanding of the point I am trying to make:

function socketManagement() {
  if (socketInstance) {
    socketInstance.onopen = function (e) {
      console.log("[open] Connection established");
      postMessage("[SOCKET] Connection established");
      socketInstance.send(JSON.stringify({ socketStatus: true }));
      postMessage({ disableStartButton: true });
    };

    socketInstance.onmessage = function (event) {
      console.log(`[message] Data received from server: ${event.data}`);
      postMessage( event.data);
    };

    socketInstance.onclose = function (event) {
      if (event.wasClean) {
        console.log(`[close] Connection closed cleanly, code=${event.code}`);
        postMessage(`[SOCKET] Connection closed cleanly, code=${event.code}`);
      } else {
        // e.g. server process killed or network down
        // event.code is usually 1006 in this case
        console.log('[close] Connection died');
        postMessage('[SOCKET] Connection died');
      }
      postMessage({ disableStartButton: false });
    };

    socketInstance.onerror = function (error) {
      console.log(`[error] ${error.message}`);
      postMessage(`[SOCKET] ${error.message}`);
      socketInstance.close();
    };
  }
}

This is a function that will help you manage your socket connection:

• For receiving the message from the socket server we have the onmessage property which is assigned an event handler.
• Whenever a socket connection is opened, you can perform certain operations. To do that we have the onopen property which is assigned to an event handler.
• And if any error occurs or when we are closing the connection then, we use onerror and onclose properties of the socket.

For creating a socket connection there is a separate function altogether:

function createSocketInstance() {
  let socket = new WebSocket("ws://localhost:8080");

  return socket;
}

Now all of these functions are called in a switch case like below in the main.worker.js file:

self.onmessage = function (e) {
  const workerData = e.data;
  postMessage("[WORKER] Web worker onmessage established");
  switch (workerData.connectionStatus) {
    case "init":
      socketInstance = createSocketInstance();
      socketManagement();
      break;

    case "stop":
      socketInstance.close();
      break;

    default:
      socketManagement();
  }
}

So based on what message the client UI sends to the worker the appropriate function will be executed. It is pretty self-explanatory on what message which particular function should be triggered, based on the above code.

Now consider a scenario where we placed all the code inside self.onmessage.

self.onmessage = function(e){
    console.log("Worker object present ", e);
    postMessage({isLoading: true, data: null});

    let socket = new WebSocket("ws://localhost:8080");

        socket.onopen = function(e) {
          console.log("[open] Connection established");
          console.log("Sending to server");
          socket.send("My name is John");
        };

        socket.onmessage = function(event) {
          console.log(`[message] Data received from server: ${event.data}`);
        };

        socket.onclose = function(event) {
          if (event.wasClean) {
            console.log(`[close] Connection closed cleanly, code=${event.code} reason=${event.reason}`);
          } else {
            // e.g. server process killed or network down
            // event.code is usually 1006 in this case
            console.log('[close] Connection died');
          }
        };

            socket.onerror = function(error) {
              console.log(`[error] ${error.message}`);
            };
}

This would cause the following problems:

1. On every postMessage call made by the client UI, there would have been a new socket instance.
2. It would have been difficult to close the socket connection.

Because of these reasons, all the socket management code is written in a function socketManagement and catered using a switch case.

Why we need the socketInstance variable at the top

We do need a socketInstance variable at the top because this will store the socket instance which was previously created. It is a safe practice since no one can access this variable externally as main.worker.js is a separate module altogether.

Communication between the UI and the socket via web worker

Now that we understand which part of the code is responsible for which section, we will take a look at how we establish a socket connection via webworkers. We'll also see how we respond via socket server to display a line chart on the UI.

End-to-end flow of the application

NOTE: Some calls are purposefully not shown in the diagram since it will make the diagram cluttered. Make sure you refer to the code as well while referring to this diagram.

Now let's first understand what happens when you click on the start connection button on the UI:

1. One thing to notice over here is that our web worker thread is created once the component is mounted, and is removed/terminated when the component is unmounted.
2. Once the start connection button is clicked, a postMessage call is made with {connectionStatus: init}
3. The web worker’s onmessage event handler which is listening to all the message events comes to know that it has received connectionStatus as init. It matches the case, that is in the switch case of main.worker.js. It then calls the createSocketInstance() which returns a new socket connection at the URL: ws://localhost:8080
4. After this a socketManagement() function is called which checks if the socket is created and then executes a couple of operations.
5. In this flow, since the socket connection is just established therefore, socketInstance’s onpen event handler is executed.
6. This will send a {socketStatus: true} message to the socket server. This will also send a message back to the client UI via postMessage({ disableStartButton: true}) which tells the client UI to disable the start button.
7. Whenever the socket connection is established, then the server socket’s on('connection', ()=>{}) is invoked. So in step 3, this function is invoked at the server end.
8. Socket’s on('message', () => {}) is invoked whenever a message is sent to the socket. So at step 6, this function is invoked at the server end. This will check if the socketStatus is true, and then it will start sending a random integer every 1.5 seconds to the client UI via web workers.

Now that we understood how the connection is established, let's move on to understand how the socket server sends the data to the client UI:

1. As discussed above, socket server received the message to send the data, that is a random number every 1.5 second.
2. This data is recieved on the web worker’s end using the onmessage handler.
3. This handler then calls the postMessage function and sends this data to the UI.
4. After receiving the data it appends it to an array as a stockPrice object.
5. This acts as a data source for our line chart component and gets updated every 1.5 seconds.

Now that we understand how the connection is established, let's move on to understand how the socket server sends the data to the client UI:

1. As discussed above, socket server recieved the message to send the data, that is a random number, every 1.5 seconds.
2. This data is recieved on the web worker’s end using the socket's onmessage handler.
3. This handler then calls the postMessage function of the web worker and sends this data to the UI.
4. After receiving the data via useEffect2 it appends it to an array as a stockPrice object.
5. This acts as a data source for our line chart component and gets updated every 1.5 seconds.

NOTE: We are using recharts for plotting the line chart. You can find more information about it at the official docs.

Here is how our application will look in action:

Working Example

Summary

So this was a quick introduction to what web workers are and how you can use them to solve complex problems and create better UIs. You can use web workers in your projects to handle complex UI scenarios.

If you want to optimize your workers, read up on the below libraries:

• comlink
• thread.js

Thank you for reading!

Follow me on twitter, github, and linkedIn.

Here is a full breakdown of what he will cover in this 40 minute talk:

• The reason for this talk
• Why Websockets?
• Websockets minimal API example
• Rest Web Services
• Multiple Listeners
• Full Duplex
• Websockets time-line
• TypeScript
• Vendor Tools
• Auction IDL Definition

This article was written by Ania Kubow in support of the conference talk made by Steven Lemmo.

Code with Ania Kubów

Hello everyone. This channel is run by Ania Kubow. In this channel, I will be teaching you JavaScript,React, HTML, CSS, React-native, Node.js and so much more! A little bit about me:My background is in the financial markets, where I worked as a derivates broker our of University. After starting m…

YouTube

Have you ever wondered how chat applications work behind the scenes? Well, today I am going to walk you through how to make a REST + Sockets-based application built on top of NodeJS/ExpressJS using MongoDB.

I have been working on the content for this article for over a week now – I really hope it helps someone out there.

Prerequisites

• Set up Mongodb on your machine [Installation guide written here]
• For windows users, you can find the installation guide [here]
• For macOS users, you can find the installation guide [here][To the point installation that I wrote]
• For Linux users, you can find the installation guide [here]
• Install Node/NPM on your machine [Installation link here] (I am using Node version v12.18.0)

Topics we'll cover

General

• Create an express server
• How to do API validations
• Create basic skeleton for the entire application
• Setting up MongoDB (installation, setup in express)
• Creating users API + Database (Create a user, Get a user by id, Get all users, Delete a user by id)
• Understanding what a middleware is
• JWT (JSON web tokens) authentication (decode/encode) - Login middleware
• Web socket class that handles events when a user disconnects, adds its identity, joins a chat room, wants to mute a chat room
• Discussing chat room & chat message database model

For the API

• Initiate a chat between users
• Create a message in chat room
• See conversation for a chat room by its id
• Mark an entire conversation as read (similar to Whatsapp)
• Get recent conversation from all chats (similar to Facebook messenger)

Bonus - API

• Delete a chat room by id along with all its associated messages
• Delete a message by id

Before we begin, I wanted to touch on some basics in the following videos.

Understanding the basics of ExpressJS

What are routes? Controllers? How do we allow for CORS (cross origin resource sharing)? How do we allow enduser to send data in JSON format in API request?

I talk about all this and more (including REST conventions) in this video:

Also, here's a GitHub link to the entire source code of this video [Chapter 0]

Do have a look at the README.md for "Chapter 0" source code. It has all the relevant learning links I mention in the video along with an amazing half hour tutorial on postman.

Adding API validation to your API end-point

In the below video, you'll learn how to write your own custom validation using a library called "make-validation":

Here's the GitHub link to the entire source code of this video [Chapter 0].

And here's the make-validation library link [GitHub][npm][example].

The entire source code of this tutorial can be found here. If you have any feedback, please just reach out to me on http://twitter.com/adeelibr. If you like this tutorial kindly leave a star on the github repository**.**

Let's begin now that you know the basics of ExpressJS and how to validate a user response.

Getting started

Create a folder called chat-app:

mkdir chat-app;
cd chat-app;

Next initialize a new npm project in your project root folder by typing the following:

npm init -y

and install the following packages:

npm i cors @withvoid/make-validation express jsonwebtoken mongoose morgan socket.io uuid --save;
npm i nodemon --save-dev;

And in your package.json scripts section add the following 2 scripts:

"scripts": {
    "start": "nodemon server/index.js",
    "start:server": "node server/index.js"
},

Your package.json now should look something like this:

{
  "name": "chapter-1-chat",
  "version": "0.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "start": "nodemon server/index.js",
    "start:server": "node server/index.js"
  },
  "dependencies": {
    "@withvoid/make-validation": "1.0.5",
    "cors": "2.8.5",
    "express": "4.16.1",
    "jsonwebtoken": "8.5.1",
    "mongoose": "5.9.18",
    "morgan": "1.9.1",
    "socket.io": "2.3.0",
    "uuid": "8.1.0"
  },
  "devDependencies": {
    "nodemon": "2.0.4"
  }
}

Awesome!

Now in your project's root folder create a new folder called server:

cd chat-app;
mkdir server;
cd server;

Inside your server folder create a file called index.js and add the following content to it:

import http from "http";
import express from "express";
import logger from "morgan";
import cors from "cors";
// routes
import indexRouter from "./routes/index.js";
import userRouter from "./routes/user.js";
import chatRoomRouter from "./routes/chatRoom.js";
import deleteRouter from "./routes/delete.js";
// middlewares
import { decode } from './middlewares/jwt.js'

const app = express();

/** Get port from environment and store in Express. */
const port = process.env.PORT || "3000";
app.set("port", port);

app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));

app.use("/", indexRouter);
app.use("/users", userRouter);
app.use("/room", decode, chatRoomRouter);
app.use("/delete", deleteRouter);

/** catch 404 and forward to error handler */
app.use('*', (req, res) => {
  return res.status(404).json({
    success: false,
    message: 'API endpoint doesnt exist'
  })
});

/** Create HTTP server. */
const server = http.createServer(app);
/** Listen on provided port, on all network interfaces. */
server.listen(port);
/** Event listener for HTTP server "listening" event. */
server.on("listening", () => {
  console.log(`Listening on port:: http://localhost:${port}/`)
});

Let's add the routes for indexRouter userRouter chatRoomRouter & deleteRouter.

In your project's root folder create a folder called routes. Inside the routes folder add the following files:

• index.js
• user.js
• chatRoom.js
• delete.js

Let's add content for routes/index.js first:

import express from 'express';
// controllers
import users from '../controllers/user.js';
// middlewares
import { encode } from '../middlewares/jwt.js';

const router = express.Router();

router
  .post('/login/:userId', encode, (req, res, next) => { });

export default router;

Let's add content for routes/user.js next:

import express from 'express';
// controllers
import user from '../controllers/user.js';

const router = express.Router();

router
  .get('/', user.onGetAllUsers)
  .post('/', user.onCreateUser)
  .get('/:id', user.onGetUserById)
  .delete('/:id', user.onDeleteUserById)

export default router;

And now let's add content for routes/chatRoom.js:

import express from 'express';
// controllers
import chatRoom from '../controllers/chatRoom.js';

const router = express.Router();

router
  .get('/', chatRoom.getRecentConversation)
  .get('/:roomId', chatRoom.getConversationByRoomId)
  .post('/initiate', chatRoom.initiate)
  .post('/:roomId/message', chatRoom.postMessage)
  .put('/:roomId/mark-read', chatRoom.markConversationReadByRoomId)

export default router;

Finally, let's add content for routes/delete.js:

import express from 'express';
// controllers
import deleteController from '../controllers/delete.js';

const router = express.Router();

router
  .delete('/room/:roomId', deleteController.deleteRoomById)
  .delete('/message/:messageId', deleteController.deleteMessageById)

export default router;

Awesome now that our routes are in place let's add the controllers for each route.

Create a new folder called controllers. Inside that folder create the following files:

• user.js
• chatRoom.js
• delete.js

Let's start of with controllers/user.js:

export default {
  onGetAllUsers: async (req, res) => { },
  onGetUserById: async (req, res) => { },
  onCreateUser: async (req, res) => { },
  onDeleteUserById: async (req, res) => { },
}

Next let's add content in controllers/chatRoom.js:

export default {
  initiate: async (req, res) => { },
  postMessage: async (req, res) => { },
  getRecentConversation: async (req, res) => { },
  getConversationByRoomId: async (req, res) => { },
  markConversationReadByRoomId: async (req, res) => { },
}

And finally let's add content for controllers/delete.js:

export default {
  deleteRoomById: async (req, res) => {},
  deleteMessageById: async (req, res) => {},
}

So far we have added empty controllers for each route, so they don't do much yet. We'll add functionality in a bit.

Just one more thing – let's add a new folder called middlewares and inside that folder create a file called jwt.js. Then add the following content to it:

import jwt from 'jsonwebtoken';

export const decode = (req, res, next) => {}

export const encode = async (req, res, next) => {}

I will talk about what this file does in a bit, so for now let's just ignore it.

We are done with our basic boilerplate of the code base

We have ended up doing the following:

• Created an Express server that listens on port 3000
• Added cross-origin-resource (CORS) to our server.js
• Added a logger to our server.js
• And also added route handlers with empty controllers.

Nothing fancy so far that I haven't covered in the videos above.

Let's setup MongoDB in our application

Before we add MongoDB to our code base, make sure it is installed in your machine by running one of the following:

• For Windows users installation guide [here]
• For macOS users installation guide [here][To the point installation that I wrote]
• For Linux users installation guide [here]

If you are having issues installing MongoDB, just let me know at https://twitter.com/adeelibr and I'll write a custom guide for you or make an installation video guide. :)

I am using Robo3T as my MongoDB GUI.

Now you should have your MongoDB instance running and Robo3T installed. (You can use any GUI client that you like for this. I like Robo3T a lot so I'm using it. Also, it's open source.)

Here is a small video I found on YouTube that gives a 6 minute intro to Robo3T:

Once your MongoDB instance is up and running let's begin integrating MongoDB in our code as well.

In your root folder create a new folder called config. Inside that folder create a file called index.js and add the following content:

const config = {
  db: {
    url: 'localhost:27017',
    name: 'chatdb'
  }
}

export default config

Usually the default port that MongoDB instances will run on is 27017.

Here we set info about our database URL (which is in db) and the name of database which is chatdb (you can call this whatever you want).

Next create a new file called config/mongo.js and add the following content:

import mongoose from 'mongoose'
import config from './index.js'

const CONNECTION_URL = `mongodb://${config.db.url}/${config.db.name}`

mongoose.connect(CONNECTION_URL, {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

mongoose.connection.on('connected', () => {
  console.log('Mongo has connected succesfully')
})
mongoose.connection.on('reconnected', () => {
  console.log('Mongo has reconnected')
})
mongoose.connection.on('error', error => {
  console.log('Mongo connection has an error', error)
  mongoose.disconnect()
})
mongoose.connection.on('disconnected', () => {
  console.log('Mongo connection is disconnected')
})

Next import config/mongo.js in your server/index.js file like this:

.
.
// mongo connection
import "./config/mongo.js";
// routes
import indexRouter from "./routes/index.js";

If you get lost at any time, the entire source code for this tutorial is right here.

Let's discuss what we are doing here step by step:

We first import our config.js file in config/mongo.js. Next we pass in the value to our CONNECTION_URL like this:

const CONNECTION_URL = `mongodb://${config.db.url}/${config.db.name}`

Then using the CONNECTION_URL we form a Mongo connection, by doing this:

mongoose.connect(CONNECTION_URL, {
  useNewUrlParser: true,
  useUnifiedTopology: true
})

This tells mongoose to make a connection with the database with our Node/Express application.

The options we are giving Mongo here are:

• useNewUrlParser: MongoDB driver has deprecated their current connection string parser. useNewUrlParser: true tells mongoose to use the new parser by Mongo. (If it's set to true, we have to provide a database port in the CONNECTION_URL.)
• useUnifiedTopology: False by default. Set to true to opt in to using MongoDB driver's new connection management engine. You should set this option to true, except for the unlikely case that it prevents you from maintaining a stable connection.

Next we simply add mongoose event handlers like this:

mongoose.connection.on('connected', () => {
  console.log('Mongo has connected succesfully')
})
mongoose.connection.on('reconnected', () => {
  console.log('Mongo has reconnected')
})
mongoose.connection.on('error', error => {
  console.log('Mongo connection has an error', error)
  mongoose.disconnect()
})
mongoose.connection.on('disconnected', () => {
  console.log('Mongo connection is disconnected')
})

• connected will be called once the database connection is established
• disconnected will be called when your Mongo connection is disabled
• error is called if there is an error connecting to your Mongo database
• reconnected event is called when the database loses connection and then makes an attempt to successfully reconnect.

Once you have this in place, simply go in your server/index.js file and import config/mongo.js. And that is it. Now when you start up your server by typing this:

npm start;

You should see something like this:

Logs when you start your server

If you see this you have successfully added Mongo to your application.

Congratulations!

If you got stuck here for some reason, let me know at twitter.com/adeelibr and I will try to sort it out for you. :)

Let's setup our first API section for users/

The setup of our API for users/ will have no authentication token for this tutorial, because my main focus is to teach you about the Chat application here.

User Modal Scheme

Let's create our first model (database scheme) for the user collection.

Create a new folder called models. Inside that folder create a file called User.js and add the following content:

import mongoose from "mongoose";
import { v4 as uuidv4 } from "uuid";

export const USER_TYPES = {
  CONSUMER: "consumer",
  SUPPORT: "support",
};

const userSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    firstName: String,
    lastName: String,
    type: String,
  },
  {
    timestamps: true,
    collection: "users",
  }
);

export default mongoose.model("User", userSchema);

Let's break this down into pieces:

export const USER_TYPES = {
  CONSUMER: "consumer",
  SUPPORT: "support",
};

We are basically going to have 2 types of users, consumer and support. I have written it this way because I want to programmatically ensure API and DB validation, which I will talk about later.

Next we create a schema on how a single document (object/item/entry/row) will look inside our user collection (a collection is equivalent to a MySQL table). We define it like this:

const userSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    firstName: String,
    lastName: String,
    type: String,
  },
  {
    timestamps: true,
    collection: "users",
  }
);

Here we are telling mongoose that for a single document in our users collection we want the structure to be like this:

{
    id: String // will get random string by default thanks to uuidv4
        firstName: String,
        lastName: String,
        type: String // this can be of 2 types consumer/support
}

In the second part of the schema we have something like this:

{
    timestamps: true,
    collection: "users",
}

Setting timestamps to true will add 2 things to my schema: a createdAt and a updatedAt date value. Every time when we create a new entry the createdAt will be updated automatically and updatedAt will update once we update an entry in the database using mongoose. Both of these are done automatically by mongoose.

The second part is collection. This shows what my collection name will be inside my database. I am assigning it the name of users.

And then finally we'll export the object like this:

export default mongoose.model("User", userSchema);

So mongoose.model takes in 2 parameters here.

• The name of the model, which is User here
• The schema associated with that model, which is userSchema in this case

Note: Based on the name of the model, which is User in this case, we don't add collection key in the schema section. It will take this User name and append an s to it and create a collection by its name, which becomes user.

Great, now we have our first model.

If you've gotten stuck anywhere, just have a look at the source code.

Create a new user API [POST request]

Next let's write our first controller for this route: .post('/', user.onCreateUser).

Go inside controllers/user.js and import 2 things at the top:

// utils
import makeValidation from '@withvoid/make-validation';
// models
import UserModel, { USER_TYPES } from '../models/User.js';

Here we are importing the validation library that I talked about in the video at the very top. We are also importing our user modal along with the USER_TYPES from the same file.

This is what USER_TYPES represents:

export const USER_TYPES = {
  CONSUMER: "consumer",
  SUPPORT: "support",
};

Next find the controller onCreateUser and add the following content to it:

onCreateUser: async (req, res) => {
    try {
      const validation = makeValidation(types => ({
        payload: req.body,
        checks: {
          firstName: { type: types.string },
          lastName: { type: types.string },
          type: { type: types.enum, options: { enum: USER_TYPES } },
        }
      }));
      if (!validation.success) return res.status(400).json(validation);

      const { firstName, lastName, type } = req.body;
      const user = await UserModel.createUser(firstName, lastName, type);
      return res.status(200).json({ success: true, user });
    } catch (error) {
      return res.status(500).json({ success: false, error: error })
    }
  },

Let's divide this into 2 sections.

First we validate the user response by doing this:

const validation = makeValidation(types => ({
  payload: req.body,
  checks: {
    firstName: { type: types.string },
    lastName: { type: types.string },
    type: { type: types.enum, options: { enum: USER_TYPES } },
  }
}));
if (!validation.success) return res.status(400).json({ ...validation });

Please make sure that you have seen the video (above) on validate an API request in Node using custom validation or by using make-validation library.

Here we are using the [make-validation](https://www.npmjs.com/package/@withvoid/make-validation) library (that I ended up making while writing this tutorial). I talk about it's usage in the video at the start of this tutorial.

All we are doing here is passing req.body to payload. Then in the checks we're adding an object where against each key we are telling what are the requirements for each type, for example:

firstName: { type: types.string },

Here we are telling it that firstName is of type string. If the user forgets to add this value while hitting the API, or if the type is not string, it will throw an error.

The validation variable will return an object with 3 things: {success: boolean, message: string, errors: object}.

If validation.success is false we simply return everything from the validation and give it to the user with a status code of 400.

Once our validation is in place and we know that the data we are getting are valid, then we do the following:

const { firstName, lastName, type } = req.body;
const user = await UserModel.createUser(firstName, lastName, type);
return res.status(200).json({ success: true, user });

Then we destruct firstName, lastName, type from req.body and pass those values to our UserModel.createUser. If everything goes right, it simply returns success: true with the new user created along with a status 200.

If anywhere in this process anything goes wrong, it throws an error and goes to the catch block:

catch (error) {
  return res.status(500).json({ success: false, error: error })
}

There we simply return an error message along with the HTTP status 500.

The only thing we are missing here is the UserModel.createUser() method.

So let's go back into our models/User.js file and add it:

userSchema.statics.createUser = async function (
    firstName, 
        lastName, 
        type
) {
  try {
    const user = await this.create({ firstName, lastName, type });
    return user;
  } catch (error) {
    throw error;
  }
}


export default mongoose.model("User", userSchema);

So all we are doing here is adding a static method to our userSchema called createUser that takes in 3 parameters: firstName, lastName, type.

Next we use this:

const user = await this.create({ firstName, lastName, type });

Here the this part is very important, since we are writing a static method on userSchema. Writing this will ensure that we are using performing operations on the userSchema object

One thing to note here is that userSchema.statics.createUser = async function (firstName, lastName, type) => {} won't work. If you use an => arrow function the this context will be lost and it won't work.

If you want to learn more about static methods in mongoose, see this very short but helpful doc example here.

Now that we have everything set up, let's start our terminal by running the following command in the project's root folder:

npm start;

Go into postman, set up a POST request on this API http://localhost:3000/users, and add the following body to the API:

{
    firstName: 'John'
        lastName: 'Doe',
        type: 'consumer'
}

Like this:

You can also get the entire postman API collection from here so that you don't have to write the APIs again and again.

Awesome – we just ended up creating our first API. Let's create a couple more user APIs before we move to the chat part because there is no chat without users (unless we have robots, but robots are users as well ?).

Get a user by its ID API [GET request]

Next we need to write an API that gets us a user by its ID. So for our route .get('/:id', user.onGetUserById) let's write down its controller.

Go to controllers/user.js and for the method onGetUserById write this:

onGetUserById: async (req, res) => {
  try {
    const user = await UserModel.getUserById(req.params.id);
    return res.status(200).json({ success: true, user });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Cool, this looks straightforward. Let's add UserModel.getUserById() in our models/User.js file.

Add this method below the last static method you wrote:

userSchema.statics.getUserById = async function (id) {
  try {
    const user = await this.findOne({ _id: id });
    if (!user) throw ({ error: 'No user with this id found' });
    return user;
  } catch (error) {
    throw error;
  }
}

We pass in an id parameter and we wrap our function in try/catch. This is very important when you are using async/await. The lines to focus on here are these 2:

const user = await this.findOne({ _id: id });
if (!user) throw ({ error: 'No user with this id found' });

We use mongoose's findOne method to find an entry by id. We know that only one item exists in the collection by this id because the id is unique. If no user is found we simply throw an error with the message No user with this id found.

And that is it! Let's start up our server:

npm start;

Open up postman and create a GET request http://localhost:3000/users/:id.

Note: I am using the ID of the last user we just created.

Nicely done! Good job.

Two more API's to go for our user section.

Get all users API [GET request]

For our router in .get('/', user.onGetAllUsers) let's add information to its controller.

Go to controllers/user.js and add code in the onGetAllUsers() method:

onGetAllUsers: async (req, res) => {
  try {
    const users = await UserModel.getUsers();
    return res.status(200).json({ success: true, users });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Next let's create the static method for getUsers() in the models/User.js file. Below the last static method you wrote in that file, type:

userSchema.statics.getUsers = async function () {
  try {
    const users = await this.find();
    return users;
  } catch (error) {
    throw error;
  }
}

We use the mongoose method called await this.find(); to get all the records for our users collection and return it.

Note: I am not handling pagination in our users API because that's not the main focus here. I'll talk about pagination once we move towards our chat APIs.

Let's start our server:

npm start;

Open up postman and create a GET request for this route http://localhost:3000/users:

I went ahead and ended up creating a couple more users. ?

Delete a user by ID API [DELETE request] (More of a bonus section, you can skip this if you want)

Let's create our final route to delete a user by their ID. For the route .delete('/:id', user.onDeleteUserById) go to its controller in controllers/user.js and write this code in the onDeleteUserById() method:

onDeleteUserById: async (req, res) => {
  try {
    const user = await UserModel.deleteByUserById(req.params.id);
    return res.status(200).json({ 
      success: true, 
      message: `Deleted a count of ${user.deletedCount} user.` 
    });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Let's add the static method deleteByUserById in models/User.js:

userSchema.statics.deleteByUserById = async function (id) {
  try {
    const result = await this.remove({ _id: id });
    return result;
  } catch (error) {
    throw error;
  }
}

We pass in the id here as a parameter and then use the mongoose method called this.remove to delete a record item from a specific collection. In this case, it's the users collection.

Let's start up our server:

npm start;

Go to postman and create a new DELETE route:

With this we'll conclude our USER API section.

Next we will cover how to authenticate routes with an authentication token. This is the last thing I want to touch on before moving on to the chat section – because all of the chat APIs will be authenticated.

What are middlewares in ExpressJS?

How can we write them? By adding JWT middleware in your application:

And here's the GitHub link to the entire source code of this video [Chapter 0].

And again, all the relevant info can be found in the READ.ME.

Coming back to our code base, let's create a JWT middleware to authenticate our routes. Go to middlewares/jwt.js and add the following:

import jwt from 'jsonwebtoken';
// models
import UserModel from '../models/User.js';

const SECRET_KEY = 'some-secret-key';

export const encode = async (req, res, next) => {
  try {
    const { userId } = req.params;
    const user = await UserModel.getUserById(userId);
    const payload = {
      userId: user._id,
      userType: user.type,
    };
    const authToken = jwt.sign(payload, SECRET_KEY);
    console.log('Auth', authToken);
    req.authToken = authToken;
    next();
  } catch (error) {
    return res.status(400).json({ success: false, message: error.error });
  }
}

export const decode = (req, res, next) => {
  if (!req.headers['authorization']) {
    return res.status(400).json({ success: false, message: 'No access token provided' });
  }
  const accessToken = req.headers.authorization.split(' ')[1];
  try {
    const decoded = jwt.verify(accessToken, SECRET_KEY);
    req.userId = decoded.userId;
    req.userType = decoded.type;
    return next();
  } catch (error) {

    return res.status(401).json({ success: false, message: error.message });
  }
}

Let's discuss the encode method first:

export const encode = async (req, res, next) => {
  try {
    const { userId } = req.params;
    const user = await UserModel.getUserById(userId);
    const payload = {
      userId: user._id,
      userType: user.type,
    };
    const authToken = jwt.sign(payload, SECRET_KEY);
    console.log('Auth', authToken);
    req.authToken = authToken;
    next();
  } catch (error) {
    return res.status(400).json({ 
        success: false, message: error.error 
    });
  }
}

Let's go through it step by step.

We get the userId from our req.params. If you remember from the video earlier, req.params is the /:<identifier> defined in our routes section.

Next we use the const user = await UserModel.getUserById(userId); method we just created recently to get user information. If it exists, that is – otherwise this line will throw an error and it will directly go to the catch block where we will return the user with a 400 response and and an error message.

But if we get a response from the getUserById method we then make a payload:

const payload = {
      userId: user._id,
      userType: user.type,
};

Next we sign that payload in JWT using the following:

const authToken = jwt.sign(payload, SECRET_KEY);

Once we have the JWT signed we then do this:

req.authToken = authToken;
next();

Set it to our req.authToken and then forward this information as next().

Next let's talk about the decode method:

export const decode = (req, res, next) => {
  if (!req.headers['authorization']) {
    return res.status(400).json({ success: false, message: 'No access token provided' });
  }
  const accessToken = req.headers.authorization.split(' ')[1];
  try {
    const decoded = jwt.verify(accessToken, SECRET_KEY);
    req.userId = decoded.userId;
    req.userType = decoded.type;
    return next();
  } catch (error) {

    return res.status(401).json({ success: false, message: error.message });
  }
}

Let's break this down:

if (!req.headers['authorization']) {
  return res.status(400).json({ 
      success: false, 
        message: 'No access token provided' 
  });
}

First we check if the authorization header is present or not. If not we simply return an error message to user.

Then we do this:

const accessToken = req.headers.authorization.split(' ')[1];

It's being split(' ') by space and then we are getting the second index of the array by accessing its [1] index because the convention is authorization: Bearer <auth-token>. Want to read more on this? Check out this nice thread on quora.

Then we try to decode our token:

try {
  const decoded = jwt.verify(accessToken, SECRET_KEY);
  req.userId = decoded.userId;
  req.userType = decoded.type;
  return next();
} catch (error) {
  return res.status(401).json({ 
      success: false, message: error.message 
  });
}

If this is not successful jwt.verify(accessToken, SECRET_KEY) will simply throw an error and our code will go in the catch block immediately. If it is successful, then we can decode it. We get userId and type from the token and save it as req.userId, req.userType and simply hit next().

Now, moving forward, every route that goes through this decode middleware will have the current user's id & it's type.

This was it for the middleware section. Let's create a login route so that we can ask a user for their information and give a token in return (because moving forward they'll need a token to access the rest of chat APIs).

Creating a login route [POST request]

Go to your routes/index.js file and paste the following content:

import express from 'express';
// middlewares
import { encode } from '../middlewares/jwt.js';

const router = express.Router();

router
  .post('/login/:userId', encode, (req, res, next) => {
    return res
      .status(200)
      .json({
        success: true,
        authorization: req.authToken,
      });
  });

export default router;

So all we are doing is adding the encode middleware to our http://localhost:3000/login/:<user-id> [POST] route. If everything goes smoothly the user will get an authorization token.

Note: I am not adding a login/signup flow, but I still wanted to touch on JWT/middleware in this tutorial.

Usually authentication is done in a similar way. The only addition here is that the user doesn't provide their ID. They provide their username, password (which we verify in the database), and if everything checks out we give them an authorization token.

If you got stuck anywhere up to this point, just write to me at twitter.com/adeelibr, so that way I can improve the content. You can also write to me if you would like to learn something else.

As a reminder, the entire source code is available here. You don't have to code along with this tutorial, but if you do the concepts will stick better.

Let's just check our /login route now.

Start your server:

npm start;

Let's run postman. Create a new POST request http://localhost:3000/login/<user-id>:

When the user ID is correct

When the user ID is invalid

With this we are done with our login flow as well.

This was a lot. But now we can focus only on our chat routes.

Create a web socket class

This web socket class will handle events when a user disconnects, joins a chat room, or wants to mute a chat room.

So let's create a web-socket class that will manage sockets for us. Create a new folder called utils. Inside that folder create a file called WebSockets.js and add the following content:

class WebSockets {
  users = [];
  connection(client) {
    // event fired when the chat room is disconnected
    client.on("disconnect", () => {
      this.users = this.users.filter((user) => user.socketId !== client.id);
    });
    // add identity of user mapped to the socket id
    client.on("identity", (userId) => {
      this.users.push({
        socketId: client.id,
        userId: userId,
      });
    });
    // subscribe person to chat & other user as well
    client.on("subscribe", (room, otherUserId = "") => {
      this.subscribeOtherUser(room, otherUserId);
      client.join(room);
    });
    // mute a chat room
    client.on("unsubscribe", (room) => {
      client.leave(room);
    });
  }

  subscribeOtherUser(room, otherUserId) {
    const userSockets = this.users.filter(
      (user) => user.userId === otherUserId
    );
    userSockets.map((userInfo) => {
      const socketConn = global.io.sockets.connected(userInfo.socketId);
      if (socketConn) {
        socketConn.join(room);
      }
    });
  }
}

export default new WebSockets();

The WebSockets class has three major things here:

• users array
• connection method
• subscribing members of a chat room to it. subscribeOtherUser

Let's break this down.

We have a class:

class WebSockets {

}

export default new WebSocket();

We create a class and export an instance of that class.

Inside the class we have an empty users array. This array will hold a list of all the active users that are online using our application.

Next we have a connection method, the core of this class:

connection(client) {
  // event fired when the chat room is disconnected
  client.on("disconnect", () => {
    this.users = this.users.filter((user) => user.socketId !== client.id);
  });
  // add identity of user mapped to the socket id
  client.on("identity", (userId) => {
    this.users.push({
      socketId: client.id,
      userId: userId,
    });
  });
  // subscribe person to chat & other user as well
  client.on("subscribe", (room, otherUserId = "") => {
    this.subscribeOtherUser(room, otherUserId);
    client.join(room);
  });
  // mute a chat room
  client.on("unsubscribe", (room) => {
    client.leave(room);
  });
}

The connection method takes in a parameter called client (client here will be our server instance, I will talk more about this in a bit).

We take the param client and add some event to it

• client.on('disconnect') // when a user connection is lost this method will be called
• client.on('identity') // when user logs in from the front end they will make a connection with our server by giving their identity
• client.on('subscribe') // when a user joins a chat room this method is called
• client.on('unsubscribe') // when a user leaves or wants to mute a chat room

Let's talk about disconnect:

client.on("disconnect", () => {
  this.users = this.users.filter((user) => user.socketId !== client.id);
});

As soon as the connection is disconnected, we run a filter on users array. Where we find user.id === client.id we remove it from our sockets array. ( client here is coming from the function param.)

Let's talk about identity:

client.on("identity", (userId) => {
  this.users.push({
    socketId: client.id,
    userId: userId,
  });
});

When a user logs in through he front end application web/android/ios they will make a socket connection with our backend app and call this identity method. They'll also send their own user id.

We will take that user id and the client id (the user's own unique socket id that socket.io creates when they make a connection with our BE).

Next we have unsubscribe:

client.on("unsubscribe", (room) => {
  client.leave(room);
});

The user passes in the room id and we just tell client.leave() to remove the current user calling this method from a particular chat room.

Next we have subscribe:

client.on("subscribe", (room, otherUserId = "") => {
  this.subscribeOtherUser(room, otherUserId);
  client.join(room);
});

When a user joins a chat room, they will tell us about the room they want to join along with the other person who is part of that chat room.

Note: We will see later that when we initiate a chat room we get all the users associated with that room in the API response.

In my opinion: Another thing we could have done here was when the user sends in the room number, we can make a DB query to see all the members of the chat room and make them join if they are online at the moment (that is, in our users list).

The subscribeOtherUser method is defined like this:

subscribeOtherUser(room, otherUserId) {
  const userSockets = this.users.filter(
    (user) => user.userId === otherUserId
  );
  userSockets.map((userInfo) => {
    const socketConn = global.io.sockets.connected(userInfo.socketId);
    if (socketConn) {
      socketConn.join(room);
    }
  });
}

We pass in room and otherUserId as params to this function.

Using the otherUserId we filter on our this.users array and all the results that match are stored in userSockets array.

You might be thinking – how can one user have multiple presences in the user array? Well, think of a scenario where the same user is logged in from both their web application and mobile phone. It will create multiple socket connections for the same user.

Next we map on userSockets. For each item in this array we pass it into this method: const socketConn = global.io.sockets.connected(userInfo.socketId)

I will talk more about this global.io.sockets.connected in a bit. But what this initially does is it takes in userInfo.socketId and if it exists in our socket connection, it will return the connection, otherwise null.

Next we simply see if socketConn is available. If so, we take that socketConn and make this connection join the room passed in the function:

if (socketConn) {
    socketConn.join(room);
}

And this is it for our WebSockets class.

Let's import this file in our server/index.js file:

import socketio from "socket.io";
// mongo connection
import "./config/mongo.js";
// socket configuration
import WebSockets from "./utils/WebSockets.js";

So just import socket.io and import WebSockets somewhere at the top.

Next where we are creating our server add the content below this:

/** Create HTTP server. */
const server = http.createServer(app);
/** Create socket connection */
global.io = socketio.listen(server);
global.io.on('connection', WebSockets.connection)

The server was created and we do two things:

• assign global.io to socketio.listen(server) (As soon as a port starts listening on the server, sockets starts listening for events happening on that port as well.)
• then we assign global.io.on('connection', WebSockets.connection) method. Every time someone from the front end makes a socket connection, the connection method will be called which will invoke our Websockets class and inside that class the connection method.

global.io is equivalent to windows object in browser. But since we don't have windows in NodeJS we use global.io. Whatever we put in global.io is available in the entire application.

This is the same global.io we used in the WebSockets class inside the subscribeOtherUser method.

If you got lost here is the entire source code of this chat application. Also free to drop me a message with your feedback and I will try to improve the content of this tutorial.

Discussing chat room & chat message database model

Before starting off with Chat, I think it is really important to discuss the database model on which we will create our chat application. Have a look at the below video:

Now that you have a clear idea about what our chat structure will be like, let's start off by making our chat room model.

Go inside your models folder and create the following ChatRoom.js. Add the following content to it:

import mongoose from "mongoose";
import { v4 as uuidv4 } from "uuid";

export const CHAT_ROOM_TYPES = {
  CONSUMER_TO_CONSUMER: "consumer-to-consumer",
  CONSUMER_TO_SUPPORT: "consumer-to-support",
};

const chatRoomSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    userIds: Array,
    type: String,
    chatInitiator: String,
  },
  {
    timestamps: true,
    collection: "chatrooms",
  }
);

chatRoomSchema.statics.initiateChat = async function (
    userIds, type, chatInitiator
) {
  try {
    const availableRoom = await this.findOne({
      userIds: {
        $size: userIds.length,
        $all: [...userIds],
      },
      type,
    });
    if (availableRoom) {
      return {
        isNew: false,
        message: 'retrieving an old chat room',
        chatRoomId: availableRoom._doc._id,
        type: availableRoom._doc.type,
      };
    }

    const newRoom = await this.create({ userIds, type, chatInitiator });
    return {
      isNew: true,
      message: 'creating a new chatroom',
      chatRoomId: newRoom._doc._id,
      type: newRoom._doc.type,
    };
  } catch (error) {
    console.log('error on start chat method', error);
    throw error;
  }
}

export default mongoose.model("ChatRoom", chatRoomSchema);

We have three things going on here:

• We have a const for CHAT_ROOM_TYPES which has only two types
• We define our ChatRoom schema
• We add a static method to initiate chat

Chat related APIs

Initiate a chat between users (/room/initiate [POST request])

Let's discuss our static method defined in models/ChatRoom.js called initiateChat:

chatRoomSchema.statics.initiateChat = async function (userIds, type, chatInitiator) {
  try {
    const availableRoom = await this.findOne({
      userIds: {
        $size: userIds.length,
        $all: [...userIds],
      },
      type,
    });
    if (availableRoom) {
      return {
        isNew: false,
        message: 'retrieving an old chat room',
        chatRoomId: availableRoom._doc._id,
        type: availableRoom._doc.type,
      };
    }

    const newRoom = await this.create({ userIds, type, chatInitiator });
    return {
      isNew: true,
      message: 'creating a new chatroom',
      chatRoomId: newRoom._doc._id,
      type: newRoom._doc.type,
    };
  } catch (error) {
    console.log('error on start chat method', error);
    throw error;
  }
}

This function takes in three parameters:

• userIds (array of users)
• type (type of chatroom)
• chatInitiator (the user who created the chat room)

Next we are doing two things here: either returning an existing chatroom document or creating a new one.

Let's break this one down:

const availableRoom = await this.findOne({
  userIds: {
    $size: userIds.length,
    $all: [...userIds],
  },
  type,
});
if (availableRoom) {
  return {
    isNew: false,
    message: 'retrieving an old chat room',
    chatRoomId: availableRoom._doc._id,
    type: availableRoom._doc.type,
  };
}

First using the this.findOne() API in mongoose, we find all the chatrooms where the following criteria is met:

userIds: { $size: userIds.length, $all: [...userIds] },
type: type,

You can read more on the $size operator here, and more on the $all operator here.

We're checking to find a chatroom document where an item exists in our chatrooms collection where

1. the userIds are the same as the one we are passing to this function (irrespective of the user ids order), and
2. the length of the userIds is the same as that my userIds.length that we are passing through the function.

Also we're checking that the chat room type should be the same.

If something like this is found, we simply return the existing chatroom.

Otherwise we create a new chat room and return it by doing this:

const newRoom = await this.create({ userIds, type, chatInitiator });
return {
  isNew: true,
  message: 'creating a new chatroom',
  chatRoomId: newRoom._doc._id,
  type: newRoom._doc.type,
};

Create a new room and return the response.

We also have an isNew key where, if it's retrieving an old chatroom, we set it to false otherwise true.

Next for your route created in routes/chatRoom.js called post('/initiate', chatRoom.initiate) go to its appropriate controller in controllers/chatRoom.js and add the following in the initiate method:

initiate: async (req, res) => {
  try {
    const validation = makeValidation(types => ({
      payload: req.body,
      checks: {
        userIds: { 
          type: types.array, 
          options: { unique: true, empty: false, stringOnly: true } 
        },
        type: { type: types.enum, options: { enum: CHAT_ROOM_TYPES } },
      }
    }));
    if (!validation.success) return res.status(400).json({ ...validation });

    const { userIds, type } = req.body;
    const { userId: chatInitiator } = req;
    const allUserIds = [...userIds, chatInitiator];
    const chatRoom = await ChatRoomModel.initiateChat(allUserIds, type, chatInitiator);
    return res.status(200).json({ success: true, chatRoom });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

We are using the [make-validation](https://www.npmjs.com/package/@withvoid/make-validation) library here to validate the user's request. For the initiate API, we expect the user to send an array of users and also define the type of the chat-room that is being created.

Once the validation passes, then:

const { userIds, type } = req.body;
const { userId: chatInitiator } = req;
const allUserIds = [...userIds, chatInitiator];
const chatRoom = await ChatRoomModel.initiateChat(allUserIds, type, chatInitiator);
return res.status(200).json({ success: true, chatRoom });

One thing to notice here is userIds, type is coming from req.body while userId that is being aliased as chatInitiatorId is coming from req thanks to our decode middleware.

If you remember, we attached app.use("/room", decode, chatRoomRouter); in our server/index.js file. This means this route /room/initiate is authenticated. So const { userId: chatInitiator } = req; is the id of the current user logged in.

We simply call our initiateChat method from ChatRoomModel and pass it allUserIds, type, chatInitiator. Whatever result comes we simply pass it to the user.

Let's run this and see if it works (here is a video of me doing it):

Create a message in chat room (/:roomId/message) [POST request]

Let's create a message for the chat room we just created with pikachu.

But before we create a message we need to create a model for our chatmessages. So let's do that first. In your models folder create a new file called ChatMessage.js and add the following content to it:

import mongoose from "mongoose";
import { v4 as uuidv4 } from "uuid";

const MESSAGE_TYPES = {
  TYPE_TEXT: "text",
};

const readByRecipientSchema = new mongoose.Schema(
  {
    _id: false,
    readByUserId: String,
    readAt: {
      type: Date,
      default: Date.now(),
    },
  },
  {
    timestamps: false,
  }
);

const chatMessageSchema = new mongoose.Schema(
  {
    _id: {
      type: String,
      default: () => uuidv4().replace(/\-/g, ""),
    },
    chatRoomId: String,
    message: mongoose.Schema.Types.Mixed,
    type: {
      type: String,
      default: () => MESSAGE_TYPES.TYPE_TEXT,
    },
    postedByUser: String,
    readByRecipients: [readByRecipientSchema],
  },
  {
    timestamps: true,
    collection: "chatmessages",
  }
);

chatMessageSchema.statics.createPostInChatRoom = async function (chatRoomId, message, postedByUser) {
  try {
    const post = await this.create({
      chatRoomId,
      message,
      postedByUser,
      readByRecipients: { readByUserId: postedByUser }
    });
    const aggregate = await this.aggregate([
      // get post where _id = post._id
      { $match: { _id: post._id } },
      // do a join on another table called users, and 
      // get me a user whose _id = postedByUser
      {
        $lookup: {
          from: 'users',
          localField: 'postedByUser',
          foreignField: '_id',
          as: 'postedByUser',
        }
      },
      { $unwind: '$postedByUser' },
      // do a join on another table called chatrooms, and 
      // get me a chatroom whose _id = chatRoomId
      {
        $lookup: {
          from: 'chatrooms',
          localField: 'chatRoomId',
          foreignField: '_id',
          as: 'chatRoomInfo',
        }
      },
      { $unwind: '$chatRoomInfo' },
      { $unwind: '$chatRoomInfo.userIds' },
      // do a join on another table called users, and 
      // get me a user whose _id = userIds
      {
        $lookup: {
          from: 'users',
          localField: 'chatRoomInfo.userIds',
          foreignField: '_id',
          as: 'chatRoomInfo.userProfile',
        }
      },
      { $unwind: '$chatRoomInfo.userProfile' },
      // group data
      {
        $group: {
          _id: '$chatRoomInfo._id',
          postId: { $last: '$_id' },
          chatRoomId: { $last: '$chatRoomInfo._id' },
          message: { $last: '$message' },
          type: { $last: '$type' },
          postedByUser: { $last: '$postedByUser' },
          readByRecipients: { $last: '$readByRecipients' },
          chatRoomInfo: { $addToSet: '$chatRoomInfo.userProfile' },
          createdAt: { $last: '$createdAt' },
          updatedAt: { $last: '$updatedAt' },
        }
      }
    ]);
    return aggregate[0];
  } catch (error) {
    throw error;
  }
}

export default mongoose.model("ChatMessage", chatMessageSchema);

There are a couple of things happening here:

• We have a MESSAGE_TYPES object which has only one type called text
• We are defining our schema for chatmessage and readByRecipient
• Then we are writing our static method for createPostInChatRoom

I know this is a lot of content, but just bear with me. Let's just write the controller for the route that creates this message.

For the route defined in our routes/chatRoom.js API called .post('/:roomId/message', chatRoom.postMessage) let's go to its controller in controllers/chatRoom.js and define it:

postMessage: async (req, res) => {
  try {
    const { roomId } = req.params;
    const validation = makeValidation(types => ({
      payload: req.body,
      checks: {
        messageText: { type: types.string },
      }
    }));
    if (!validation.success) return res.status(400).json({ ...validation });

    const messagePayload = {
      messageText: req.body.messageText,
    };
    const currentLoggedUser = req.userId;
    const post = await ChatMessageModel.createPostInChatRoom(roomId, messagePayload, currentLoggedUser);
    global.io.sockets.in(roomId).emit('new message', { message: post });
    return res.status(200).json({ success: true, post });
  } catch (error) {
    return res.status(500).json({ success: false, error: error })
  }
},

Cool, let's discuss what we are doing here:

Operators discussed in this video are:

• $match
• $last
• $addToSet
• $lookup
• $unwind
• $group

See conversation for a chat room by it's id [Get request]

Now that we have

• Created a chat room
• Are able to add messages in that chat room

Let's see the entire conversation for that chat as well (with pagination).

For your route .get('/:roomId', chatRoom.getConversationByRoomId) in routes/chatRoom.js open its controller in the file controllers/chatRoom.js and add the following content to the chat room:

getConversationByRoomId: async (req, res) => {
  try {
    const { roomId } = req.params;
    const room = await ChatRoomModel.getChatRoomByRoomId(roomId)
    if (!room) {
      return res.status(400).json({
        success: false,
        message: 'No room exists for this id',
      })
    }
    const users = await UserModel.getUserByIds(room.userIds);
    const options = {
      page: parseInt(req.query.page) || 0,
      limit: parseInt(req.query.limit) || 10,
    };
    const conversation = await ChatMessageModel.getConversationByRoomId(roomId, options);
    return res.status(200).json({
      success: true,
      conversation,
      users,
    });
  } catch (error) {
    return res.status(500).json({ success: false, error });
  }
},

Next let's create a new static method in our ChatRoomModel file called getChatRoomByRoomId in models/ChatRoom.js:

chatRoomSchema.statics.getChatRoomByRoomId = async function (roomId) {
  try {
    const room = await this.findOne({ _id: roomId });
    return room;
  } catch (error) {
    throw error;
  }
}

Very straightforward – we are getting the room by roomId here.

Next in our UserModel, create a static method called getUserByIds in the file models/User.js:

userSchema.statics.getUserByIds = async function (ids) {
  try {
    const users = await this.find({ _id: { $in: ids } });
    return users;
  } catch (error) {
    throw error;
  }
}

The operator used here is $in – I'll talk about this in a bit.

And then at last, go to your ChatMessage model in models/ChatMessage.js and write a new static method called getConversationByRoomId:

chatMessageSchema.statics.getConversationByRoomId = async function (chatRoomId, options = {}) {
  try {
    return this.aggregate([
      { $match: { chatRoomId } },
      { $sort: { createdAt: -1 } },
      // do a join on another table called users, and 
      // get me a user whose _id = postedByUser
      {
        $lookup: {
          from: 'users',
          localField: 'postedByUser',
          foreignField: '_id',
          as: 'postedByUser',
        }
      },
      { $unwind: "$postedByUser" },
      // apply pagination
      { $skip: options.page * options.limit },
      { $limit: options.limit },
      { $sort: { createdAt: 1 } },
    ]);
  } catch (error) {
    throw error;
  }
}

Let's discuss all that we have done so far:

All the source code is available here.

Mark an entire conversation as read (feature similar to WhatsApp)

Once the other person is logged in and they view a conversation for a room id, we need to mark that conversation as read from their side.

To do this, in your routes/chatRoom.js for the route

put('/:roomId/mark-read', chatRoom.markConversationReadByRoomId)

go to its appropriate controller in controllers/chatRoom.js and add the following content in the markConversationReadByRoomId controller.

markConversationReadByRoomId: async (req, res) => {
  try {
    const { roomId } = req.params;
    const room = await ChatRoomModel.getChatRoomByRoomId(roomId)
    if (!room) {
      return res.status(400).json({
        success: false,
        message: 'No room exists for this id',
      })
    }

    const currentLoggedUser = req.userId;
    const result = await ChatMessageModel.markMessageRead(roomId, currentLoggedUser);
    return res.status(200).json({ success: true, data: result });
  } catch (error) {
    console.log(error);
    return res.status(500).json({ success: false, error });
  }
},

All we are doing here is first checking if the room exists or not. If it does, we proceed further. We take in the req.user.id as currentLoggedUser and pass it to the following function:

ChatMessageModel.markMessageRead(roomId, currentLoggedUser);

Which in our ChatMessage model is defined like this:

chatMessageSchema.statics.markMessageRead = async function (chatRoomId, currentUserOnlineId) {
  try {
    return this.updateMany(
      {
        chatRoomId,
        'readByRecipients.readByUserId': { $ne: currentUserOnlineId }
      },
      {
        $addToSet: {
          readByRecipients: { readByUserId: currentUserOnlineId }
        }
      },
      {
        multi: true
      }
    );
  } catch (error) {
    throw error;
  }
}

A possible use case is that the user might not have read the last 15 messages once they open up a specific room conversation. They should all be marked as read. So we're using the this.updateMany function by mongoose.

The query itself is defined in 2 steps:

• Find
• Update

And there can be multiple statements be updated.

To find a section, do this:

{
  chatRoomId,
  'readByRecipients.readByUserId': { $ne: currentUserOnlineId }
},

This says I want to find all the message posts in the chatmessages collection where chatRoomId matches and readByRecipients array does not. The userId that I am passing to this function is currentUserOnlineId.

Once it has all those documents where the criteria matches, it's then time to update them:

{
  $addToSet: {
    readByRecipients: { readByUserId: currentUserOnlineId }
  }
},

$addToSet will just push a new entry to the readByRecipients array. This is like Array.push but for mongo.

Next we want to tell mongoose to not just update the first record it finds, but also to update all the records where the condition matches. So doing this:

{
  multi: true
}

And that is all – we return the data as is.

Let's run this API.

Start up the server:

npm start;

Open your postman and create a new PUT request to test this route ocalhost:3000/room/<room=id-here>/mark-read:

Bonus Section

• How to delete a chat room and all its related messages
• How to delete a message by its message id

And we are done! Wow that was a lot of learning today.

You can find the source code of this tutorial here.

Reach out to me on twitter with your feedback – I would love to hear if you have any suggestions for improvements: twitter.com/adeelibr

If you liked to this article, please do give the github repository a star and subscribe to my youtube channel.

The other day, some fine developers at my company were getting ready to roll out a status update page. We'd tested it extensively but now we were about to put it out at scale.

I was worried about its dependency on an API server that had been acting up recently. We haven't determined the root cause of our problems on the API side, and this application uses polling - that is, it constantly asks the API for new data. If that API goes down, it takes our app with it and the increased load from our app might exacerbate the problems we're seeing.

One way to step away from polling is to integrate SignalR, a persistent connection tool that uses websockets and related technologies to allow servers to push updates to clients.

The technology is written in .NET, and most of the documentation you will find around the web is using C#. This tutorial will cover a basic JavaScript implementation.

What does it do?

Open-sourced SignalR creates a persistent connection between a client and server. It uses websockets first, then longpolling and other technologies when websockets are unavailable.

Once the client and server have created a connection, SignalR can be used to "broadcast" messages to the client. When the client receives those messages, it can perform functions like updating a store.

The most common example given for websockets is a chat app - new data must be shown to the user without her needing to refresh the page. But if your server gets any updates about changing data that you need to show to a client, this might be the service for you.

SignalR on the Azure platform

Perhaps because it was developed by Microsoft, SignalR has a very clean integration on the Azure cloud platform. Like other function apps, you'll create an "in" trigger and an "out" binding for broadcasting messages.

Costs

Because I was the first to look at this technology at scale at my company, I had to dig in a little about costs for this service. Azure charges about $50/month for one "unit" of SignalR service - 1000 simultaneous connections and one million messages a day. There is also a free service for those playing around or small businesses.

It was really good I dug into those numbers, as you'll see a little below.

Create a SignalR hub

Let's get started. We'll need a SignalR hub, two function apps, and client code to add to our web app.

Go to SignalR -> Add and fill out your details. It takes a second for the worker to build your service. Make sure you give the service a decent resource name, as you'll be using it with the rest of your apps. Also grab Keys -> Connection String for use in our binding.

Create your function app for sending SignalR messages

Because we're working with Azure, we're going to be creating function apps to interface with SignalR. I wrote a getting-started blog post about Azure function apps a little while ago.

This tutorial assumes you already know how to work with function apps. Naturally you can work with these libraries without the binding magic, but you'll have to do your own translation of the .NET code!

The connection app

The first thing we need is a way for clients to request permission to connect to our SignalR service. The code for this function couldn't be more basic:

module.exports = function (context, _req, connectionInfo) {
  context.res = { body: connectionInfo }
  context.done()
}

The magic all happens in the bindings, where we pull in our SignalR service. The trigger is an HTTP request that our client can call.

{
  "bindings": [
      {
          "authLevel": "function",
          "type": "httpTrigger",
          "direction": "in",
          "name": "req",
          "methods": ["get"]
      },
      {
          "type": "signalRConnectionInfo",
          "name": "connectionInfo",
          "hubName": "your-signalr-service-name",
          "connectionStringSetting": "connection-string",
          "direction": "in"
      }
  ]
}

The client code

To access this method, our client will call:

import * as signalR from '@microsoft/signalr'

const { url: connectionUrl, accessToken } = await axios
  .get(url-to-your-connection-app)
  .then(({ data }) => data)
  .catch(console.error)

Our function app will return a url and accessToken, which we can then use to connect to our SignalR service. Note that we created the binding with the hubName of our SignalR service - that means you could have multiple connections to different hubs in one client.

The broadcasting service

Now we are ready to start sending messages. Again we'll start with the function app. It takes in a trigger and puts out a SignalR message.

A trigger could be another using posting a message, an event from an event hub, or any other trigger Azure supports. I need to trigger off database changes.

{
  "bindings": [
      {
          "type": "cosmosDBTrigger",
          "name": "documents",
          "direction": "in",
          [...]
      },
      {
        "type": "signalR",
        "name": "signalRMessages",
        "hubName": "your-signalr-service-name",
        "connectionStringSetting": "connection-string",
        "direction": "out"
      }
  ]
}

And the code. Again, dead simple.

module.exports = async function (context, documents) {
  const messages = documents.map(update => {
    return {
      target: 'statusUpdates',
      arguments: [update]
    }
  })
  context.bindings.signalRMessages = messages
}

SignalR messages take a target and arguments object. Once your triggers start firing, that's everything you need to get started with SignalR on the server! Microsoft has made all of this very easy for us.

The client code

On the client side, things are a little more complex, but not unmanageable. Here's the rest of the client code:

const connection = new signalR.HubConnectionBuilder()
  .withUrl(connectionUrl, { accessTokenFactory: () => accessToken })
  // .configureLogging(signalR.LogLevel.Trace)
  .withAutomaticReconnect()
  .build()

connection.on('statusUpdates', data => {
  // do something with the data you get from SignalR
})
connection.onclose(function() {
  console.log('signalr disconnected')
})
connection.onreconnecting(err =>
  console.log('err reconnecting  ', err)
)

connection
  .start()
  .then(res => // Potential to do something on initial load)
  .catch(console.error)

We consume the connectionUrl and accessToken we received from the connect function earlier, then build our connection using those values.

Then we listen to messages with the shared key (for me, it's statusUpdates), and provide handlers for close and reconnecting functions.

Finally, we start the connection. Here we can provide an initial load function. I needed one to fetch initial data to show current status. If you are building a chat app, you might need to fetch initial messages here.

This is (almost, maybe) everything you need to get started in JavaScript with SignalR on Azure!

Scoping by user

But maybe you, like me, need to send a lot of messages to a lot of users.

When I first put this into production, on a sub-set of users, I was blasting every connection with every single update. Because the client code can scope the messages it listens to, I used something like statusUpdates-${userId} so that the client would only see his own updates.

That could work just fine if you have very low volume, and the more general one is great if everybody in your system needs the same message. But the status I work with is particular to an individual.

Remember how Azure charges per "unit" and each unit has one million messages? I hit that during a few hours of testing this during a not-busy time.

Azure counts each message SignalR has to send as one message. That is, if five connections are hooked up to your hub and you send ten messages, that counts as 50, not 10. This was a surprise to me, and also required a couple more hours of research.

We can scope our SignalR function code to send only to certain users. First, we update the connection app to accept userId as a query param:

      {
          "type": "signalRConnectionInfo",
          "name": "connectionInfo",
          "userId": "{userId}",
          "hubName": "your-signalr-service-name",
          "connectionStringSetting": "connection-string",
          "direction": "in"
      }

Then we update the broadcasting function to send only to that user:

const messages = documents.map(update => {
  return {
    target: 'statusUpdates',
    userId: update.user.id,
    arguments: [update]
  }
})

The broadcasting service won't know who has connected, so you'll need to trigger it with something that has access to a unique ID that the client will also have access to.

The client code simply passes in the userId as a query param:

const { url: connectionUrl, accessToken } = await axios
  .get(`${url-to-your-connection-app}&userId=${userId}`)
  .then(({ data }) => data)
  .catch(console.error)

I swear to you, the only place on the entire internet I found to let me know how to request a connection using the userId was an answer on this Stack Overflow question.

The internet is amazing, and JavaScript Azure docs are hard to come by.

Resources

• SignalR Javascript client docs from Microsoft
• Configuring Users and Groups when sending SignalR messages - examples in C# but you can maybe figure out how the JavaScript client is going to behave and make some educated guesses.
• SignalR Service bindings for Azure Functions
• Client API
• Working with Groups in SignalR
• Tutorial: Azure SignalR Service authentication with Azure Functions

This post originally appeared on wilkie.tech.

Recently AWS has announced the launch of a widely-requested feature: WebSockets for Amazon API Gateway. With WebSockets, we are able to create a two-way communication line which can be used in many scenarios like real-time applications. This brings up the question: what are real-time applications? So let’s first answer that question.

Most of the applications that are currently operational use client-server architecture. In client-server architecture, the client sends the requests over the Internet using network communication and then the server processes that request and sends the response back to the client.

Here you can see that it is the client that starts off the communication with the server. So first the client initiates the communication and the server responds to the request sent by the server. So what if the server wants to start off the communication and push responses without the client requesting them first? That is where real-time applications come into the play.

Real-time applications are applications where the server gets the ability to push to the clients without the client requesting data first. Assume we have a chat application where two chat clients can communicate via a server. In this situation, it is a waste if all the chat clients request data from the server like every second. What is more efficient is if the server sends data to client chat applications when a chat is received. This functionality can be done through real-time applications.

Amazon announced that they are going to support WebSockets in API Gateway at AWS re:Invent 2018. Later in December, they launched it in the API Gateway. So now using AWS infrastructure we are able to create real-time applications using API Gateway.

In this post, we are going to create a simple chat application using API Gateway WebSockets. Before we start implementing our chat application, there are some concepts that we need to understand regarding real-time applications and API Gateway.

WebSocket API Concepts

A WebSocket API is composed of one or more routes. A route selection expression is there to determine which route a particular inbound request should use, which will be provided in the inbound request. The expression is evaluated against an inbound request to produce a value that corresponds to one of your route’s routeKey values. For example, if our JSON messages contain a property call action, and you want to perform different actions based on this property, your route selection expression might be ${request.body.action}.

For example: if your JSON message looks like {“action” : “onMessage” , “message” : “Hello everyone”}, then the onMessage route will be chosen for this request.

By default, there are three routes which are already defined in the WebSocket API. In addition to the below-mentioned routes, we can add custom routes for our needs.

• $default — Used when the route selection expression produces a value that does not match any of the other route keys in your API routes. This can be used, for example, to implement a generic error handling mechanism.
• $connect — The associated route is used when a client first connects to your WebSocket API.
• $disconnect — The associated route is used when a client disconnects from your API.

Once a device is successfully connected through WebSocket API, the device will be allocated with a unique connection id. This connection id will be persisted throughout the lifetime if the connection. To send messages back to the device we need to use the following POST request using the connection id.

POST https://{api-id}.execute-api.us-east 1.amazonaws.com/{stage}/@connections/{connection_id}

Implementing chat application

After learning the basic concepts of the WebSocket API, let us look at how we can create a real-time application using WebSocket API. In this post, we are going to implement a simple chat application using WebSocket API, AWS LAmbda and DynamoDB. The following diagram shows the architecture of our real-time application.

In our application, devices will be connected to the API Gateway. When a device gets connected, a lambda function will save the connection id in a DynamoDB table. In an instance where we want to send a message back to the device, another lambda function will retrieve the connection id and POST data back to the device using a callback URL.

Creating WebSocket API

In order to create the WebSocket API, we need first go to Amazon API Gateway service using the console. In there choose to create new API. Click on WebSocket to create a WebSocket API, give an API name and our Route Selection Expression. In our case add $request.body.action as our selection expression and hit Create API.

After creating the API we will be redirected to the routes page. Here we can see already predefined three routes: $connect, $disconnect and $default. We will also create a custom route $onMessage. In our architecture, $connect and $disconnect routes achieve the following tasks:

• $connect — when this route is called, a Lambda function will add the connection id of the connected device to DynamoDB.
• $disconnect — when this route is called, a Lambda function will delete the connection id of the disconnected device from DynamoDB.
• onMessage — when this route is called, the message body will be sent to all the devices that are connected at the time.

Before adding the route according to the above, we need to do four tasks:

• Create a DynamoDB table
• Create connect lambda function
• Create disconnect lambda function
• Create onMessage lambda function

First, let us create the DynamoDB table. Go to DynamoDB service and create a new table called Chat. Add primary key as ‘connectionid’.

Next, let’s create the connect Lambda function. To create the Lambda function, go to Lambda services and click create function. Select Author from scratch and give the name as ‘ChatRoomConnectFunction’ and a role with necessary permissions. (The role should have the permission to get, put and delete items from DynamoDB, call API calls in API gateway.)

In the code of the lambda function add the following code. This code will add the connection id of the connected device to the DynamoDB table that we have created.

exports.handler = (event, context, callback) => {    const connectionId = event.requestContext.connectionId;    addConnectionId(connectionId).then(() => {    callback(null, {        statusCode: 200,        })    });}

function addConnectionId(connectionId) {    return ddb.put({        TableName: 'Chat',        Item: {            connectionid : connectionId        },    }).promise();}

Next, let us create the disconnect lambda function as well. Using the same steps create a new lambda function named
‘ChatRoomDonnectFunction’. Add the following code to the function. This code will remove the connection id from the DynamoDB table when a device gets disconnected.

const AWS = require('aws-sdk');const ddb = new AWS.DynamoDB.DocumentClient();

exports.handler = (event, context, callback) => {    const connectionId = event.requestContext.connectionId;    addConnectionId(connectionId).then(() => {    callback(null, {        statusCode: 200,        })    });}

function addConnectionId(connectionId) {    return ddb.delete({        TableName: 'Chat',        Key: {            connectionid : connectionId,        },    }).promise();}

Now we have created the DynamoDB table and two lambda functions. Before creating the third lambda function, let us go back again to API Gateway and configure the routes using our created lambda functions. First, click on $connect route. As integration type, select Lambda function and select the ChatRoomConnectionFunction.

We can do the same on $disconnect route as well where the lambda function will be ChatRoomDisconnectionFunction:

Now that we have configured our $connect and $disconnect routes, we can actually test whether out WebSocket API is working. To do that we must first to deploy the API. In the Actions button, click on Deploy API to deploy. Give a stage name such as Test since we are only deploying the API for testing.

After deploying, we will be presented with two URLs. The first URL is called WebSocket URL and the second is called Connection URL.

The WebSocket URL is the URL that is used to connect through WebSockets to our API by devices. And the second URL, which is Connection URL, is the URL which we will use to call back to the devices which are connected. Since we have not yet configured call back to devices, let’s first only test the $connect and $disconnect routes.

To call through WebSockets we can use the wscat tool. To install it, we need to just issue the npm install -g wscat command in the command line. After installing, we can use the tool using wscat command. To connect to our WebSocket API, issue the following command. Make sure to replace the WebSocket URL with the correct URL provided to you.

wscat -c wss://bh5a9s7j1e.execute-api.us-east-1.amazonaws.com/Test

When the connection is successful, a connected message will be displayed on the terminal. To check whether our lambda function is working, we can go to DynamoDB and look in the table for the connection id of the connected terminal.

As above, we can test the disconnect as well by pressing CTRL + C which will simulate a disconnection.

Now that we have tested our two routes, let us look into the custom route onMessage. What this custom route will do is it will get a message from the device and send the message to all the devices that are connected to the WebSocket API. To achieve this we are going to need another lambda function which will query our DynamoDB table, get all the connection ids, and send the message to them.

Let’s first create the lambda function in the same way we created other two lambda functions. Name the lambda function ChatRoomOnMessageFunction and copy the following code to the function code.

const AWS = require('aws-sdk');const ddb = new AWS.DynamoDB.DocumentClient();require('./patch.js');

let send = undefined;function init(event) {  console.log(event)    const apigwManagementApi = new AWS.ApiGatewayManagementApi({    apiVersion: '2018-11-29',    endpoint: event.requestContext.domainName + '/' + event.requestContext.stage  });        send = async (connectionId, data) => {  await apigwManagementApi.postToConnection({ ConnectionId: connectionId, Data: `Echo: ${data}` }).promise();  }}

exports.handler =  (event, context, callback) => {  init(event);  let message = JSON.parse(event.body).message    getConnections().then((data) => {        console.log(data.Items);        data.Items.forEach(function(connection) {           console.log("Connection " +connection.connectionid)           send(connection.connectionid, message);        });    });        return {}};

function getConnections(){    return ddb.scan({        TableName: 'Chat',    }).promise();}

The above code will scan the DynamoDB to get all the available records in the table. For each record, it will POST a message using the Connection URL provided to us in the API. In the code, we expect that the devices will send the message in the attribute named ‘message’ which the lambda function will parse and send to others.

Since WebSockets API is still new there are some things we need to do manually. Create a new file named patch.js and add the following code inside it.

require('aws-sdk/lib/node_loader');var AWS = require('aws-sdk/lib/core');var Service = AWS.Service;var apiLoader = AWS.apiLoader;

apiLoader.services['apigatewaymanagementapi'] = {};AWS.ApiGatewayManagementApi = Service.defineService('apigatewaymanagementapi', ['2018-11-29']);Object.defineProperty(apiLoader.services['apigatewaymanagementapi'], '2018-11-29', {  get: function get() {    var model = {      "metadata": {        "apiVersion": "2018-11-29",        "endpointPrefix": "execute-api",        "signingName": "execute-api",        "serviceFullName": "AmazonApiGatewayManagementApi",        "serviceId": "ApiGatewayManagementApi",        "protocol": "rest-json",        "jsonVersion": "1.1",        "uid": "apigatewaymanagementapi-2018-11-29",        "signatureVersion": "v4"      },      "operations": {        "PostToConnection": {          "http": {            "requestUri": "/@connections/{connectionId}",            "responseCode": 200          },          "input": {            "type": "structure",            "members": {              "Data": {                "type": "blob"              },              "ConnectionId": {                "location": "uri",                "locationName": "connectionId"              }            },            "required": [              "ConnectionId",              "Data"            ],            "payload": "Data"          }        }      },      "shapes": {}    }    model.paginators = {      "pagination": {}    }    return model;  },  enumerable: true,  configurable: true});

module.exports = AWS.ApiGatewayManagementApi;

I took the above code from this article. The functionality of this code is to automatically create the Callback URL for our API and send the POST request.

Now that we have created the lambda function we can go ahead and create our custom route in API Gateway. In the New Route Key, add ‘OnMessage’ as a route and add the custom route. As configurations were done for other routes, add our lambda function to this custom route and deploy the API.

Now we have completed our WebSocket API and we can fully test the application. To test that sending messages works for multiple devices, we can open and connect using multiple terminals.

After connecting, issue the following JSON to send messages:

{"action" : "onMessage" , "message" : "Hello everyone"}

Here, the action is the custom route we defined and the message is the data that need to be sent to other devices.

That is it for our simple chat application using AWS WebSocket API. We have not actually configured the $defalut route which is called on every occasion where there no route is found. I will leave the implementation of that route to you. Thank you and see you in another post. :)

The Web is growing at a massive rate. More and more web apps are dynamic, immersive and do not require the end user to refresh. There is emerging support for low latency communication technologies like websockets. Websockets allow us to achieve real-time communication among different clients connected to a server.

A lot of people are unaware of how to secure their websockets against some very common attacks. Let us see what they are and what should you do to protect your websockets.

#0: Enable CORS

WebSocket doesn’t come with CORS inbuilt. That being said, it means that any website can connect to any other website’s websocket connection and communicate without any restriction! I’m not going into reasons why this is the way it is, but a quick fix to this is to verify Origin header on the websocket handshake.

Sure, Origin header can be faked by an attacker, but it doesn’t matter, because to exploit it, attacker needs to fake the Origin header on victim’s browser, and modern browsers do not allow normal javascript sitting in web browsers to change Origin header.

Moreover, if you’re actually authenticating users using, preferably, cookies, then this is not really a problem for you (more on this on point #4)

#1: Implement rate limiting

Rate limiting is important. Without it, clients can knowingly or unknowingly perform a DoS attack on your server. DoS stands for Denial of Service. DoS means a single client is keeping the server so busy that the server is unable to handle other clients.

In most of the cases it is a deliberate attempt by an attacker to bring down a server. Sometimes poor frontend implementations can also lead to DoS by normal clients.

We’re gonna make use of the leaky bucket algorithm (which apparently is a very common algorithm for networks to implement) for implementing rate limiting on our websockets.

The idea is that you have a bucket which has a fixed size hole at its floor. You start putting water in it and the water goes out through the hole at the bottom. Now, if your rate of putting water into the bucket is larger than the rate of flowing out of the hole for a long time, at some point, the bucket will become full and start leaking. That’s all.

Let’s now understand how it relates to our websocket:

1. Water is the websocket traffic sent by the user.
2. Water passes down the hole. This means the server successfully processed that particular websocket request.
3. Water which is still in the bucket and has not overflowed is basically pending traffic. The server will process this traffic later on. This could also be bursty traffic flow (i.e. too much traffic for a very small time is okay as long as bucket doesn’t leak)
4. Water which is overflowing is the traffic discarded by the server (too much traffic coming from a single user)

The point here is, you have to check your websocket activity and determine these numbers. You’re going to assign a bucket to every user. We decide how big the bucket should be (traffic which a single user could send over a fixed period) depending on how large your hole is (how much time on average does your server need to process a single websocket request, say saving a message sent by a user into a database).

This is a trimmed down implementation I’m using at codedamn for implementing leaky bucket algorithm for the websockets. It is in NodeJS but the concept remains same.

if(this.limitCounter >= Socket.limit) {
  if(this.burstCounter >= Socket.burst) {
     return 'Bucket is leaking'
  }
  ++this.burstCounter
  return setTimeout(() => {
  this.verify(callingMethod, ...args)
  setTimeout(_ => --this.burstCounter, Socket.burstTime)
  }, Socket.burstDelay)
}
++this.limitCounter

So what’s happening here? Basically, if the limit is crossed as well as the burst limit (which are constants set), the websocket connection drops. Otherwise, after a particular delay, we’re gonna reset the burst counter. This leaves space again for another burst.

#2: Restrict payload size

This should be implemented as a feature within your server-side websocket library. If not, its time to change it to a better one! You should limit the maximum length of the message that could be sent over your websocket. Theoretically there is no limit. Of course, getting a huge payload is very likely to hang that particular socket instance and eat up more system resources than required.

For example, if you’re using WS library for Node for creating websockets on server, you can use the maxPayload option to specify the maximum payload size in bytes. If the payload size is bigger than that, the library will natively drop the connection.

Do not try to implement this on your own by determining message length. We don’t want to read the whole message into the system RAM first. If it is even 1 byte greater than our set limit, drop it. That could be only implemented by the library (which handles messages as a stream of bytes rather than fixed strings).

#3: Create a solid communication protocol

Because now you’re on a duplex connection, you could be sending anything to the server. The server could send any text back to client. You would need to have a way for effective communication between both.

You can’t send raw messages if you want to scale the messaging aspect of your website. I prefer using JSON, but there are other optimized ways to set up a communication. However, considering JSON, here’s what a basic messaging schema would look like for a generic site:

Client to Server (or vice versa): { status: "ok"|"error", event: EVENT_NAME, data: <any arbitrary data> }

Now it’s easier for you on the server to check for valid events and format. Drop the connection immediately and log the IP address of the user if the message format differs. There’s no way the format would change unless someone is manually tingling with your websocket connection. If you’re on node, I recommend using the Joi library for further validation of incoming data from user.

#4: Authenticate users before WS connection establishes

If you’re using websockets for authenticated users, it is a pretty good idea to only allow authenticated users to establish a successful websocket connection. Don’t allow anyone to establish a connection and then wait for them to authenticate over the websocket itself. First of all, establishing a websocket connection is a bit expensive anyway. So you don’t want unauthorized people hopping on your websockets and hogging connections which could be used by other people.

To do this, when you’re establishing a connection on frontend, pass some authentication data to websocket. It could be a header like X-Auth-Token: . By default, cookies would be passed anyway.

Again, it really comes down to the library you’re using on the server for implementing websockets. But if you’re on Node and using WS, there’s this verifyClient function which gets you access to info object passed to a websocket connection. (Just like you have access to req object for HTTP requests.)

#5: Use SSL over websockets

This is a no-brainer, but still needs to be said. Use wss:// instead of ws://. This adds a security layer over your communication. Use a server like Nginx for reverse proxying websockets and enable SSL over them. Setting up Nginx would be a whole another tutorial. I’ll leave the directive you need to use for Nginx for those folks who are familiar with it. More info here.

location /your-websocket-location/ {
    proxy_pass ​http://127.0.0.1:1337;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
}

Here it is assumed your websocket server is listening on port 1337 and your users connect to your websocket in this fashion:

const ws = new WebSocket('wss://yoursite.com/your-websocket-location')

Questions?

Have any questions or suggestions? Ask away!