In this case, there is no database to store the information. Rather, you just have a free Discord text channel that keeps the data as a sequence of messages in chat. And an admin/moderator/user with the required access rights can read through these messages and take the appropriate action.

In case you are new to Discord, it’s a great platform for chatting, playing video games, having calls, and even running a virtual team for your startup. It’s free, and you can download it from here.

Alright, coming back to the topic, I had a requirement like this to implement using Discord’s webhook and Next.js. I learned a lot from the activity. So I wrote this step-by-step tutorial about that process.

The primary objective here is to understand:

• What is a Webhook and what are the use cases?

• How to Integrate a webhook with a web application framework like Next.js

• How to build a user interface with the latest APIs and components from Next.js

I hope you’ll like it. If you want to learn from a video, the article is also available as a video tutorial:

Table of Contents

1. What is a Webhook?

2. How to Configure a Webhook on Discord

3. How to Create a Next.js Project

4. How to Set Environment Variables

5. How to Integrate the Webhook with the Next.js Form Component

   • Create the Form

   • Create the Server Action

6. How to Update the App Page

7. Let’s Test it Out

8. How to Improve the Message feedback using React 19’s useActionState Hook

   • Set up a toaster

   • Enhance the Form with the useActionState Hook

   • How to Update the Server Action to Return Results

9. Let’s Test it One More Time

10. Source Code and Resources

What is a Webhook?

Assume we have an email service that connects to a third-party email sender to send emails to users. Traditionally, there will be an API gateway to talk to the email service from the client devices. When the request is received, the email service will contact the email-sending provider.

While the email-sending provider processes the emails, the email service needs to know their status. One generic way to handle this is for the email service to poll for the status at a short frequency and update the client as and when the status changes. But this method has many drawbacks, such as resource mis-utilization, unnecessary connections, and poor performance.

On the contrary, how about the email service registers a callback URL that the email sender can call with relevant information when the email-sending action has been completed? This way, there is no unnecessary polling. Rather, the email sender can proactively inform the email service of a success or a failure after attempting to send the email. Then, the email service can respond to the clients regarding their status accordingly.

This registered callback is called a Webhook. Webhooks are widely used today in various industries like payments and checkout, system health monitoring, and third-party app integrations.

How to Configure a Webhook on Discord

You should now have an idea of how webhooks work, so let’s configure one with Discord.

Create or select a text channel on your Discord server. Click on the Edit Channel button.

Then, from the left-side navigation menu, select the Integration option. You should see the Webhooks option listed there.

Click on the Webhooks option and then click on the New Webhook button.

Give your webhook a name, and you can optionally upload a photo. This can be helpful when you have multiple webhooks and you want to quickly identify them by their photo and name.

Now, click on the Copy Webhook URL button to copy the webhook URL. Keep it safe somewhere, as we will be using it shortly in our application.

That’s it. Now let’s create a Next.js 15 application so we can integrate the webhook with it.

How to Create a Next.js Project

Open the terminal and use the following command to create a Next.js application:

npx create-next-app@latest

You’ll have to provide a few inputs for the initial project to create it. I will be using JavaScript (as opposed to TypeScript), Tailwind CSS, App Router, and Turbopack for this project. So I’ve opted for those choices by providing Yes as the response.

With that, you’ve created a Next.js project you can use for the rest of the tutorial.

How to Set Environment Variables

Create a .env file at the root of your project. We will now create an environment variable secret with the webhook URL you had copied before. Create an entry in the .env file like this:

DISCORD_WEBHOOK_URL=<YOUR_DISCORD_WEBHOOK_URL>

Make sure you replace the <YOUR_DISCORD_WEBHOOK_URL> with your actual webhook URL in the .env file. Remember, you must not commit and push this file to your version control. So, make sure that the .env file has been added to the .gitignore file of the project.

How to Integrate the Webhook with the Next.js Form component

Now, we will create the user interface to capture inputs from the user and send those to the Discord text channel using the webhook.

Let’s create a simple message form using the <Form /> component from Next.js. The <Form/> component is an extension of HTML’s native form with more flexibilities and features introduced by the release of Next.js 15. I would suggest that you go over this project-based video tutorial if you are interested in learning more about this new addition to Next.js.

Our strategy here is very straightforward:

• We will create a form and add an action to it using the form’s action prop.

• We will then create a Server Action using the webhook URL to communicate with Discord.

• The action will be invoked when the user fills up the form and submits it. Thus the webhook communication will be done.

Let’s write the code for these functionalities now.

Create the Form

Create a folder called _components under the app/ directory. Now, create a file message-form.jsx under the app/_components/ folder with the following code:

import Form from "next/form";
import { sendDiscordMessage } from "../actions";

const MessageForm = () => {
  return (
    <Form className="flex flex-col items-center" action={sendDiscordMessage}>
      <input
        type="text"
        placeholder="Your name"
        name="username"
        className="border p-1 rounded w-[300px] my-2"
        required
      />

      <input
        type="email"
        placeholder="Your e-mail"
        name="email"
        className="border p-1 rounded w-[300px] my-2"
        required
      />

      <input
        type="text"
        placeholder="Your Image Url"
        name="dp"
        className="border p-1 rounded w-[300px] my-2"
      />

      <select
        name="type"
        className="p-1 rounded border my-2 w-[300px]"
        required
      >
        <option value="">Message Type</option>
        <option value="thanks">Say, Thanks!</option>
        <option value="qa">QA</option>
        <option value="general">General</option>
      </select>

      <textarea
        placeholder="What do you want to say?"
        name="message"
        className="border p-1 rounded w-[300px] my-2"
        required
      />

      <button
        type="submit"
        className="bg-blue-500 w-[70px] text-white rounded-md"
      >
        Send
      </button>
    </Form>
  );
};

export default MessageForm;

Here, we have created a basic form with five fields for users to enter their name, email, profile image URL, message type (general, thanks, qa), and the message. There is a send button as well to submit the form.

We have used the Form component from the next/form package. The Form component allows us to execute a function on the server (server action) with the help of its action attribute. As you can see in the code above, we have imported a server action sendDiscordMessage and used that as the value for the action attribute of the form.

 <Form className="flex flex-col items-center" action={sendDiscordMessage}>

Now, let’s create the server action so that we can submit the form with it.

Create the Server Action

We’ll use a server action to send messages to Discord using webhooks. Create a folder called actions under the app/ directory. It’s a convention to keep all the actions colocated under the actions/ directory. Now, create a file called index.js under the app/actions/ directory with the following code:

"use server";

export const sendDiscordMessage = async (formData) => {
  try {
    const rawFormEntries = Object.fromEntries(formData);

    console.log(rawFormEntries);

    await fetch(process.env.DISCORD_WEBHOOK_URL, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        username: rawFormEntries?.username,
        avatar_url: rawFormEntries?.dp || "https://i.imgur.com/mDKlggm.png",
        content: rawFormEntries?.message,
        embeds: [
          {
            fields: [
              {
                name: "Email",
                value: rawFormEntries?.email,
                inline: true,
              },
              {
                name: "Message Type",
                value: rawFormEntries?.type,
                inline: true,
              },
            ],
          },
        ],
      }),
    })
  } catch (err) {
    console.log(err.message);
  }
};

Let’s understand what’s going on in the above code:

• In Next.js (or with React 19), a server function (aka server action) needs a special directive called ”use server” at the top of the file. So we have declared that.

• A server action is an async function that gets the formData as a parameter. The formData will contain the values of all the form fields submitted by the user. We can use the Object.formEntries() API to get a key-value pair from the formData. Check this out to learn the best way to handle formData in JavaScript.

• Next, we used the Discord webhook URL to make a POST call with the required payload to create the message.

• Let’s understand the payload format well. We need to follow a specific payload structure for the discord webhook to create the message. It follows a schema to have the following:

  • username: the name of the message sender. It appears at the top of the message. We’re reading the name field from the submitted data and populating the username field.

  • avatar_url: the profile photo of the message sender. We have a dp field in the form to capture the profile photo URL. We’re using that, and in case the user does not provide it, we’re using a default image as the profile photo.

  • content: The content field is the actual message content. We are reading the value of the message field and populating the value.

  • embeds: In the Discord message, you can use embeds. These embeds could be text messages, images, or videos. We will utilize the embeds to show the email and message type information. The embeds is an array of fields. In each of the fields, we have passed the email and message type values as the inline values. Inline values will appear in line, side by side.

How to Update the App Page

Finally, let’s update the application’s page with our form component so that everything gets stitched together. Open the page.js file under the app/ directory and paste in the following code:

import MessageForm from "./_components/message-form";

export default function Home() {
    return (
        <div className="flex flex-col justify-center items-center">
          <h1 className="text-3xl my-3">Send a message to tapaScript</h1>
          <MessageForm />
        </div>
      ) 
}

Here we have integrated the form with the app page so that we can access it on the browser.

Let’s Test it Out

Now you can run the app with the command yarn dev from your terminal. The app will be available on localhost:3000 by default. Open a browser tab and hit the URL http://localhost:3000. You should see a form like the one below. Fill it out with values and hit the Send button.

Go to the text channel of your Discord server and check for a message that should appear there momentarily. Check that the fields of the message match the input values you provided in the form.

TADA! We have done it. But hold on – we can make it even better. Did you notice that we aren’t showing any kind of feedback saying the “Message has been sent successfully”, or “There are issues in sending the message”? We’re not taking care of providing the result of the server action to the form component. Let’s fix that.

How to Improve the Message Feedback using React 19’s useActionState Hook

React 19 introduced a hook called useActionState that helps you update the state based on the result of a server action. Let’s use this hook to enhance the message form and the server action so that when the action gets executed successfully (or fails), we can notify the form component to change its state and show the success/error messages accordingly.

Set up a toaster

We will use a toast message to show the success/error messages. Let’s use the toast library called sonner. First, install it with this command:

yarn add sonner

Now, we have to add the Toaster from sonner to our app. The best place to add it is at the root level Layout of the application. Open the layout.js file and import the Toaster:

import { Toaster } from 'sonner';

Then, use the Toaster in the JSX of the layout:

 <html lang="en">
      <body
        className={`${geistSans.variable} ${geistMono.variable} antialiased`}
      >
        {children}
        <Toaster richColors />
      </body>
 </html>

That’s it. The toaster set-up is done. Now we will be able to use it in our components to show the toast messages.

Enhance the Form with the useActionState Hook

First, import the useActionState and useEffect hooks into the message-form.jsx file. Also, import the toast from sonner. As we will be using the hooks now, we have to make the message-form component a client component. So, add the ”use client” directive at the top of the file.

"use client";

import Form from "next/form";
import { useActionState } from "react";
import { sendDiscordMessage } from "../actions";
import { useEffect } from "react"
import { toast } from "sonner";

Next, use the useActionState hook to get the formState updated from the server action result. An isPending state tells us if the form submission has been completed. The useActionState returns a new formAction from the existing server action instance.

The syntax goes like this:

const [formState, formAction, isPending] = useActionState(sendDiscordMessage,null);

Now, we need to use this formAction as the value of the action attribute of the <Form /> component:

<Form className="flex flex-col items-center" action={formAction}>

Next, use the useEffect hook to track the formState changes and show the toast message accordingly. The shape of the formState can be customized based on our needs. We will see shortly how the server action can return this state value.

  useEffect (() => {
    if (formState?.success) {
      toast.success(formState?.message);
    } else if (formState?.success === false) {
      toast.error(formState?.message);
    }
  },[formState?.success])

The last thing here is to improve the UX of the form using the isPending state we got from the useActionState hook. The value of the isPending state will be true if the form is still in transition and being submitted. It will be changed to false when the form is submitted. So we can use the state value to customize the submit button text.

<button
  type="submit"
  className="bg-blue-500 w-[70px] text-white rounded-md">
    {isPending ? "Sending..." : "Send"}
</button>

Those are all the changes we need to make in the form component. Here’s the complete code of the modified form component for you to check out and use:

"use client";

import Form from "next/form";
import { useActionState } from "react";
import { sendDiscordMessage } from "../actions";
import { useEffect } from "react"
import { toast } from "sonner";

const MessageForm = () => {
  const [formState, formAction, isPending] = useActionState(
    sendDiscordMessage,
    null
  );

  useEffect (() => {
    if (formState?.success) {
      toast.success(formState?.message);
    } else if (formState?.success === false) {
      toast.error(formState?.message);
    }
  },[formState?.success])



  return (
    <Form className="flex flex-col items-center" action={formAction}>
      <input
        type="text"
        placeholder="Your name"
        name="username"
        className="border p-1 rounded w-[300px] my-2"
        required
      />

      <input
        type="email"
        placeholder="Your e-mail"
        name="email"
        className="border p-1 rounded w-[300px] my-2"
        required
      />

      <input
        type="text"
        placeholder="Your Image Url"
        name="dp"
        className="border p-1 rounded w-[300px] my-2"
      />

      <select
        name="type"
        className="p-1 rounded border my-2 w-[300px]"
        required
      >
        <option value="">Message Type</option>
        <option value="thanks">Say, Thanks!</option>
        <option value="qa">QA</option>
        <option value="general">General</option>
      </select>

      <textarea
        placeholder="What do you want to say?"
        name="message"
        className="border p-1 rounded w-[300px] my-2"
        required
      />

      <button
        type="submit"
        className="bg-blue-500 w-[70px] text-white rounded-md"
      >
        {isPending ? "Sending..." : "Send"}
      </button>
    </Form>
  );
};

export default MessageForm;

How to Update the Server Action to Return Results

The changes in the server action should be as follows:

1. When you use the useActionState hook, pass a returned formAction to the form. We have seen this before. It helps in multiple ways, and one of them is getting the previous state value of the formState inside the server action. You must be mindful to pass it as the first argument to the server function, and then the formData.

2. We can now return the result of the server action so that the formState gets updated with the result. We will create a result structure with a boolean success field indicating if it is a success or error scenario and a message field with the actual message.

Here’s the changed code with the above two updates:

"use server";

export const sendDiscordMessage = async (prevState, formData) => {
  try {
    const rawFormEntries = Object.fromEntries(formData);

    console.log(rawFormEntries);

    await fetch(process.env.DISCORD_WEBHOOK_URL, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        username: rawFormEntries?.username,
        avatar_url: rawFormEntries?.dp || "https://i.imgur.com/mDKlggm.png",
        content: rawFormEntries?.message,
        embeds: [
          {
            fields: [
              {
                name: "Email",
                value: rawFormEntries?.email,
                inline: true,
              },
              {
                name: "Message Type",
                value: rawFormEntries?.type,
                inline: true,
              },
            ],
          },
        ],
      }),
    });

    return {
      success: true,
      message: `Your message has been sent successfully.`,
    };
  } catch (err) {
    console.log(err.message);
    return {
      success: false,
      message: `Problem is sending message ${err.message}`,
    };
  }
};

As you must have noticed, the returned result is the formState we have used inside the form component to show the toast messages.

Let’s Test it One More Time

Let’s test things out with all the changes.

1. First, fill out the form with the details. Now we’re using a profile picture URL rather than keeping it empty. Click on the Send button.

2. You should see the that button text gets changed to Sending… while the form is being submitted.

3. After the submission, you should get the success message in the toast. Also, the form state should be restored to its original state.

4. On the Discord text channel, you should see the message posted successfully.

That’s amazing! We have now made the app much better with the message and error handling.

Source Code and Resources

All the source code used in this article is in the GitHub repository. You can take a look, follow the README to set it up, and run it locally.

• The next-js-discord GitHub repository

Here are the resources I mentioned in the article that you may find helpful:

• Next.js 15 Form Component - All That You Need To Know

• Next.js Server Actions || Learn Patterns & Project Building

• MASTER React 19 useActionState Hook with Project & Usecases

• The useActionState official doc

Also, you can connect with me by:

• Subscribing to my YouTube Channel. If you are willing to learn React and its ecosystem, like Next.js, with both fundamental concepts and projects, I have great news for you: you can check out this playlist on my YouTube channel with 30+ video tutorials and 20+ hours of engaging content so far, for free. I hope you like them as well.

• Following me on X (Twitter) or LinkedIn if you don't want to miss the daily dose of up-skilling tips.

• Checking out and following my Open Source work on GitHub.

• I regularly publish meaningful posts on my GreenRoots Blog, you may find them helpful, too.

See you soon with my next article. Until then, please take care of yourself, and keep learning.

The python-telegram-bot v20 release introduced major structural changes. According to the documentation,

“All networking and I/O-related logic now works via coroutine functions (i.e. _async def ..._ functions). In particular, all methods of the _telegram.Bot_ class that make requests to the Bot API are now coroutine functions and need to be await-ed.” — python-telegram-bot v20 change logs

The process of transitioning from python-telegram-bot version 13 to version 20 turned out to be more complex than I had originally anticipated. Converting synchronous def functions to async def and adding await to new coroutines was fairly easy. But the main difficulty was finding detailed documentation on how to build and deploy python-telegram-bot v20 webhooks in a production setting.

This article explains why and how I migrated from:

1. python-telegram-bot v13 → v20
2. Flask → FastAPI
3. Gunicorn → Gunicorn + Uvicorn

Why Upgrade from v13 to v20?

v13.x and older are no longer supported by the python-telegram-bot dev team. If Telegram API introduces any new features, they will only be available on v20 and above.

Why Use a Webhook and Not Polling?

Most examples provided by the python-telegram-bot dev team use Application.run_polling. But webhooks are generally recommended over polling for most Telegram bot use cases, because polling requires your bot to constantly make requests to Telegram’s servers, which can consume significant resources. On the other hand, webhooks offer extended functionality, update faster, and scale better.

The Challenge of using Flask with python-telegram-bot v20

Using a WGSI like Flask with python-telegram-bot v20 is awkward.

Flask, a WSGI (Web Server Gateway Interface), is synchronous and can handle only one request at a time. But you can still run async functions in Flask using asyncio.run(), as in the custom webhook bot example provided by the python-telegram-bot dev team.

asyncio.run() starts an event loop and executes the given coroutine until it completes. If there are any asynchronous tasks running before or after the request is handled, those tasks will be executed in a separate event loop.

# Code snippet from https://docs.python-telegram-bot.org/en/v20.6/examples.customwebhookbot.html

webserver = uvicorn.Server(
    config=uvicorn.Config(
        app=WsgiToAsgi(flask_app),
        port=PORT,
        use_colors=False,
        host="127.0.0.1",
    )
)

async with application:
  await application.start()
  await webserver.serve() # start bot's webserver
  await application.stop()

However, this implementation is slightly awkward because Flask is intrinsically incompatible with async globals.

Examples in the documentation are not suitable for production environments.

Using asyncio.run() as an entry point in production is usually not recommended. The asyncio.run() function is designed for development and testing purposes, and it may not provide the same level of robustness and reliability as production servers like Gunicorn or UWSGI.

These production servers offer many additional features, such as logging, monitoring, and health checks, which are essential for ensuring the stability and security of a production application.

If you want to deploy your bot in production, it’s much cleaner to use an ASGI (Asynchronous Server Gateway Interface) with an ASGI web server implementation.

How to Do It — Migration and Deployment

From Flask (WSGI) to FastAPI (AGSI)

Migrating a Flask application to an ASGI is straightforward. I chose FastAPI because I found a comprehensive migration tutorial here. The syntaxes of both frameworks are quite similar, which means you won’t have to make too many code changes.

# From python-telegram-bot v20
application = (
    Application.builder()
    .updater(None)
    .token(<your-bot-token>) # replace <your-bot-token>
    .read_timeout(7)
    .get_updates_read_timeout(42)
    .build()
)

# From FastAPI
@asynccontextmanager
async def lifespan(app: FastAPI):
    async with application:
        await application.start()
        yield
        await application.stop()

Quart appears to be a feasible alternative, but it does not offer support for deployments using Uvicorn, which is the webserver implementation I was adapting from the script provided by the python-telegram-bot team.

A working example

The following code shows a minimal example that uses FastAPI to build a python-telegram-bot v20 webhook. This bot will respond with “starting…” when it receives the /start command.

# main.py

from contextlib import asynccontextmanager
from http import HTTPStatus
from telegram import Update
from telegram.ext import Application, CommandHandler
from telegram.ext._contexttypes import ContextTypes
from fastapi import FastAPI, Request, Response

# Initialize python telegram bot
ptb = (
    Application.builder()
    .updater(None)
    .token(<your-bot-token>) # replace <your-bot-token>
    .read_timeout(7)
    .get_updates_read_timeout(42)
    .build()
)

@asynccontextmanager
async def lifespan(_: FastAPI):
    await ptb.bot.setWebhook(<your-webhook-url>) # replace <your-webhook-url>
    async with ptb:
        await ptb.start()
        yield
        await ptb.stop()

# Initialize FastAPI app (similar to Flask)
app = FastAPI(lifespan=lifespan)

@app.post("/")
async def process_update(request: Request):
    req = await request.json()
    update = Update.de_json(req, ptb.bot)
    await ptb.process_update(update)
    return Response(status_code=HTTPStatus.OK)

# Example handler
async def start(update, _: ContextTypes.DEFAULT_TYPE):
    """Send a message when the command /start is issued."""
    await update.message.reply_text("starting...")

ptb.add_handler(CommandHandler("start", start))

To start the bot, pip install all required dependencies and run the start command: gunicorn main:app -k uvicorn.workers.UvicornWorker.

This code snippet is adapted from a real Telegram bot in production. Check out the source code for @cron_telebot here to see how it is implemented. Feel free to adapt the script to suit your use case.

Conclusion

In this article, we learnt how to build and deploy a python-telegram-bot v20 webhook.

Hope this tutorial helped you. If you enjoyed this article, please follow me on Medium to show your support.

Thank you for reading!

The interface is what the software presents to other humans or programs, allowing them to interact with it.

A good analogy for an interface is a remote control. Imagine you have a universal remote that can control your TV, lights, and fan.

Image showing a remote control and a TV, light fixture, and fan.

Let’s break down what a universal remote control can do:

1. The remote control has various buttons, each serving a different purpose. One button might change the channel, while another can dim the lights of the chandelier, and another can turn on the fan.

2. When you press a button, it sends a specific signal via infrared, bluetooth, or wifi to the object you are controlling, instructing it to perform a particular action.

3. The key thing about the remote is that it allows you to interact with the TV, chandelier, and the fan without understanding the internal workings of these objects. All that complexity is abstracted away from you. You simply press a button, and you get a response that you can observe straight away.

APIs work in a similar way.

1. APIs can have various endpoints, each designed to perform a specific action. One endpoint might retrieve data, while another updates or deletes it.

2. When you send a request to an endpoint, it communicates with the server using HTTP methods – GET, POST, PUT, DELETE to instruct it to perform a particular action (like retrieving, sending, updating, or deleting data).

3. The key thing about APIs, as with remote controls, is that APIs abstract away the inner workings of the server and the database behind the API. The API allows users, developers and applications to interact with a software application or platform without needing to understand its internal code or database structure. You simply send a request, the server processes it and provides a response.

This analogy only holds true for so long, as APIs are more complex than a remote control. But the basic principles of operation between an API and a universal remote are quite similar.

This article will explain API integration patterns, which can be split into two broad groups: Request-response (REST, RPC & GraphQL) and event driven APIs (Polling, WebSockets & WebHooks).

Request-Response Integration

In a request-response integration, the client initiates the action by sending a request to the server and then waits for a response.

Different patterns of the request-response integration exist, but at a high level, they all conform to the same rule of the client initiating a request and waiting for a response from the server.

1. REST

Rest stands for Representational State Transfer – the acronym is a combination of the first one or two letters from these three words. This is the simplest and most popular form of a request-response integration.

REST APIs use a stateless, client-server communication model, wherein each message contains all the information necessary to understand and process the message.

REST is all about resources. Resources are entities that the API exposes, which can be accessed and manipulated using URL paths.

To understand REST APIs, consider the following analogy. Imagine you go into a restaurant to order some food. The menu is extensive and items are categorically organised. Each item on the menu can be equated to a resource.

First, you call the waiter to get their attention, then you place an order. Each request receives a response, before you proceed with another request, like ordering a dish.

Restaurant analogy for REST API

In REST API terms, the client initiates requests to the server by specifying exactly what it wants using HTTP methods (such as GET, POST, PUT, DELETE) on specific URLs (the menu items). Each interaction is stateless, meaning that each request from the client to the server must contain all the information needed to understand and process the request.

The server then processes the request and returns the appropriate response – in our analogy, bringing the ordered item to the table.

Simple sequence diagram for REST API

2. RPC

RPC stands for Remote Procedure Call. Unlike REST APIs which are all about resources, RPC is all about actions. With RPC, the client executes a block of code on the server

Think of a restaurant without a menu. There is no dish you can request in this restaurant. Instead, you request a specific action to be performed by the restaurant.

Restaurant analogy for RPC

With a REST API, the guest would have simply asked for some fish and chips. With RPC, they have to give instructions on what they want the kitchen to prepare.

In the RPC pattern, the client calls a specific procedure on the server and waits for the result. The procedure to prepare and what gets prepared are tightly bound together. This might give the client very specific and tailored results, but lacks the flexibility and ease of use of REST.

There is a reason most restaurants use menus, instead of following the custom requests of their customers. This partly explains why RPC is a less popular integration pattern compared to REST.

3. GraphQL

With GraphQL, the client specifies exactly what data it needs, which can include specific fields from various resources. The server processes this query, retrieves the exact data, and returns it to the client.

This enables the client to have a high degree of flexibility and only retrieve exactly the data it needs. It also requires the server to be capable of handling more complex and unique queries.

In this way, GraphQL is a more customisable form of REST. You still deal with resources (unlike actions in RPC) but you can customise how you want the resource returned to you.

Think of a restaurant that allows you to customise your own dish by specifying exact quantities or ingredients you want.

Restaurant analogy for GraphQL

This may look similar to the RPC pattern, but notice that the customer is not saying how the food should be made, they're just customising their order by removing some ingredients (no salt) and reducing the number of some items (two pieces of fish instead of four).

One of the drawbacks of GraphQL is that it adds complexity to the API since the server needs to do additional processing to parse complex queries. This additional complexity would also apply to the restaurant analogy, since each order would need to be customised to the guest.

GraphQL has one clear benefit over REST and RPC. Since clients can specify exactly what they need, the response payload sizes are typically smaller, which means faster response times.

Event Driven Integration

This integration pattern is ideal for services with fast changing data.

Some of these integration patterns are also asynchronous and initiated by the server, unlike the request-response patterns which are synchronous and initiated by the client.

1. Polling

Let’s bring back the restaurant analogy. When you order food, it will take some time for it to be prepared.

You can get updates on your order by asking the waiter if it is ready yet. The more frequently you ask, the closer you will be to having real-time information about your order.

This, however, puts unnecessary strain on the waiter since they have to constantly check the status of your order and have them update you whenever you ask.

Restaurant analogy for polling

Polling is when the client continuously asks the server if there is new data available, with a set frequency. It's not efficient because many requests may return no new data, thus unnecessarily consuming resources.

The more frequently you poll (make requests) the closer the client gets to real-time communication with the server.

Simple sequence diagram showing polling in action

Most of the requests during polling are wasted, since they only return something useful to the client once there is a change on the server.

There is, however, another version of polling called long polling. With long polling, the waiter does not respond to the guest straightaway about the status of the order. Instead, the waiter only responds if there is an update.

Naturally, this only works if the guest and the waiter agree beforehand that a slow response from the waiter does not mean that the waiter is being rude and the guest is being ignored.

Restaurant analogy for long polling

With long polling, the server does not respond to the client immediately. It waits until something has changed before responding.

As long as the client and server agree that the server will hold on to the client’s request, and the connection between the client and server remains open, this pattern works and can be more efficient than simply polling.

These two assumptions for long polling may be unrealistic, though – the server can lose the client's request and/or the connection can be broken.

To address these limitations, long polling adds extra complexity to the process by requiring a directory of which server contains the connection to the client, which is used to send data to the client whenever the server is ready.

Standard polling on the other hand can remain stateless, making it more fault tolerant and scalable.

2. WebSockets

WebSockets provide a persistent, two-way communication channel between the client and server. Once a WebSocket connection is established, both parties can communicate freely, which enables real-time data flows and is more resource-efficient than polling.

Using the restaurant analogy again, a guest orders a meal and then establishes a dedicated communication channel with the waiter so they can freely communicate back and forth about updates or changes to the order until the meal is ready. This means the waiter can also initiate the communication with the guest, which is not the case for the other integration patterns mentioned so far.

Restaurant analogy for WebSockets

WebSockets are similar to long polling. They both avoid the wasteful requests of polling, but WebSockets have the added benefit of having a persistent connection between the client and the server.

WebSockets are ideal for fast, live streaming data, like real-time chat applications. The downside of WebSockets is that the persistent connection consumes bandwidth, so may not be ideal for mobile applications or in areas with poor connectivity

3. WebHooks

WebHooks allow the server to notify the client when there's new data available. The client registers a callback URL with the server and the server sends a message to that URL when there is data to send.

With WebHooks, the client sends requests as usual, but can also listen for and receive requests like a server.

Simple sequence diagram showing WebHooks in action

Using the restaurant analogy, when the guest orders a meal, they give the waiter a bell (analogous to the callback URL). The waiter goes to the kitchen and rings the bell as soon as the meal is ready. This allows the client to know, in real-time, about the progress of his order.

WebHooks are superior to polling because you get real-time updates from the server once something changes, without having to make frequent, wasteful requests to the server about that change.

They're also superior to long polling because long polling can consume more client and server resources as it involves keeping connections open, potentially resulting in many open connections.

Bringing it Together

In conclusion, APIs are crucial tools in software development, allowing users and applications to interact with software without understanding its inner workings.

They come in different integration patterns, such as REST, RPC, GraphQL, Polling, WebSockets, and WebHooks.

If you need a simple request-response integration, then REST, RPC or GraphQL could be ideal. For real-time or near-real-time applications, polling, WebScokets, or WebHooks are ideal.

As with any design problem, the right choice depends on the business case and what tradeoffs you are willing to tolerate.

We've released a webhooks course on the freeCodeCamp YouTube channel that will take you from being a complete beginner to being able to use webhooks in your own apps.

We've released a lot of courses on the freeCodeCamp.org YouTube channel and I can honestly say that this is one of the most well-made courses on the channel.

Craig Dennis teaches this course. He is a Developer Educator at Twilio and has created many popular courses.

You will learn all about webhooks through expert instruction, fun animations, and hands-on practice. You will learn how to use webhooks with no code and with low code. Craig prepared extensive notes for you to use while going through this course.

Application events animation from the course.

Here are the sections of this course:

Unit 1 - Integration

• Welcome
• Defining Events, Handlers, and Hooks
• Lightbulb moment
• Finding Inspiration

Unit 2 - Capturing Data from a Webhook

• Diving into Webhooks
• Explore the Request
• Using the Data
• Developing Locally
• Opening a Tunnel
• Serverless

Unit 3 - Hooking it altogether

• Introducing the projects
• Text Affirmation
• Setting up the flow
• Handle things locally
• Deploying your serverless function
• That’s a Wrap

Webhooks animation from the course.

By the end of this course, you will have learned how to create a project using webhooks and you will understand how to use webhooks in your own projects.

You can watch the complete course on the freeCodeCamp.org YouTube channel (2.5 hour watch).

Webhooks can be used by an external system for notifying your system about a certain event or update. Probably the most well known type is the one where a Payment Service Provider (PSP) informs your system about status updates of payments.

Often they come in the form where you listen on a predefined URL. For example http://example.com/webhooks/payment-update). Meanwhile the other system sends a POST request with a certain payload to that URL (for example a payment ID). As soon as the request comes in, you fetch the payment ID, ask the PSP for the latest status via their API, and update your database afterward.

Other examples can be found in this excellent explanation about Webhooks. https://sendgrid.com/blog/whats-webhook/.

Testing these webhooks goes fairly smoothly as long as the system is publicly accessible over the internet. This might be your production environment or a publicly accessible staging environment. It becomes harder when you are developing locally on your laptop or inside a Virtual Machine (VM, for example, a Vagrant box). In those cases, the local URL’s are not publicly accessible by the party sending the webhook. Also, monitoring the requests being sent around is be difficult, which might make development and debugging hard.

What will this example solve:

• Testing webhooks from a local development environment, which is not accessible over the internet. It cannot be accessed by the service sending the data to the webhook from their servers.
• Monitor the requests and data being sent around, but also the response your application generates. This will allow easier debugging, and therefore a shorter development cycle.

Prerequisites:

• Optional: in case you are developing using a Virtual Machine (VM), make sure it’s running and make sure the next steps are done in the VM.
• For this tutorial, we assume you have a vhost defined at webhook.example.vagrant. I used a Vagrant VM for this tutorial, but you are free in choosing the name of your vhost.
• Install ngrokby following the installation instructions. Inside a VM, I find the Node version of it also useful: https://www.npmjs.com/package/ngrok, but feel free to use other methods.

I assume you don’t have SSL running in your environment, but if you do, feel free to replace port 80 with port 433 and _http://_ with _https://_ in the examples below.

Make the webhook testable

Let’s assume the following example code. I’ll be using PHP, but read it as pseudo-code as I left some crucial parts out (for example API keys, input validation, etc.)

The first file: payment.php. This file creates a payment object and then registers it with the PSP. It then fetches the URL the customer needs to visit in order to pay and redirects the user to the customer in there.

Note that the webhook.example.vagrant in this example is the local vhost we’ve defined for our development set-up. It’s not accessible from the outside world.

<?php
/*
 * This file creates a payment and tells the PSP what webhook URL to use for updates
 * After creating the payment, we get a URL to send the customer to in order to pay at the PSP
 */
$payment = [
    'order_id' => 123,
    'amount' => 25.00,
    'description' => 'Test payment',
    'redirect_url' => 'http://webhook.example.vagrant/redirect.php',
    'webhook_url' => 'http://webhook.example.vagrant/webhook.php',
];

$payment = $paymentProvider->createPayment($payment);
header("Location: " . $payment->getPaymentUrl());

Second file: webhook.php. This file waits to be called by the PSP to get notified about updates.

<?php
/*
 * This file gets called by the PSP and in the $_POST they submit an 'id'
 * We can use this ID to get the latest status from the PSP and update our internal systems afterward
 */

$paymentId = $_POST['id'];
$paymentInfo = $paymentProvider->getPayment($paymentId);
$status = $paymentInfo->getStatus();

// Perform actions in here to update your system
if ($status === 'paid') {
    ..
}
elseif ($status === 'cancelled') {
    ..
}

Our webhook URL is not accessible over the internet (remember: webhook.example.vagrant). Thus, the file webhook.php will never be called by the PSP. Your system will never get to know about the payment status. This ultimately leads to orders never being shipped to customers.

Luckily, ngrok can in solving this problem. ngrok describes itself as:

ngrok exposes local servers behind NATs and firewalls to the public internet over secure tunnels.

Let’s start a basic tunnel for our project. On your environment (either on your system or on the VM) run the following command:

ngrok http -host-header=rewrite webhook.example.vagrant:80

Read about more configuration options in their documentation: https://ngrok.com/docs.

A screen like this will come up:

ngrok output

What did we just start? Basically, we instructed ngrok to start a tunnel to [http://webhook.example.vagrant](http://webhook.example.vagrnat/) at port 80. This same URL can now be reached via [http://39741ffc.ngrok.io](http://39741ffc.ngrok.io/) or [https://39741ffc.ngrok.io](http://39741ffc.ngrok.io/), They are publicly accessible over the internet by anyone that knows this URL.

Note that you get both HTTP and HTTPS available out of the box. The documentation gives examples of how to restrict this to HTTPS only: https://ngrok.com/docs#bind-tls.

So, how do we make our webhook work now? Update payment.php to the following code:

<?php
/*
 * This file creates a payment and tells the PSP what webhook URL to use for updates
 * After creating the payment, we get a URL to send the customer to in order to pay at the PSP
 */
$payment = [
    'order_id' => 123,
    'amount' => 25.00,
    'description' => 'Test payment',
    'redirect_url' => 'http://webhook.example.vagrant/redirect.php',
    'webhook_url' => 'https://39741ffc.ngrok.io/webhook.php',
];

$payment = $paymentProvider->createPayment($payment);
header("Location: " . $payment->getPaymentUrl());

Now, we told the PSP to call the tunnel URL over HTTPS. ngrok will make sure your internal URL get’s called with an unmodified payload, as soon as the PSP calls the webhook via the tunnel.

How to monitor calls to the webhook?

The screenshot you’ve seen above gives an overview of the calls being made to the tunnel host. This data is rather limited. Fortunately, ngrok offers a very nice dashboard, which allows you to inspect all calls:

I won’t go into this very deep because it’s self-explanatory as soon as you have it running. Therefore I will explain how to access it on the Vagrant box as it doesn’t work out of the box.

The dashboard will allow you to see all the calls, their status codes, the headers and data being sent around. You will also see the response your application generated.

Another neat feature of the dashboard is that it allows you to replay a certain call. Let’s say your webhook code ran into a fatal error, it would be tedious to start a new payment and wait for the webhook to be called. Replaying the previous call makes your development process way faster.

The dashboard by default is accessible at http://localhost:4040.

Dashboard in a VM

In order to make this work inside a VM, you have to perform some additional steps:

First, make sure the VM can be accessed on port 4040. Then, create a file inside the VM holding this configuration:

web_addr: **0.0.0.0:4040**

Now, kill the ngrok process that’s still running and start it with this slightly adjusted command:

ngrok http -config=/path/to/config/ngrok.conf -host-header=rewrite webhook.example.vagrant:80

You will get a screen looking similar to the previous screenshot though the ID’s have changed. The previous URL doesn’t work anymore, but you got a new URL. Also, the Web Interface URL got changed:

Now direct your browser to [http://webhook.example.vagrant:4040](http://webhook.example.vagrant:4040) to access the dashboard. Also, make a call to [https://e65642b5.ngrok.io/webhook.php](https://e65642b5.ngrok.io/webhook.php.).This will probably result in an error in your browser, but the dashboard should show the request being made.

Final remarks

The examples above are pseudo-code. The reason is that every external system uses webhooks in a different way. I tried to give an example based on a fictive PSP implementation, as probably many developers have to deal with payments at some moment.

Please be aware that your webhook URL can also be used by others with bad intentions. Make sure to validate any input being sent to it.

Preferably also add a token to the URL which is unique for each payment. This token must only be known by your system and the system sending the webhook.

Good luck testing and debugging your webhooks!

Note: I haven’t tested this tutorial on Docker. However, this Docker container looks like a good starting point and includes clear instructions. https://github.com/wernight/docker-ngrok.

Stefan Doorn

https://github.com/stefandoorn
https://twitter.com/stefan_doorn
https://www.linkedin.com/in/stefandoorn