An API fails in production. You restart the service, errors go away, and it feels resolved. Until it happens again. And again. What's happening here is simple: you're treating symptoms, not the underlying cause.

The 5 Whys technique is a straightforward way to deal with this. It comes from the Toyota Production System and was designed to help teams dig deeper into problems instead of settling for quick fixes.

The idea is simple. Ask "why" repeatedly until you reach the real cause.

But in practice, this is where things go wrong.

Teams often:

• Stop too early

• Assume answers without checking data

• Focus on people instead of systems

• Treat "five" as a rule instead of a guideline

So even though the process looks structured, the outcome is still shallow.

In this article, we'll focus on how to actually use the 5 Whys in real situations. Not just the theory, but what it looks like when you apply it to an engineering problem.

Here's What We'll Cover:

• What is the 5 Whys Technique?

• Origins of the 5 Whys Method

• How to Conduct an Effective 5 Whys Analysis

• Real-World Example: Applying 5 Whys in an Engineering Scenario

• When to Use (and When Not to Use) 5 Whys

• Benefits of the 5 Whys Technique

• Common Pitfalls and Limitations

• Tips for Using 5 Whys Effectively

• Summary

What is the 5 Whys Technique?

The 5 Whys technique is a way to break down a problem by repeatedly asking why it happened, with the goal of reaching a cause that actually explains the issue and can be addressed.

At its core, it's not about the number five. The name can be misleading. What matters is the process of following a chain of cause and effect until the explanation stops being superficial and starts becoming useful.

Each answer you uncover should move you one level deeper. You start with what went wrong, then explore what led to it, and continue until you reach something that is both believable and actionable. In most real situations, that final answer is not a single event but a gap in a system, a missing check, or an assumption that was never validated.

The technique became widely known through the Toyota Production System, where it was used to improve processes by focusing on causes rather than quick fixes.

That context is important because it highlights the original intent. The goal was not just to explain problems, but to prevent them from happening again.

A simple example makes this clearer. Imagine a mobile app suddenly starts crashing after a release. Asking "Why?" might look like this:

1. Why is the app crashing? → Because a null value is being accessed in the code.

2. Why is there a null value? → Because the API response is missing a required field

3. Why is the field missing? → Because a recent backend change made the field optional.

4. Why was this change not handled in the app? → Because the app assumes the field is always present.

5. Why was this assumption not caught earlier? → Because there are no contract tests validating API responses.

At this point, the issue is no longer just "fix the null check". The deeper problem is the lack of validation between systems, which allows breaking changes to slip through.

A useful way to think about the 5 Whys is that it forces you to stay with the problem a little longer than you normally would. Most of the time, the first explanation feels sufficient, so it's easy to stop there. This method pushes you to go one step further, and then another, until the explanation holds up under scrutiny.

At the same time, it's not a rigid formula. You might reach a solid root cause in three steps, or it might take more than five. The quality of the reasoning matters more than the count.

Origins of the 5 Whys Method

The 5 Whys method comes from the Toyota Production System, a manufacturing approach focused on continuous improvement and problem solving at the source.

It's often associated with Sakichi Toyoda, whose philosophy was simple: don’t just fix a problem. Understand why it happened so it doesn't happen again.

Inside Toyota, this wasn't treated as a formal tool or checklist. It was part of the day-to-day way of working. When something went wrong on the production line, the goal wasn't to get things running quickly and move on. The goal was to stop, investigate, and make sure the same issue wouldn't repeat.

That mindset is important to understand. The 5 Whys was never meant to be a rigid exercise where you ask five questions and stop. It was a way to encourage deeper thinking and accountability in processes.

Another key idea in the Toyota system is that problems are usually caused by processes, not people. Instead of asking "who made the mistake", the focus is on "what allowed this mistake to happen". The 5 Whys fits naturally into this approach because it pushes you toward system level causes rather than individual blame.

Over time, the method spread beyond manufacturing and is now used in software engineering, product teams, operations, and many other fields. The context has changed, but the core idea remains the same: if you don't understand the cause, you're likely to see the same problem again.

This origin story is useful not just as background, but as a reminder of intent. The value of the 5 Whys doesn't come from the questions themselves. It comes from the discipline of not settling for the first answer.

How to Conduct an Effective 5 Whys Analysis

A 5 Whys analysis works best when it is treated as a structured way of thinking, not a checklist to rush through. The quality of the outcome depends less on how many times you ask "why" and more on how carefully you reason through each step.

It helps to approach it in stages, each with a clear purpose.

Step 1: Define the Problem Clearly

Start with a problem statement that is specific and observable. Avoid vague descriptions like "the system is slow" or "things are failing". Instead, describe what actually happened in a way that can be verified.

For example, "API response time exceeded 5 seconds for 30 percent of requests between 2 PM and 3 PM" is much more useful than "API is slow".

A clear problem statement keeps the analysis grounded. If the starting point is fuzzy, the entire chain of reasoning will drift.

Step 2: Ask "Why" Iteratively

Once the problem is defined, begin asking why it happened. Each answer should directly address the question before it and naturally lead to the next one.

The key here is continuity. Every step should feel like a logical extension of the previous one. If you find yourself jumping topics or introducing unrelated explanations, it's a sign that the chain is breaking.

Keep going until the answers stop being immediate symptoms and start pointing toward underlying conditions or decisions.

Also, don't force the process to stop at five. Some problems may need fewer steps, while others may need more. What matters is reaching a point where the explanation is meaningful and actionable.

Step 3: Validate Each Answer with Evidence

This is where many analyses go wrong. It's easy to come up with plausible answers, but plausibility is not enough.

Each "why" should be backed by some form of evidence. This could be logs, metrics, recent changes, or direct observation. If an answer can't be verified, treat it as a hypothesis and confirm it before moving forward.

Without validation, the entire analysis becomes a chain of assumptions. Even if the final answer sounds reasonable, it may not reflect reality.

Step 4: Identify the Root Cause

A good root cause is one that explains the sequence of events and can be acted upon to prevent the issue in the future.

In many cases, this turns out to be a gap in a process rather than a single technical failure. It could be a missing validation step, an incomplete test, or an assumption that was never challenged.

If the final answer still feels like a symptom, you probably need to go one level deeper. On the other hand, if the answer points to something you can change in your system or workflow, you are likely in the right place.

Step 5: Define Corrective Actions

The analysis is only useful if it leads to meaningful action.

Once you've identified the root cause, the next step is to define changes that prevent the problem from happening again. These should go beyond quick fixes and address the underlying issue.

For example, instead of just fixing a bug, you might introduce better testing, add monitoring, or improve review processes.

Good corrective actions share a few traits: they're specific, practical to implement, and they directly address the root cause identified in the analysis.

Real-World Example: Applying 5 Whys in an Engineering Scenario

To see how this works in practice, let’s walk through a realistic backend issue. The goal here is not just to reach an answer, but to show how each step builds on evidence and leads to something actionable.

The Problem:

Users report intermittent failures while fetching order details:

GET /api/orders/{id}
→ HTTP 500 Internal Server Error

Application logs show:

// Java 21 example (Spring Boot style logging)
logger.error("Database connection timeout while fetching order", ex);

At this point, it's tempting to conclude that the database is the problem. But that's only what we can see on the surface.

Applying the 5 Whys

1. Why did the API return a 500 error?

Because the database query timed out.

This is directly supported by the error logs, so we can treat it as a confirmed fact.

2. Why did the query time out?

Because the database connection pool was exhausted.

Metrics show that all available connections were in use during peak traffic.

3. Why was the connection pool exhausted?

Because some requests were holding database connections for too long.

Slow query logs confirm that a subset of queries had unusually high execution times.

4. Why were some queries slow?

Because a recently introduced feature added a query on a non-indexed column.

Looking at recent deployments reveals a change that introduced filtering without proper indexing.

5. Why was an unoptimized query deployed to production?

Because there is no performance validation step in the development or release process.

There are no checks in code review or CI/CD to catch inefficient database queries before deployment.

Root Cause

The issue is not the timeout itself.

It's this:

The system allows inefficient database queries to reach production without any safeguards.

What a Shallow Fix Would Look Like

If we stopped early, we might:

• Increase the database timeout

• Increase the connection pool size

These might reduce the frequency of failures, but they don't solve the underlying problem.

What a Strong Fix Looks Like

A proper 5 Whys analysis leads to changes that improve the system:

• Add appropriate indexing for frequently queried fields

• Introduce query performance checks in CI/CD pipelines

• Add monitoring and alerts for slow queries

• Include database considerations in code reviews

Why This Example Matters

The difference between a shallow fix and a real solution is depth.

The first explanation often feels sufficient, especially under pressure. But stopping there means the issue is likely to return in a different form.

The value of the 5 Whys comes from following the chain all the way to something you can change in your system.

When to Use (and When Not to Use) 5 Whys

Like any problem-solving method, the 5 Whys is useful in the right context and less effective in others. Knowing when to apply it is just as important as knowing how to use it.

If used appropriately, it can uncover meaningful insights. If used in the wrong situation, it can lead to oversimplified or misleading conclusions

When to Use 5 Whys

The 5 Whys is most useful when your goal is to understand why something happened, not just to fix it and move on.

It works well in situations where problems are recurring or not fully explained by the first answer. For example, production incidents, repeated bugs, or issues that reappear after a quick fix are strong signals that you need deeper analysis. In these cases, the technique helps uncover what is happening beneath the surface.

It's also effective during retrospectives and postmortems. When a release doesn't go as expected or a sprint runs into issues, the 5 Whys helps teams move beyond observations like "this failed" and get to "why did this fail in the first place".

In general, use it when:

• The problem is not obvious

• The issue has occurred more than once

• You want to prevent recurrence, not just resolve the current instance

When Not to Use 5 Whys

The 5 Whys has its limits, and using it in the wrong context can lead to oversimplified conclusions.

If a problem involves multiple interacting factors, a single chain of "why" questions may not capture the full picture. Complex systems often have several contributing causes, and forcing them into one linear explanation can hide important details. In such cases, the 5 Whys should be combined with other approaches.

It's also less effective when there's not enough data. If each answer is based on assumptions rather than evidence, the analysis quickly becomes unreliable. The method depends on validation at every step.

Another limitation is in time-critical situations. During an active incident, the priority is to restore the system. The deeper analysis should happen later, once things are stable.

Finally, if your goal is quantitative analysis or optimization, the 5 Whys alone isn't enough. You'll need more data-driven methods to support decision making.

A simple rule of thumb is this. If you are trying to learn from a problem, use the 5 Whys. If you are trying to fix something immediately or analyze complex data, use it carefully or alongside other techniques.

Benefits of the 5 Whys Technique

The 5 Whys technique is simple, but it offers several powerful benefits that can help you solve problems more effectively and make lasting improvements. Here are the key advantages:

Simple and Easy to Apply

One of the biggest strengths of the 5 Whys is how easy it is to start using. You don't need special tools, training, or complex frameworks. It can be applied in a quick discussion, during debugging, or as part of a formal postmortem.

This low barrier makes it accessible across teams, regardless of experience level.

Encourages Deeper Thinking

The method naturally pushes you to go beyond the first explanation. Instead of reacting to what's visible, it encourages you to question why the problem occurred in the first place.

This shift from surface-level fixes to deeper understanding often leads to better decisions.

Promotes System-Level Improvements

When used correctly, the focus moves away from individual people and toward systems. Instead of asking who made a mistake, the analysis asks what allowed the mistake to happen.

This leads to improvements in processes, safeguards, and overall system design rather than one-off fixes.

Works Well in Team Settings

Because the approach is simple, it's easy for multiple people to contribute. Different perspectives help uncover gaps that might otherwise be missed.

It also creates a shared understanding of the problem, which is valuable during retrospectives and incident reviews.

Helps Prevent Recurring Issues

Quick fixes often solve the immediate problem but don't stop it from happening again. The 5 Whys helps identify underlying causes, which makes it easier to prevent similar issues in the future.

Over time, this leads to more stable systems and fewer repeated incidents.

Common Pitfalls and Limitations

While the 5 Whys technique is useful, it’s not always perfect. There are some limitations to keep in mind, so you can use it effectively and know when it might not be enough.

Stopping Too Early

One of the most common mistakes is ending the analysis after the first or second answer. These early answers usually describe symptoms, not causes.

Stopping too soon leads to fixes that address the surface but leave the underlying issue unresolved.

Treating Assumptions as Facts

It's easy to come up with explanations that sound reasonable. But without evidence, they're just assumptions.

If each step isn't validated with logs, metrics, or observations, the entire analysis can drift away from reality.

Focusing on Individuals Instead of Systems

Answers like "someone made a mistake" don't add much value. While they may be true, they don't explain why the system allowed that mistake to have an impact.

Focusing on processes and safeguards leads to more meaningful improvements.

Oversimplifying Complex Problems

The 5 Whys follows a linear chain of reasoning, but real-world systems often have multiple contributing factors.

Relying on a single chain can hide important interactions. In such cases, the method should be combined with other approaches.

Treating It as a Rigid Formula

The name suggests asking "why" five times, but this shouldn't be taken literally. Some problems require fewer steps, while others need more.

Forcing the structure can lead to artificial or weak conclusions.

Not a Replacement for Deeper Analysis

The 5 Whys isn't designed for every type of problem. For complex system failures, performance optimization, or data-heavy investigations, additional tools and methods are often required.

It works best as a starting point or a complement to other techniques, not a complete solution on its own.

Tips for Using 5 Whys Effectively

To get the most out of the 5 Whys technique, there are a few tips that can help you use it effectively. These will guide you to ask the right questions and reach useful, actionable insights.

Start with a Clear, Specific Problem

A vague problem leads to vague answers. Spend a little extra time making sure the problem statement is precise and based on observable facts. This keeps the analysis grounded and avoids unnecessary detours.

Base Every Step on Evidence

Treat each answer as something that needs to be verified. Use logs, metrics, recent changes, or direct observations to support your reasoning. If something can't be validated, call it out as a hypothesis and confirm it before moving forward.

Keep the Chain Logical and Connected

Each "why" should naturally follow from the previous answer. If the reasoning starts to jump between unrelated ideas, pause and re-evaluate. A clean, logical chain is a strong indicator that you're on the right track.

Focus on Systems, Not Individuals

Avoid stopping at explanations that point to human error. Instead, ask what allowed that error to have an impact. This shift in thinking leads to improvements that actually reduce the chances of similar issues in the future.

Do Not Force Exactly Five Steps

The number five is a guideline, not a rule. Some problems become clear in three steps, while others need more exploration. Stop when you reach a cause that's both convincing and actionable.

Involve the Right People

If possible, do the analysis as a group. People from different parts of the system bring different perspectives, which helps uncover details that might otherwise be missed. It also creates shared ownership of both the problem and the solution.

Turn Insights into Actions

The analysis only matters if it leads to change. Make sure the final outcome includes clear, practical steps that address the root cause. Without this, even a well-done analysis has limited impact.

Summary

The 5 Whys is a simple technique, but using it well takes some discipline.

At its core, it's about resisting the urge to stop at the first explanation. By following the chain of cause and effect, you move from symptoms to something you can actually fix. In many cases, that turns out to be a gap in a process rather than a one-off failure.

When applied thoughtfully, it helps teams learn from problems instead of just reacting to them. Over time, this leads to better systems, fewer recurring issues, and more confidence in how problems are handled.

The key is to treat it as a way of thinking, not just a set of steps.

In this tutorial, you’ll learn how to build a simple website screenshot generator using Python and Flask. The app will let users enter any website URL and instantly get a screenshot of that page – all powered by a screenshot API.

You’ll use Flask, a lightweight web framework, to create a simple web interface and handle requests. Then, you’ll integrate an external API to capture website screenshots programmatically.

By the end of this tutorial, you’ll have learned how to:

• Build a basic Flask web app

• Accept user input through an HTML form

• Make HTTP requests to an external API

• Display images dynamically on a web page

This project is a great way to learn how APIs can extend the capabilities of your web applications, and how Python can easily handle tasks like image generation and display.

What we’ll cover:

• Prerequisites

• Setting Up the Project

• Integrating the ScreenshotBase API

• Adding Customization Options

• Wrapping Up

Prerequisites

Before we start building the app, make sure you have a few basics covered:

1. Python Installed

You’ll need Python 3.9 or higher installed on your machine. You can check your version by running:

python --version

If you don’t have it, you can download it from python.org/downloads.

2. Basic Knowledge of Python and Flask

You don’t need to be a Flask expert. Just have some familiarity with how to create routes and templates, and how to handle form data in Flask.

If you’re new to Flask, don’t worry – we’ll go step-by-step.

3. Screenshot API Key

We’ll be using the ScreenshotBase API to capture website screenshots. It provides a simple REST endpoint that takes a URL and returns a screenshot image.

You can get a free API key by signing up on their website. Once you have the key, keep it handy, as we’ll use it when we integrate the API in the later steps.

4. A Code Editor and Terminal

You can use any editor you prefer, like VS Code, PyCharm, or even a simple text editor. Make sure your terminal or command prompt can run Python commands and install packages using pip.

Setting Up the Project

Let’s start by setting up a new Flask project from scratch.

Step 1: Create a New Folder

Create a new folder for your project. You can name it screenshot-generator:

mkdir screenshot-generator
cd screenshot-generator

Step 2: Create a Virtual Environment

A virtual environment helps keep your project dependencies isolated from other Python projects.

Run the following commands:

python -m venv venv

Then activate it:

• On macOS/Linux:

    source venv/bin/activate
  

• On Windows:

    venv\Scripts\activate
  

Once activated, you should see (venv) at the beginning of your terminal prompt.

Step 3: Install Dependencies

We’ll need two packages for this project:

• Flask for building the web app

• Requests for making API calls to ScreenshotBase

Install them using pip:

pip install flask requests

Step 4: Set Up the Project Structure

Inside your project folder, create the following files and folders:

screenshot-generator/
├── app.py
├── templates/
│   └── index.html
└── static/

Here’s what each part does:

• app.py: main Python file that runs your Flask app

• templates/: stores HTML templates

• static/: stores images, CSS, and JavaScript files

Step 5: Add a Basic Flask App

Open app.py and add the following starter code:

from flask import Flask, render_template, request
import requests
import os

app = Flask(__name__)
API_KEY = os.getenv("SCREENSHOTBASE_API_KEY")
SCREENSHOTBASE_BASE_ENDPOINT = "https://api.screenshotbase.com/v1/take"

@app.route('/', methods=['GET', 'POST'])
def home():
    if request.method == 'POST':
        url = request.form.get('url')
        # Placeholder: We’ll add the API call here in the next section
        return render_template('index.html', url=url)
    return render_template('index.html')

if __name__ == '__main__':
    app.run(debug=True)

This sets up a minimal Flask application with one route (/). We’ll add the ScreenshotBase API call in the next section.

Step 6: Create a Simple HTML Template

Now that we’ve written the Flask backend to handle the screenshot request, let’s create the frontend for our app.

Inside your project folder, create a new directory named templates, and within it, add a file called index.html. Flask will automatically look for templates in this folder.

Here’s how the template should look:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Website Screenshot Generator</title>
    <link
      href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.8/dist/css/bootstrap.min.css"
      rel="stylesheet"
      integrity="sha384-sRIl4kxILFvY47J16cr9ZwB07vP4J8+LH7qKQnuqkuIAvNWLzeN8tE5YBujZqJLB"
      crossorigin="anonymous"
    />
    <style>
        body {
            padding-top: 2rem;
        }
        .screenshot-container {
            max-height: 80vh;
            overflow-y: auto;
            border: 1px solid #ddd;
            border-radius: 8px;
            padding: 1rem;
            background: #f8f9fa;
        }
    </style>
</head>
<body>
    <div class="container text-center">
        <h1 class="mb-4">Website Screenshot Generator</h1>
        <form method="POST" class="d-flex justify-content-center mb-4">
            <input 
                type="text" 
                name="url" 
                placeholder="Enter website URL" 
                class="form-control w-50 me-2"
                required
            >
            <button type="submit" class="btn btn-primary">Capture Screenshot</button>
        </form>

        {% if screenshot %}
            <h4>Screenshot Preview:</h4>
            <div class="screenshot-container mx-auto">
                <img src="{{ screenshot }}" alt="Website Screenshot" class="img-fluid rounded shadow">
            </div>
        {% elif request.method == 'POST' %}
            <div class="alert alert-danger mt-3">
                Sorry, something went wrong while capturing the screenshot. Please try again.
            </div>
        {% endif %}
    </div>
</body>
</html>

This HTML template uses Bootstrap to keep the design clean and responsive without much custom styling. The form at the top allows users to enter any website URL and submit it to the Flask app using the POST method. Once the app retrieves the screenshot URL from the API, it dynamically renders the image on the page.

The img-fluid class from Bootstrap ensures the screenshot scales properly across all screen sizes while maintaining its aspect ratio. Also, the .screenshot-container provides a scrollable area, which helps display full-page screenshots without shrinking them too much or breaking the layout.

Now your project is set up and ready to capture real screenshots using the ScreenshotBase API.

In the next section, we’ll write the logic to call the ScreenshotBase API and display the resulting screenshot dynamically in the browser.

Integrating the ScreenshotBase API

Now that our Flask app is set up, let’s connect it to the ScreenshotBase API so we can generate real screenshots from any URL.

Step 1: Understanding the API Endpoint

The ScreenshotBase API provides a simple endpoint that takes a website URL and returns a screenshot image.

A typical API call looks like this:

GET https://api.screenshotbase.com/v1/take

You send the target website URL as a query parameter (url) and include your API key in the request header as apikey.

Here’s the cURL example from their documentation:

curl -G https://api.screenshotbase.com/v1/take?url=https%3A%2F%2Fbbc.com \
    -H "apikey: YOUR-API-KEY"

This request captures a screenshot of https://bbc.com and returns the screenshot image as the response.

Step 2: Add the API Call in Flask

Let’s update our home route in app.py to send a request to the ScreenshotBase API when a user submits a URL.

Here’s the updated version of app.py:

from flask import Flask, render_template, request
import requests
import os

app = Flask(__name__)
API_KEY = os.getenv("SCREENSHOTBASE_API_KEY")
SCREENSHOTBASE_BASE_ENDPOINT = "https://api.screenshotbase.com/v1/take"

@app.route('/', methods=['GET', 'POST'])
def home():
    screenshot_url = None

    if request.method == 'POST':
        target_url = request.form.get('url')

        params = {"url": target_url}
        headers = {"apikey": API_KEY}

        try:
            # Send GET request to ScreenshotBase API
            response = requests.get(SCREENSHOTBASE_BASE_ENDPOINT, params=params, headers=headers, timeout=30)
            response.raise_for_status()

            # Save the returned image
            image_path = os.path.join('static', 'screenshot.png')
            with open(image_path, 'wb') as f:
                f.write(response.content)

            screenshot_url = image_path

        except requests.exceptions.RequestException as e:
            print(f"Error capturing screenshot: {e}")

    return render_template('index.html', screenshot=screenshot_url)

if __name__ == '__main__':
    app.run(debug=True)

Here’s what this code does:

1. Captures the user input (URL).

2. Sends a GET request to the /v1/take endpoint.

3. Passes your API key in the request header (apikey).

4. Saves the returned image to the static folder.

5. Displays the screenshot on the page.

Step 3: Test Your App

Run the app again:

python app.py

Then open http://127.0.0.1:5000 in your browser.

Enter a URL like:

https://github.com/ashutoshkrris

After submitting, you should see the screenshot of that website displayed below the form.

Your Flask app is now fully functional! It takes a URL, sends it to the ScreenshotBase API, and displays the resulting screenshot.

In the next section, we’ll enhance the app by adding customization options like full-page screenshots, viewport settings, and delays. Later, we’ll explore the ScreenshotBase SDKs for popular languages like Python, Node.js, and PHP.

Adding Customization Options

Right now, our app simply takes a screenshot of the given website URL using the default settings. However, most screenshot APIs (including the one we’re using) allow you to customize the output by passing additional parameters in the request.

The ScreenshotBase API offers several powerful customization options to help you tailor each screenshot to your needs:

• Image format: Choose from png, jpg, gif, or webp, depending on your image quality or compression requirements.

• Full page capture: Capture the entire scrollable webpage (full_page=1) or just the visible viewport (full_page=0).

• Viewport dimensions: Set the browser window size with viewport_width and viewport_height to simulate screenshots from desktop, tablet, or mobile screens.

These options make ScreenshotBase ideal for building automation tools, thumbnail generators, testing dashboards, and visual documentation systems. Let’s add these options to make our screenshot generator more flexible.

Updating the Flask Code

Open your app.py file and update the route that handles form submissions to include these optional parameters:

from flask import Flask, render_template, request
import requests
import os

app = Flask(__name__)
API_KEY = os.getenv("SCREENSHOTBASE_API_KEY", "scr_live_9Qkn1gs01rivZqrFk7lusXPiqAUg85J86aU6bHvG")
SCREENSHOTBASE_BASE_ENDPOINT = "https://api.screenshotbase.com/v1/take"


@app.route('/', methods=['GET', 'POST'])
def home():
    screenshot_url = None

    if request.method == 'POST':
        target_url = request.form.get('url')
        format_ = request.form.get('format', 'png')
        full_page = request.form.get('full_page') == 'on'

        params = {
            "url": target_url,
            "format": format_,
            "full_page": int(full_page)
        }
        headers = {"apikey": API_KEY}

        try:
            # Send GET request to ScreenshotBase API
            response = requests.get(SCREENSHOTBASE_BASE_ENDPOINT, params=params, headers=headers, timeout=30)
            response.raise_for_status()

            # Save the returned image
            image_extension = format_ if format_ != 'jpeg' else 'jpg'
            image_path = os.path.join('static', f'screenshot.{image_extension}')
            with open(image_path, 'wb') as f:
                f.write(response.content)

            screenshot_url = image_path

        except requests.exceptions.RequestException as e:
            print(f"Error capturing screenshot: {e}")

    return render_template('index.html', screenshot=screenshot_url)


if __name__ == '__main__':
    app.run(debug=True)

Updating the HTML Template

Next, let’s modify the form in our index.html to include the customization options:

<form method="POST" class="d-flex flex-column justify-content-center mb-4">
  <!-- URL and Submit Button in One Row -->
  <div class="input-group mb-4">
    <input
      type="text"
      name="url"
      placeholder="Enter website URL"
      class="form-control w-50 me-2"
      required
    />
    <button type="submit" class="btn btn-primary">Capture Screenshot</button>
  </div>

  <!-- Options in Two Columns -->
  <div class="row g-3">
    <!-- Full Page Checkbox -->
    <div class="col-md-6">
      <div class="form-check">
        <input
          type="checkbox"
          name="full_page"
          id="full_page"
          class="form-check-input"
        />
        <label class="form-check-label" for="full_page">
          Capture Full Page Screenshot
        </label>
      </div>
    </div>

    <!-- Format Dropdown -->
    <div class="col-md-6">
      <label for="format" class="form-label">Screenshot Format</label>
      <select class="form-select" name="format" id="format" required>
        <option value="png">PNG</option>
        <option value="jpg">JPG</option>
        <option value="gif">GIF</option>
        <option value="webp">WEBP</option>
      </select>
    </div>

    <!-- Viewport Width -->
    <div class="col-md-6">
      <label for="viewport_width" class="form-label">Viewport Width</label>
      <input
        type="number"
        name="viewport_width"
        id="viewport_width"
        class="form-control"
        value="1280"
        min="320"
      />
    </div>

    <!-- Viewport Height -->
    <div class="col-md-6">
      <label for="viewport_height" class="form-label">Viewport Height</label>
      <input
        type="number"
        name="viewport_height"
        id="viewport_height"
        class="form-control"
        value="720"
        min="320"
      />
    </div>
  </div>
</form>

The updated form gives users control over how the screenshot is generated. The format dropdown lets them choose between PNG, JPG, GIF, or WEBP. The Full Page checkbox toggles whether the API captures the entire scrollable webpage or just the visible viewport. The viewport width and height fields define the browser window dimensions, which is useful if you want to simulate different device sizes or responsive layouts.

When the form is submitted, Flask reads these values and sends them as query parameters in the API request.

For example, a request to capture a full-page, 1920×1080 PNG screenshot of your site would look like this:

https://api.screenshotbase.com/v1/take?url=https%3A%2F%2Fgithub.com%2Fashutoshkrris&format=png&full_page=1&viewport_width=1920&viewport_height=1080

This flexibility makes it easy to fine-tune screenshots for different use cases – whether you’re generating thumbnails, testing responsiveness, or automating visual reports.

Note: If you prefer working with SDKs instead of direct API calls, ScreenshotBase also offers official SDKs for popular languages like Python, JavaScript, Ruby, PHP, and Go.

These SDKs provide a simpler and more convenient way to interact with the API, handling authentication and request formatting behind the scenes.

You can explore them in the ScreenshotBase SDK documentation.

Wrapping Up

In this tutorial, you learned how to build a simple Flask application that captures website screenshots using a third-party API. We explored how to send requests, handle image responses, and add customization options like image format, full-page capture, and viewport size – all while keeping the project lightweight and easy to extend.

This small project demonstrates how web automation tasks, such as generating previews or visual reports, can be simplified using modern APIs. You can build upon this foundation to create batch screenshot tools, visual monitoring systems, or even integrate screenshots into larger web applications.

The key takeaway is understanding how to interact with external APIs, process responses, and design a clean interface for users. These are skills that are essential for backend and full-stack developers alike.

Instead, it stores everything in a simple file on your computer. It also requires zero configuration, so there’s no complicated setup process, making it perfect for beginners and small projects.

SQLite is a great choice for small to medium applications because it’s easy to use, fast, and can handle most tasks that bigger databases can do, but without the hassle of managing extra software. Whether you're building a personal project or prototyping a new app, SQLite is a solid option to get things up and running quickly.

In this tutorial, you'll learn how to work with SQLite using Python. Here’s what we’re going to cover in this tutorial:

• How to Set Up Your Python Environment

• How to Create an SQLite Database

• How to Create Database Tables

• How to Insert Data into a Table

• How to Query Data

• How to Update and Delete Data

• How to Use Transactions

• How to Optimize SQLite Query Performance with Indexing

• How to Handle Errors and Exceptions

• How to Export and Import Data [Bonus Section]

• Wrapping Up

This tutorial is perfect for anyone who wants to get started with databases without diving into complex setups.

How to Set Up Your Python Environment

Before working with SQLite, let’s ensure your Python environment is ready. Here’s how to set everything up.

Installing Python

If you don’t have Python installed on your system yet, you can download it from the official Python website. Follow the installation instructions for your operating system (Windows, macOS, or Linux).

To check if Python is installed, open your terminal (or command prompt) and type:

python --version

This should show the current version of Python installed. If it’s not installed, follow the instructions on the Python website.

Installing SQLite3 Module

The good news is that SQLite3 comes built-in with Python! You don’t need to install it separately because it’s included in the standard Python library. This means you can start using it right away without any additional setup.

How to Create a Virtual Environment (Optional but Recommended)

It’s a good idea to create a virtual environment for each project to keep your dependencies organized. A virtual environment is like a clean slate where you can install packages without affecting your global Python installation.

To create a virtual environment, follow these steps:

1. First, open your terminal or command prompt and navigate to the directory where you want to create your project.

2. Run the following command to create a virtual environment:

python -m venv env

Here, env is the name of the virtual environment. You can name it anything you like.

3. Activate the virtual environment:

# Use the command for Windows
env\Scripts\activate

# Use the command for macOS/Linux:
env/bin/activate

After activating the virtual environment, you’ll notice that your terminal prompt changes, showing the name of the virtual environment. This means you’re now working inside it.

Installing Necessary Libraries

We’ll need a few additional libraries for this project. Specifically, we’ll use:

• pandas: This is an optional library for handling and displaying data in tabular format, useful for advanced use cases.

• faker: This library will help us generate fake data, like random names and addresses, which we can insert into our database for testing.

To install pandas and faker, simply run the following commands:

pip install pandas faker

This installs both pandas and faker into your virtual environment. With this, your environment is set up, and you’re ready to start creating and managing your SQLite database in Python!

How to Create an SQLite Database

A database is a structured way to store and manage data so that it can be easily accessed, updated, and organized. It’s like a digital filing system that allows you to efficiently store large amounts of data, whether it’s for a simple app or a more complex system. Databases use tables to organize data, with rows and columns representing individual records and their attributes.

How SQLite Databases Work

Unlike most other database systems, SQLite is a serverless database. This means that it doesn’t require setting up or managing a server, making it lightweight and easy to use. All the data is stored in a single file on your computer, which you can easily move, share, or back up. Despite its simplicity, SQLite is powerful enough to handle many common database tasks and is widely used in mobile apps, embedded systems, and small to medium-sized projects.

How to Create a New SQLite Database

Let’s create a new SQLite database and learn how to interact with it using Python’s sqlite3 library.

Connecting to the Database

Since sqlite3 is pre-installed, you just need to import it in your Python script. To create a new database or connect to an existing one, we use the sqlite3.connect() method. This method takes the name of the database file as an argument. If the file doesn’t exist, SQLite will automatically create it.

import sqlite3

# Connect to the SQLite database (or create it if it doesn't exist)
connection = sqlite3.connect('my_database.db')

In this example, a file named my_database.db is created in the same directory as your script. If the file already exists, SQLite will just open the connection to it.

Creating a Cursor

Once you have a connection, the next step is to create a cursor object. The cursor is responsible for executing SQL commands and queries on the database.

# Create a cursor object
cursor = connection.cursor()

Closing the Connection

After you’ve finished working with the database, it’s important to close the connection to free up any resources. You can close the connection with the following command:

# Close the database connection
connection.close()

However, you should only close the connection once you’re done with all your operations.

When you run your Python script, a file named my_database.db will be created in your current working directory. You’ve now successfully created your first SQLite database!

How to Use Context Manager to Open and Close Connections

Python provides a more efficient and cleaner way to handle database connections using the with statement, also known as a context manager. The with statement automatically opens and closes the connection, ensuring that the connection is properly closed even if an error occurs during the database operations. This eliminates the need to manually call connection.close().

Here’s how you can use the with statement to handle database connections:

import sqlite3

# Step 1: Use 'with' to connect to the database (or create one) and automatically close it when done
with sqlite3.connect('my_database.db') as connection:

    # Step 2: Create a cursor object to interact with the database
    cursor = connection.cursor()

    print("Database created and connected successfully!")

# No need to call connection.close(); it's done automatically!

From now on, we’ll use the with statement in our upcoming code examples to manage database connections efficiently. This will make the code more concise and easier to maintain.

How to Create Database Tables

Now that we’ve created an SQLite database and connected to it, the next step is to create tables inside the database. A table is where we’ll store our data, organized in rows (records) and columns (attributes). For this example, we’ll create a table called Students to store information about students, which we’ll reuse in upcoming sections.

To create a table, we use SQL's CREATE TABLE statement. This command defines the table structure, including the column names and the data types for each column.

Here’s a simple SQL command to create a Students table with the following fields:

• id: A unique identifier for each student (an integer).

• name: The student's name (text).

• age: The student's age (an integer).

• email: The student's email address (text).

The SQL command to create this table would look like this:

CREATE TABLE Students (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    age INTEGER,
    email TEXT
);

We can execute this CREATE TABLE SQL command in Python using the sqlite3 library. Let’s see how to do that.

import sqlite3

# Use 'with' to connect to the SQLite database and automatically close the connection when done
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to create the Students table
    create_table_query = '''
    CREATE TABLE IF NOT EXISTS Students (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL,
        age INTEGER,
        email TEXT
    );
    '''

    # Execute the SQL command
    cursor.execute(create_table_query)

    # Commit the changes
    connection.commit()

    # Print a confirmation message
    print("Table 'Students' created successfully!")

• IF NOT EXISTS: This ensures that the table is only created if it doesn’t already exist, preventing errors if the table has been created before.

• connection.commit(): This saves (commits) the changes to the database.

When you run the Python code above, it will create the Students table in the my_database.db database file. You’ll also see a message in the terminal confirming that the table has been created successfully.

If you’re using Visual Studio Code, you can install the SQLite Viewer extension to view SQLite databases.

Data Types in SQLite and Their Mapping to Python

SQLite supports several data types, which we need to understand when defining our tables. Here’s a quick overview of common SQLite data types and how they map to Python types:

In our Students table:

• id is of type INTEGER, which maps to Python’s int.

• name and email are of type TEXT, which map to Python’s str.

• age is also of type INTEGER, mapping to Python’s int.

How to Insert Data into a Table

Now that we have our Students table created, it’s time to start inserting data into the database. In this section, we’ll cover how to insert both single and multiple records using Python and SQLite, and how to avoid common security issues like SQL injection by using parameterized queries.

How to Insert a Single Record

To insert data into the database, we use the INSERT INTO SQL command. Let’s start by inserting a single record into our Students table.

Here’s the basic SQL syntax for inserting a single record:

INSERT INTO Students (name, age, email) 
VALUES ('John Doe', 20, 'johndoe@example.com');

However, instead of writing SQL directly in our Python script with hardcoded values, we’ll use parameterized queries to make our code more secure and flexible. Parameterized queries help prevent SQL injection, a common attack where malicious users can manipulate the SQL query by passing harmful input.

Here’s how we can insert a single record into the Students table using a parameterized query:

import sqlite3

# Use 'with' to open and close the connection automatically
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # Insert a record into the Students table
    insert_query = '''
    INSERT INTO Students (name, age, email) 
    VALUES (?, ?, ?);
    '''
    student_data = ('Jane Doe', 23, 'jane@example.com')

    cursor.execute(insert_query, student_data)

    # Commit the changes automatically
    connection.commit()

    # No need to call connection.close(); it's done automatically!
    print("Record inserted successfully!")

The ? placeholders represent the values to be inserted into the table. The actual values are passed as a tuple (student_data) in the cursor.execute() method.

How to Insert Multiple Records

If you want to insert multiple records at once, you can use the executemany() method in Python. This method takes a list of tuples, where each tuple represents one record.

To make our example more dynamic, we can use the Faker library to generate random student data. This is useful for testing and simulating real-world scenarios.

from faker import Faker
import sqlite3

# Initialize Faker
fake = Faker(['en_IN'])

# Use 'with' to open and close the connection automatically
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # Insert a record into the Students table
    insert_query = '''
    INSERT INTO Students (name, age, email) 
    VALUES (?, ?, ?);
    '''
    students_data = [(fake.name(), fake.random_int(
        min=18, max=25), fake.email()) for _ in range(5)]

    # Execute the query for multiple records
    cursor.executemany(insert_query, students_data)

    # Commit the changes
    connection.commit()

    # Print confirmation message
    print("Fake student records inserted successfully!")

In this code:

• Faker() generates random names, ages, and emails for students. Passing the locale([‘en_IN’]) is optional.

• cursor.executemany(): This method allows us to insert multiple records at once, making the code more efficient.

• students_data: A list of tuples where each tuple represents one student’s data.

How to Handle Common Issues: SQL Injection

SQL injection is a security vulnerability where attackers can insert or manipulate SQL queries by providing harmful input. For example, an attacker might try to inject code like '; DROP TABLE Students; -- to delete the table.

By using parameterized queries (as demonstrated above), we avoid this issue. The ? placeholders in parameterized queries ensure that input values are treated as data, not as part of the SQL command. This makes it impossible for malicious code to be executed.

How to Query Data

Now that we’ve inserted some data into our Students table, let’s learn how to retrieve the data from the table. We'll explore different methods for fetching data in Python, including fetchone(), fetchall(), and fetchmany().

To query data from a table, we use the SELECT statement. Here’s a simple SQL command to select all columns from the Students table:

SELECT * FROM Students;

This command retrieves all records and columns from the Students table. We can execute this SELECT query in Python and fetch the results.

How to Fetch All Records

Here’s how we can fetch all records from the Students table:

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Execute the SQL command
    cursor.execute(select_query)

    # Fetch all records
    all_students = cursor.fetchall()

    # Display results in the terminal
    print("All Students:")
    for student in all_students:
        print(student)

In this example, the fetchall() method retrieves all rows returned by the query as a list of tuples.

All Students:
(1, 'Jane Doe', 23, 'jane@example.com')
(2, 'Bahadurjit Sabharwal', 18, 'tristanupadhyay@example.net')
(3, 'Zayyan Arya', 20, 'yashawinibhakta@example.org')
(4, 'Hemani Shukla', 18, 'gaurikanarula@example.com')
(5, 'Warda Kara', 20, 'npatil@example.net')
(6, 'Mitali Nazareth', 19, 'sparekh@example.org')

How to Fetch a Single Record

If you want to retrieve only one record, you can use the fetchone() method:

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Execute the SQL command
    cursor.execute(select_query)

    # Fetch one record
    student = cursor.fetchone()

    # Display the result
    print("First Student:")
    print(student)

Output:

First Student:
(1, 'Jane Doe', 23, 'jane@example.com')

How to Fetch Multiple Records

To fetch a specific number of records, you can use fetchmany(size):

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:

    # Create a cursor object
    cursor = connection.cursor()

    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Execute the SQL command
    cursor.execute(select_query)

    # Fetch three records
    three_students = cursor.fetchmany(3)

    # Display results
    print("Three Students:")
    for student in three_students:
        print(student)

Output:

Three Students:
(1, 'Jane Doe', 23, 'jane@example.com')
(2, 'Bahadurjit Sabharwal', 18, 'tristanupadhyay@example.net')
(3, 'Zayyan Arya', 20, 'yashawinibhakta@example.org')

How to Use pandas for Better Data Presentation

For better data presentation, we can use the pandas library to create a DataFrame from our query results. This makes it easier to manipulate and visualize the data.

Here’s how to fetch all records and display them as a pandas DataFrame:

import sqlite3
import pandas as pd

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:
    # Write the SQL command to select all records from the Students table
    select_query = "SELECT * FROM Students;"

    # Use pandas to read SQL query directly into a DataFrame
    df = pd.read_sql_query(select_query, connection)

# Display the DataFrame
print("All Students as DataFrame:")
print(df)

Output:

All Students as DataFrame:
   id                  name  age                        email
0   1              Jane Doe   23             jane@example.com
1   2  Bahadurjit Sabharwal   18  tristanupadhyay@example.net
2   3           Zayyan Arya   20  yashawinibhakta@example.org
3   4         Hemani Shukla   18    gaurikanarula@example.com
4   5            Warda Kara   20           npatil@example.net
5   6       Mitali Nazareth   19          sparekh@example.org

The pd.read_sql_query() function executes the SQL query and directly returns the results as a pandas DataFrame.

How to Update and Delete Data

In this section, we’ll learn how to update existing records and delete records from our Students table using SQL commands in Python. This is essential for managing and maintaining your data effectively.

Updating Existing Records

To modify existing records in a database, we use the SQL UPDATE command. This command allows us to change the values of specific columns in one or more rows based on a specified condition.

For example, if we want to update a student's age, the SQL command would look like this:

UPDATE Students 
SET age = 21 
WHERE name = 'Jane Doe';

Now, let’s write Python code to update a specific student's age in our Students table.

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # SQL command to update a student's age
    update_query = '''
    UPDATE Students 
    SET age = ? 
    WHERE name = ?;
    '''

    # Data for the update
    new_age = 21
    student_name = 'Jane Doe'

    # Execute the SQL command with the data
    cursor.execute(update_query, (new_age, student_name))

    # Commit the changes to save the update
    connection.commit()

    # Print a confirmation message
    print(f"Updated age for {student_name} to {new_age}.")

In this example, we used parameterized queries to prevent SQL injection.

How to Delete Records from the Table

To remove records from a database, we use the SQL DELETE command. This command allows us to delete one or more rows based on a specified condition.

For example, if we want to delete a student named 'Jane Doe', the SQL command would look like this:

DELETE FROM Students 
WHERE name = 'Jane Doe';

Let’s write Python code to delete a specific student from our Students table using the with statement.

import sqlite3

# Use 'with' to connect to the SQLite database
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # SQL command to delete a student
    delete_query = '''
    DELETE FROM Students 
    WHERE name = ?;
    '''

    # Name of the student to be deleted
    student_name = 'Jane Doe'

    # Execute the SQL command with the data
    cursor.execute(delete_query, (student_name,))

    # Commit the changes to save the deletion
    connection.commit()

    # Print a confirmation message
    print(f"Deleted student record for {student_name}.")

Important Considerations

• Conditions: Always use the WHERE clause when updating or deleting records to avoid modifying or removing all rows in the table. Without a WHERE clause, the command affects every row in the table.

  

• Backup: It’s good practice to back up your database before performing updates or deletions, especially in production environments.

How to Use Transactions

A transaction is a sequence of one or more SQL operations that are treated as a single unit of work. In the context of a database, a transaction allows you to perform multiple operations that either all succeed or none at all. This ensures that your database remains in a consistent state, even in the face of errors or unexpected issues.

For example, if you are transferring money between two bank accounts, you would want both the debit from one account and the credit to the other to succeed or fail together. If one operation fails, the other should not be executed to maintain consistency.

Why Use Transactions?

1. Atomicity: Transactions ensure that a series of operations are treated as a single unit. If one operation fails, none of the operations will be applied to the database.

2. Consistency: Transactions help maintain the integrity of the database by ensuring that all rules and constraints are followed.

3. Isolation: Each transaction operates independently of others, preventing unintended interference.

4. Durability: Once a transaction is committed, the changes are permanent, even in the event of a system failure.

When to Use Transactions?

You should use transactions when:

• Performing multiple related operations that must succeed or fail together.

• Modifying critical data that requires consistency and integrity.

• Working with operations that can potentially fail, such as financial transactions or data migrations.

How to Manage Transactions in Python

In SQLite, transactions are managed using the BEGIN, COMMIT, and ROLLBACK commands. However, when using the sqlite3 module in Python, you typically manage transactions through the connection object.

Starting a Transaction

A transaction begins implicitly when you execute any SQL statement. To start a transaction explicitly, you can use the BEGIN command:

cursor.execute("BEGIN;")

However, it’s usually unnecessary to start a transaction manually, as SQLite starts a transaction automatically when you execute an SQL statement.

How to Commit a Transaction

To save all changes made during a transaction, you use the commit() method. This makes all modifications permanent in the database.

connection.commit()

We have already used the commit() method in the above provided examples.

Rolling Back a Transaction

If something goes wrong and you want to revert the changes made during a transaction, you can use the rollback() method. This will undo all changes made since the transaction started.

connection.rollback()

Example of Using Transactions in Python

To illustrate the use of transactions in a real-world scenario, we’ll create a new table called Customers to manage customer accounts. In this example, we’ll assume each customer has a balance. We will add two customers to this table and perform a funds transfer operation between them.

First, let's create the Customers table and insert two customers:

import sqlite3

# Create the Customers table and add two customers
with sqlite3.connect('my_database.db') as connection:
    cursor = connection.cursor()

    # Create Customers table
    create_customers_table = '''
    CREATE TABLE IF NOT EXISTS Customers (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL UNIQUE,
        balance REAL NOT NULL
    );
    '''
    cursor.execute(create_customers_table)

    # Insert two customers
    cursor.execute(
        "INSERT INTO Customers (name, balance) VALUES (?, ?);", ('Ashutosh', 100.0))
    cursor.execute(
        "INSERT INTO Customers (name, balance) VALUES (?, ?);", ('Krishna', 50.0))

    connection.commit()

Now, let’s perform the funds transfer operation between Ashutosh and Krishna:

import sqlite3


def transfer_funds(from_customer, to_customer, amount):
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        try:
            # Start a transaction
            cursor.execute("BEGIN;")

            # Deduct amount from the sender
            cursor.execute(
                "UPDATE Customers SET balance = balance - ? WHERE name = ?;", (amount, from_customer))
            # Add amount to the receiver
            cursor.execute(
                "UPDATE Customers SET balance = balance + ? WHERE name = ?;", (amount, to_customer))

            # Commit the changes
            connection.commit()
            print(
                f"Transferred {amount} from {from_customer} to {to_customer}.")

        except Exception as e:
            # If an error occurs, rollback the transaction
            connection.rollback()
            print(f"Transaction failed: {e}")


# Example usage
transfer_funds('Ashutosh', 'Krishna', 80.0)

In this example, we first created a Customers table and inserted two customers, Ashutosh with a balance of ₹100, and Krishna with a balance of ₹50. We then performed a funds transfer of ₹80 from Ashutosh to Krishna. By using transactions, we ensure that both the debit from Ashutosh's account and the credit to Krishna's account are executed as a single atomic operation, maintaining data integrity in the event of any errors. If the transfer fails (for example, due to insufficient funds), the transaction will roll back, leaving both accounts unchanged.

How to Optimize SQLite Query Performance with Indexing

Indexing is a powerful technique used in databases to improve query performance. An index is essentially a data structure that stores the location of rows based on specific column values, much like an index at the back of a book helps you quickly locate a topic.

Without an index, SQLite has to scan the entire table row by row to find the relevant data, which becomes inefficient as the dataset grows. By using an index, SQLite can jump directly to the rows you need, significantly speeding up query execution.

How to Populate the Database with Fake Data

To effectively test the impact of indexing, we need a sizable dataset. Instead of manually adding records, we can use the faker library to quickly generate fake data. In this section, we’ll generate 10,000 fake records and insert them into our Students table. This will simulate a real-world scenario where databases grow large, and query performance becomes important.

We will use the executemany() method to insert the records as below:

import sqlite3
from faker import Faker

# Initialize the Faker library
fake = Faker(['en_IN'])


def insert_fake_students(num_records):
    """Generate and insert fake student data into the Students table."""
    fake_data = [(fake.name(), fake.random_int(min=18, max=25),
                  fake.email()) for _ in range(num_records)]

    # Use 'with' to handle the database connection
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Insert fake data into the Students table
        cursor.executemany('''
        INSERT INTO Students (name, age, email) 
        VALUES (?, ?, ?);
        ''', fake_data)

        connection.commit()

    print(f"{num_records} fake student records inserted successfully.")


# Insert 10,000 fake records into the Students table
insert_fake_students(10000)

By running this script, 10,000 fake student records will be added to the Students table. In the next section, we'll query the database and compare the performance of queries with and without indexing.

How to Query Without Indexes

In this section, we’ll query the Students table without any indexes to observe how SQLite performs when there are no optimizations in place. This will serve as a baseline to compare the performance when we add indexes later.

Without indexes, SQLite performs a full table scan, which means that it must check every row in the table to find matching results. For small datasets, this is manageable, but as the number of records grows, the time taken to search increases dramatically. Let’s see this in action by running a basic SELECT query to search for a specific student by name and measure how long it takes.

First, we’ll query the Students table by looking for a student with a specific name. We’ll log the time taken to execute the query using Python’s time module to measure the performance.

import sqlite3
import time


def query_without_index(search_name):
    """Query the Students table by name without an index and measure the time taken."""

    # Connect to the database using 'with'
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Measure the start time
        start_time = time.perf_counter_ns()

        # Perform a SELECT query to find a student by name
        cursor.execute('''
        SELECT * FROM Students WHERE name = ?;
        ''', (search_name,))

        # Fetch all results (there should be only one or a few in practice)
        results = cursor.fetchall()

        # Measure the end time
        end_time = time.perf_counter_ns()

        # Calculate the total time taken
        elapsed_time = (end_time - start_time) / 1000

        # Display the results and the time taken
        print(f"Query completed in {elapsed_time:.5f} microseconds.")
        print("Results:", results)


# Example: Searching for a student by name
query_without_index('Ojasvi Dhawan')

Here’s the output:

Query completed in 1578.10000 microseconds.
Results: [(104, 'Ojasvi Dhawan', 21, 'lavanya26@example.com')]

By running the above script, you'll see how long it takes to search the Students table without any indexes. For example, if there are 10,000 records in the table, the query might take 1000-2000 microseconds depending on the size of the table and your hardware. This may not seem too slow for a small dataset, but the performance will degrade as more records are added.

We use time.perf_counter_ns() to measure the time taken for the query execution in nanoseconds. This method is highly accurate for benchmarking small time intervals. We convert the time to microseconds(us) for easier readability.

Introducing the Query Plan

When working with databases, understanding how queries are executed can help you identify performance bottlenecks and optimize your code. SQLite provides a helpful tool for this called EXPLAIN QUERY PLAN, which allows you to analyze the steps SQLite takes to retrieve data.

In this section, we’ll introduce how to use EXPLAIN QUERY PLAN to visualize and understand the inner workings of a query—specifically, how SQLite performs a full table scan when no index is present.

Let’s use EXPLAIN QUERY PLAN to see how SQLite retrieves data from the Students table without any indexes. We’ll search for a student by name, and the query plan will reveal the steps SQLite takes to find the matching rows.

import sqlite3


def explain_query(search_name):
    """Explain the query execution plan for a SELECT query without an index."""

    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Use EXPLAIN QUERY PLAN to analyze how the query is executed
        cursor.execute('''
        EXPLAIN QUERY PLAN
        SELECT * FROM Students WHERE name = ?;
        ''', (search_name,))

        # Fetch and display the query plan
        query_plan = cursor.fetchall()

        print("Query Plan:")
        for step in query_plan:
            print(step)


# Example: Analyzing the query plan for searching by name
explain_query('Ojasvi Dhawan')

When you run this code, SQLite will return a breakdown of how it plans to execute the query. Here’s an example of what the output might look like:

Query Plan:
(2, 0, 0, 'SCAN Students')

This indicates that SQLite is scanning the entire Students table (a full table scan) to find the rows where the name column matches the provided value (Ojasvi Dhawan). Since there is no index on the name column, SQLite must examine each row in the table.

How to Create an Index

Creating an index on a column allows SQLite to find rows more quickly during query operations. Instead of scanning the entire table, SQLite can use the index to jump directly to the relevant rows, significantly speeding up queries—especially those involving large datasets.

To create an index, use the following SQL command:

CREATE INDEX IF NOT EXISTS index-name ON table (column(s));

In this example, we will create an index on the name column of the Students table. Here’s how you can do it using Python:

import sqlite3
import time


def create_index():
    """Create an index on the name column of the Students table."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # SQL command to create an index on the name column
        create_index_query = '''
        CREATE INDEX IF NOT EXISTS idx_name ON Students (name);
        '''

        # Measure the start time
        start_time = time.perf_counter_ns()

        # Execute the SQL command to create the index
        cursor.execute(create_index_query)

        # Measure the start time
        end_time = time.perf_counter_ns()

        # Commit the changes
        connection.commit()

        print("Index on 'name' column created successfully!")

        # Calculate the total time taken
        elapsed_time = (end_time - start_time) / 1000

        # Display the results and the time taken
        print(f"Query completed in {elapsed_time:.5f} microseconds.")


# Call the function to create the index
create_index()

Output:

Index on 'name' column created successfully!
Query completed in 102768.60000 microseconds.

Even though creating the index takes this long (102768.6 microseconds), it's a one-time operation. You will still get substantial speed-up when running multiple queries. In the following sections, we will query the database again to observe the performance improvements made possible by this index.

How to Query with Indexes

In this section, we will perform the same SELECT query we executed earlier, but this time we will take advantage of the index we created on the name column of the Students table. We'll measure and log the execution time to observe the performance improvements provided by the index.

import sqlite3
import time


def query_with_index(student_name):
    """Query the Students table using an index on the name column."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # SQL command to select a student by name
        select_query = 'SELECT * FROM Students WHERE name = ?;'

        # Measure the execution time
        start_time = time.perf_counter_ns()  # Start the timer

        # Execute the query with the provided student name
        cursor.execute(select_query, (student_name,))
        result = cursor.fetchall()  # Fetch all results

        end_time = time.perf_counter_ns()  # End the timer

        # Calculate the elapsed time in microseconds
        execution_time = (end_time - start_time) / 1000

        # Display results and execution time
        print(f"Query result: {result}")
        print(f"Execution time with index: {execution_time:.5f} microseconds")


# Example: Searching for a student by name
query_with_index('Ojasvi Dhawan')

Here’s what we get in the output:

Query result: [(104, 'Ojasvi Dhawan', 21, 'lavanya26@example.com')]
Execution time with index: 390.70000 microseconds

We can observe a significant reduction in execution time compared to when the query was performed without an index.

Let’s analyze the query execution plan for the query with the index on the name column of the Students table. If you execute the same script again to explain the query, you’ll get the below output:

Query Plan:
(3, 0, 0, 'SEARCH Students USING INDEX idx_name (name=?)')

The plan now shows that the query uses the index idx_name, significantly reducing the number of rows that need to be scanned, which leads to faster query execution.

Comparing Performance Results

Now, let's summarize the performance results we obtained when querying with and without indexes.

Execution Time Comparison

Performance Improvement Summary

• The query with the index is approximately 4.04 times faster than the query without the index.

• The execution time improved by about 75.24% after adding the index.

Best Practices for Using Indexes

Indexes can significantly enhance the performance of your SQLite database, but they should be used judiciously. Here are some best practices to consider when working with indexes:

When and Why to Use Indexes

1. Frequent Query Columns: Use indexes on columns that are frequently used in SELECT queries, especially those used in WHERE, JOIN, and ORDER BY clauses. This is because indexing these columns can drastically reduce query execution time.

2. Uniqueness Constraints: When you have columns that must hold unique values (like usernames or email addresses), creating an index can enforce this constraint efficiently.

3. Large Datasets: For tables with a large number of records, indexes become increasingly beneficial. They enable quick lookups, which is essential for maintaining performance as your data grows.

4. Composite Indexes: Consider creating composite indexes for queries that filter or sort by multiple columns. For example, if you often search for students by both name and age, an index on both columns can optimize such queries.

Potential Downsides of Indexes

While indexes provide significant advantages, there are some potential downsides:

1. Slower Insert/Update Operations: When you insert or update records in a table with indexes, SQLite must also update the index, which can slow down these operations. This is because each insert or update requires additional overhead to maintain the index structure.

2. Increased Storage Requirements: Indexes consume additional disk space. For large tables, the storage cost can be substantial. Consider this when designing your database schema, especially for systems with limited storage resources.

3. Complex Index Management: Having too many indexes can complicate database management. It may lead to situations where you have redundant indexes, which can degrade performance rather than enhance it. Regularly reviewing and optimizing your indexes is a good practice.

Indexes are powerful tools for optimizing database queries, but they require careful consideration. Striking a balance between improved read performance and the potential overhead on write operations is key. Here are some strategies for achieving this balance:

• Monitor Query Performance: Use SQLite’s EXPLAIN QUERY PLAN to analyze how your queries perform with and without indexes. This can help identify which indexes are beneficial and which may be unnecessary.

• Regular Maintenance: Periodically review your indexes and assess whether they are still needed. Remove redundant or rarely used indexes to streamline your database operations.

• Test and Evaluate: Before implementing indexes in a production environment, conduct thorough testing to understand their impact on both read and write operations.

By following these best practices, you can leverage the benefits of indexing while minimizing potential drawbacks, ultimately enhancing the performance and efficiency of your SQLite database.

How to Handle Errors and Exceptions

In this section, we’ll discuss how to handle errors and exceptions when working with SQLite in Python. Proper error handling is crucial for maintaining the integrity of your database and ensuring that your application behaves predictably.

Common Errors in SQLite Operations

When interacting with an SQLite database, several common errors may arise:

1. Constraint Violations: This occurs when you try to insert or update data that violates a database constraint, such as primary key uniqueness or foreign key constraints. For example, trying to insert a duplicate primary key will trigger an error.

2. Data Type Mismatches: Attempting to insert data of the wrong type (for example, inserting a string where a number is expected) can lead to an error.

3. Database Locked Errors: If a database is being written to by another process or connection, trying to access it can result in a "database is locked" error.

4. Syntax Errors: Mistakes in your SQL syntax will result in errors when you try to execute your commands.

How to Use Python's Exception Handling

Python’s built-in exception handling mechanisms (try and except) are essential for managing errors in SQLite operations. By using these constructs, you can catch exceptions and respond appropriately without crashing your program.

Here’s a basic example of how to handle errors when inserting data into the database:

import sqlite3


def add_customer_with_error_handling(name, balance):
    """Add a new customer with error handling."""
    try:
        with sqlite3.connect('my_database.db') as connection:
            cursor = connection.cursor()
            cursor.execute(
                "INSERT INTO Customers (name, balance) VALUES (?, ?);", (name, balance))
            connection.commit()
            print(f"Added customer: {name} with balance: {balance}")

    except sqlite3.IntegrityError as e:
        print(f"Error: Integrity constraint violated - {e}")

    except sqlite3.OperationalError as e:
        print(f"Error: Operational issue - {e}")

    except Exception as e:
        print(f"An unexpected error occurred: {e}")


# Example usage
add_customer_with_error_handling('Vishakha', 100.0)  # Valid
add_customer_with_error_handling('Vishakha', 150.0)  # Duplicate entry

In this example:

• We catch IntegrityError, which is raised for violations like unique constraints.

• We catch OperationalError for general database-related issues (like database locked errors).

• We also have a generic except block to handle any unexpected exceptions.

Output:

Added customer: Vishakha with balance: 100.0
Error: Integrity constraint violated - UNIQUE constraint failed: Customers.name

Best Practices for Ensuring Database Integrity

1. Use Transactions: Always use transactions (as discussed in the previous section) when performing multiple related operations. This helps ensure that either all operations succeed or none do, maintaining consistency.

2. Validate Input Data: Before executing SQL commands, validate the input data to ensure it meets the expected criteria (for example, correct types, within allowable ranges).

3. Catch Specific Exceptions: Always catch specific exceptions to handle different types of errors appropriately. This allows for clearer error handling and debugging.

4. Log Errors: Instead of just printing errors to the console, consider logging them to a file or monitoring system. This will help you track issues in production.

5. Graceful Degradation: Design your application to handle errors gracefully. If an operation fails, provide meaningful feedback to the user rather than crashing the application.

6. Regularly Backup Data: Regularly back up your database to prevent data loss in case of critical failures or corruption.

7. Use Prepared Statements: Prepared statements help prevent SQL injection attacks and can also provide better performance for repeated queries.

How to Export and Import Data [Bonus Section]

In this section, we will learn how to export data from an SQLite database to common formats like CSV and JSON, as well as how to import data into SQLite from these formats using Python. This is useful for data sharing, backup, and integration with other applications.

Exporting Data from SQLite to CSV

Exporting data to a CSV (Comma-Separated Values) file is straightforward with Python’s built-in libraries. CSV files are widely used for data storage and exchange, making them a convenient format for exporting data.

Here’s how to export data from an SQLite table to a CSV file:

import sqlite3
import csv

def export_to_csv(file_name):
    """Export data from the Customers table to a CSV file."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Execute a query to fetch all customer data
        cursor.execute("SELECT * FROM Customers;")
        customers = cursor.fetchall()

        # Write data to CSV
        with open(file_name, 'w', newline='') as csv_file:
            csv_writer = csv.writer(csv_file)
            csv_writer.writerow(['ID', 'Name', 'Balance'])  # Writing header
            csv_writer.writerows(customers)  # Writing data rows

        print(f"Data exported successfully to {file_name}.")

# Example usage
export_to_csv('customers.csv')

How to Export Data to JSON

Similarly, you can export data to a JSON (JavaScript Object Notation) file, which is a popular format for data interchange, especially in web applications.

Here’s an example of how to export data to JSON:

import json
import sqlite3


def export_to_json(file_name):
    """Export data from the Customers table to a JSON file."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Execute a query to fetch all customer data
        cursor.execute("SELECT * FROM Customers;")
        customers = cursor.fetchall()

        # Convert data to a list of dictionaries
        customers_list = [{'ID': customer[0], 'Name': customer[1],
                           'Balance': customer[2]} for customer in customers]

        # Write data to JSON
        with open(file_name, 'w') as json_file:
            json.dump(customers_list, json_file, indent=4)

        print(f"Data exported successfully to {file_name}.")


# Example usage
export_to_json('customers.json')

How to Import Data into SQLite from CSV

You can also import data from a CSV file into an SQLite database. This is useful for populating your database with existing datasets.

Here's how to import data from a CSV file:

import csv
import sqlite3


def import_from_csv(file_name):
    """Import data from a CSV file into the Customers table."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Open the CSV file for reading
        with open(file_name, 'r') as csv_file:
            csv_reader = csv.reader(csv_file)
            next(csv_reader)  # Skip the header row

            # Insert each row into the Customers table
            for row in csv_reader:
                cursor.execute(
                    "INSERT INTO Customers (name, balance) VALUES (?, ?);", (row[1], row[2]))

        connection.commit()
        print(f"Data imported successfully from {file_name}.")


# Example usage
import_from_csv('customer_data.csv')

How to Import Data into SQLite from JSON

Similarly, importing data from a JSON file is simple. You can read the JSON file and insert the data into your SQLite table.

Here's how to do it:

import json
import sqlite3


def import_from_json(file_name):
    """Import data from a JSON file into the Customers table."""
    with sqlite3.connect('my_database.db') as connection:
        cursor = connection.cursor()

        # Open the JSON file for reading
        with open(file_name, 'r') as json_file:
            customers_list = json.load(json_file)

            # Insert each customer into the Customers table
            for customer in customers_list:
                cursor.execute("INSERT INTO Customers (name, balance) VALUES (?, ?);", (customer['Name'], customer['Balance']))

        connection.commit()
        print(f"Data imported successfully from {file_name}.")


# Example usage
import_from_json('customer_data.json')

Wrapping Up

And that’s a wrap! This guide has introduced you to the fundamentals of working with SQLite in Python, covering everything from setting up your environment to querying and manipulating data, as well as exporting and importing information. I hope you found it helpful and that it has sparked your interest in using SQLite for your projects.

Now it's time to put your newfound knowledge into practice! I encourage you to create your project using SQLite and Python. Whether it’s a simple application for managing your library, a budgeting tool, or something unique, the possibilities are endless.

Once you’ve completed your project, share it on Twitter and tag me! I’d love to see what you’ve created and celebrate your accomplishments.

You can find all the code from this tutorial on GitHub. Thank you for following along, and happy coding!

Generate Table of Contents for your freeCodeCamp articles for free using the TOC Generator tool.

Files are the backbone of data storage. Node.js provides a powerful 'fs' (file system) module to interact with these files seamlessly. Let's assume I want to read a JSON file in Node.js. The fs module can help me with that.

In this tutorial, I'll explain the core functionalities of this module, explore various techniques to read different file types, and discover some best practices to make your file-handling operations smoother and more efficient.

Throughout this tutorial, we'll cover everything from importing the package to using it to work with files asynchronously. Let's get started on this journey of learning file operations with Node.js!

Table of Contents

• Node.js fs Module

• Prerequisites

• How to Read And Write Files With Node.js

• Ways to Read Files in Node.js

• Wrapping Up

Node.js fs Module

The Node.js File System (fs) module is an essential component of the Node.js runtime environment. It provides a variety of features for interacting with your computer's file system.

The fs module allows you to read, write, update, delete, and manage files and directories. This module is especially useful for handling file-related operations in both synchronous and asynchronous modes.

Let’s break down the key aspects of the module:

1. The fs module, at its core, provides a collection of APIs for interacting with the file system. It provides ways to perform basic activities such as reading file contents, writing data to files, creating directories, deleting files, and so on.

2. The module includes both synchronous and asynchronous methods for interacting with files. The synchronous methods block the execution of the program until the operation completes. But the asynchronous methods are ideal for scenarios where we need to perform concurrent tasks without halting the execution of the entire program.

3. The module also supports handling directories, such as creating directories, removing directories, and listing directory contents.

4. The module also supports working with file streams, allowing efficient handling of large files by reading or writing data in chunks without loading the entire content into memory. It also facilitates the use of buffers for handling binary data, which helps with activities like data transformation and manipulation.

Prerequisites

To continue with the tutorial, I recommend having the following prerequisites:

1. Basic Understanding of JavaScript: It is essential to be familiar with JavaScript, as Node.js uses JavaScript.

2. Node.js Installed: Ensure that Node.js is installed on your system. You can download and install Node.js from its official website.

3. Text Editor/IDE: Have a text editor or an Integrated Development Environment (IDE) installed and ready to use.

How to Read And Write Files With Node.js

Let's look at an example to understand the process of reading and writing files in Node.js. We'll assume a scenario where we have two files – name.json and address.json.

The content inside the name.json looks like this:

[
  { "id": 1, "name": "Alice" },
  { "id": 2, "name": "Bob" },
  { "id": 3, "name": "Charlie" }
]

The content inside address.json looks like this:

[
  { "id": 1, "address": "123 Main St" },
  { "id": 2, "address": "456 Elm St" },
  { "id": 3, "address": "789 Oak St" }
]

Our objective is to create a bio.json file that merges the id, name, and address information, creating a structure as follows:

[
  {
    "id": 1,
    "name": "Alice",
    "address": "123 Main St"
  },
  {
    "id": 2,
    "name": "Bob",
    "address": "456 Elm St"
  },
  {
    "id": 3,
    "name": "Charlie",
    "address": "789 Oak St"
  }
]

Let's build the application!

Step 1: Import Node.js Packages path and fs

Let us start with creating an app.js file. The first thing we will do is import the fs library:

const fs = require("fs");

Step 2: Read From Files

Next, let's read data from the two files using Node.js. We'll make a utility function that helps us read these files easily in our Node.js environment.

async function readJSONFile(filename) {
  try {
    const data = await fs.readFile(filename, "utf8");
    return JSON.parse(data);
  } catch (error) {
    console.error(`Error reading ${filename}: ${error}`);
    return [];
  }
}

Since we are using JSON files in our example, we have defined a readJSONFile method in our code. It's an asynchronous JavaScript function that takes a filename as input and aims to return the parsed JSON contents of that file.

Inside a try block, we attempt to read the file using fs.readFile in Node with the specified filename and "utf8" encoding. If successful, the function then parses the file content as JSON using JSON.parse and returns it.

If any error occurs during reading or parsing, the catch block takes over. It logs the error with the filename and details and then returns an empty array instead of the expected JSON object.

Step 3: Implement the Main Function

The next step is to create a main function where we make use of the above-defined method and combine the data of the two files to create a bio.json file.

async function main() {
  try {
    const names = await readJSONFile("names.json");
    const addresses = await readJSONFile("address.json");

    const bioData = names.map((name) => {
      const matchingAddress = addresses.find(
        (address) => address.id === name.id
      );
      return { ...name, ...matchingAddress };
    });

    await fs.writeFile("bio.json", JSON.stringify(bioData, null, 2));
    console.log("bio.json created successfully!");
  } catch (error) {
    console.error("Error combining data:", error);
  }
}

In the main function, we first read the two JSON files named names.json and address.json using the readJSONFile function. Both readJSONFile calls use await, so the function waits for both files to be read before moving on.

Next, we iterate through each name using a map, creating a new bioData for each. Inside the loop, it searches for a matching address from the addresses collection based on the index using find.

The search compares name.id with each address.id until it finds a match. If a match is found, the function combines the information from both files. It uses the spread operator (...) to merge all properties from both objects into a single new bioData object. If no matching address is found, the bioData object will only have the name information.

Once all bioData objects are prepared, the function writes them as a new JSON file named bio.json using fs.writeFile. This writing process also uses await, ensuring the file is created before moving on.

The try block ensures smooth execution, while the catch block takes care of any errors like missing files or incorrect data. If any error occurs, a generic error message and the specific error details are logged for debugging.

Complete Code

Our completed code looks like below:

const fs = require("fs").promises;

async function readJSONFile(filename) {
  try {
    const data = await fs.readFile(filename, "utf8");
    return JSON.parse(data);
  } catch (error) {
    console.error(`Error reading ${filename}: ${error}`);
    return [];
  }
}

async function main() {
  try {
    const names = await readJSONFile("names.json");
    const addresses = await readJSONFile("address.json");

    const bioData = names.map((name) => {
      const matchingAddress = addresses.find(
        (address) => address.id === name.id
      );
      return { ...name, ...matchingAddress };
    });

    await fs.writeFile("bio.json", JSON.stringify(bioData, null, 2));
    console.log("bio.json created successfully!");
  } catch (error) {
    console.error("Error combining data:", error);
  }
}

// Execute the main method
main();

We can run the application using the following command:

node app.js

Once the application runs, we get to see the following logs in the terminal if everything goes well:

bio.json created successfully!

This indicates that the bio.json file was created successfully. The content inside the file should look like below:

[
  {
    "id": 1,
    "name": "Alice",
    "address": "123 Main St"
  },
  {
    "id": 2,
    "name": "Bob",
    "address": "456 Elm St"
  },
  {
    "id": 3,
    "name": "Charlie",
    "address": "789 Oak St"
  }
]

Ways to Read Files in Node.js

To read files in Node.js, we mostly use two primary methods: fs.readFile() and fs.readFileSync(). The difference lies in their synchronous and asynchronous nature.

fs.readFile() Method

The fs.readFile() method in Node.js is asynchronous. It reads the content of the entire file without blocking other operations. This makes it suitable for scenarios where non-blocking I/O operations are essential.

In simple terms, the function allows other operations to continue while the reading takes place. It accepts the following three parameters:

1. path: the path to the file to be read.

2. encoding: optional, specifies the encoding of the file (for example, "utf8"). Defaults to "utf8" if not provided.

3. callback function: optional, a function to be called when the file is read. The function receives three arguments: error, data, and buffer.

Let’s look at an example:

const fs = require("fs");

fs.readFile("data.txt", "utf8", (err, data) => {
  if (err) {
    console.error(err);
  } else {
    console.log(data); // data will be a string containing the content of the file
  }
});

fs.readFileSync() Method

The fs.readFileSync() method is synchronous. It reads the file content synchronously, stopping further execution until the file is completely read. This method is beneficial in scenarios where we require synchronous processing.

It takes only two parameters:

1. path: the path to the file to be read

2. encoding: optional, specifies the encoding of the file (for example, "utf8"). Defaults to "utf8" if not provided.

Let’s look at an example:

const fs = require("fs");

try {
  const data = fs.readFileSync("data.txt", "utf8");
  console.log(data); // data will be a string containing the content of the file
} catch (err) {
  console.error(err);
}

fs.readFile() vs fs.readFileSync()

To understand the difference between the Nodejs readFile and readFileSync methods, we will write two programs and monitor their execution flow.

Let's start with the fs.readFile() method.

const fs = require("fs");

fs.readFile("example.txt", "utf8", (err, data) => {
  console.log("Content from readFile:", data);
});

console.log("Completed reading file content asynchronously");

Output:

Completed reading file content asynchronously
Content from readFile: freeCodeCamp is awesome!

Because of fs.readFile()'s asynchronous nature, the code after fs.readFile() doesn't wait for the file reading operation to finish. So the "Completed reading file content synchronously" message is immediately logged to the console, showing that the subsequent code continues executing without waiting for the file read to complete.

Eventually, when the file reading operation finishes, the callback function executes and logs the file content.

Next, let's see the execution of the fs.readFileSync() method.

const fs = require("fs");

const data = fs.readFileSync("data.txt", "utf8");
console.log("Content from readFileSync:", data);

console.log("Completed reading file content synchronously");

Output:

Content from readFileSync: freeCodeCamp is awesome!
Completed reading file content synchronously

In contrast, fs.readFileSync() reads the data.txt file synchronously, blocking further code execution until the file is completely read. Consequently, the code continues execution only after the file reading operation is finished.

Because of this, the message "Completed reading file content synchronously" is logged after successfully reading the file content.

Now, we know the difference between the two methods. Understanding this difference is important as it impacts the program flow, especially in scenarios where timing and blocking operations are critical considerations in Node.js applications.

How to Read a Text File

It's pretty straightforward to read a text file in Node.js, and we have been doing this throughout the tutorial. Let’s consider that we have a file called message.txt with the following content:

Learn Node.js with freeCodeCamp

Now, we want to read the contents of this file. We can do it like this:

const fs = require("fs");

fs.readFile("message.txt", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    console.log(data);
  }
});

The callback function returns the content of the file in a data variable. Since we set the encoding to “utf8”, the value of data is a string. So we can perform string operations on the data variable.

const fs = require("fs");

fs.readFile("message.txt", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
      let splittedWords = data.split(" ");
      console.log(splittedWords);
  }
});

In the above code, we split the data variable using a space. So the splittedWords will be a string array containing the following value:

[ 'Learn', 'Node.js', 'with', 'freeCodeCamp' ]

How to Read HTML Files

Reading HTML files follows a similar approach to reading text files in Node.js. We can use the fs module to read HTML files:

const fs = require("fs");

fs.readFile("index.html", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    console.log(data);
  }
});

We can then use the HTML content for further processing, such as rendering it with the Node.js http package.

How to Read Files by URL

Reading files by URL in Node.js involves additional steps beyond the native fs module. Typically, we’ll need to use additional modules like http or axios to fetch file content from a URL.

const fs = require("fs");
const https = require("https");

const file = fs.createWriteStream("data.txt");

https.get(
  "https://example-files.online-convert.com/document/txt/example.txt",
  (response) => {
    var stream = response.pipe(file);

    stream.on("finish", function () {
      console.log("done");
    });
  }
);

First, we set up a writable stream named file, associated with the local file data.txt. We then use Node.js's https module to perform an HTTP GET request to the specified URL. The get method triggers a callback function when we receive a response from the server.

Inside the callback, the script pipes the response directly into the writable stream. This operation efficiently directs the data received from the remote server to the local file data.txt, essentially downloading and writing the content concurrently.

Finally, we set up an event listener for the "finish" event on the stream. This event fires when all the data has been successfully written to the file. Upon completion, the script logs "done" to the console, indicating the successful download and writing of the file.

How to Read a JSON File

We have already seen how we can read a JSON file using the fs module. Let's say that we want to read our previously created bio.json file. Its data looks like the below:

[
  {
    "id": 1,
    "name": "Alice",
    "address": "123 Main St"
  },
  {
    "id": 2,
    "name": "Bob",
    "address": "456 Elm St"
  },
  {
    "id": 3,
    "name": "Charlie",
    "address": "789 Oak St"
  }
]

In Node.js, we read JSON like this:

const fs = require("fs");

fs.readFile("bio.json", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    console.log(data);
  }
});

With this, our JSON data is stored within the data variable as a string. If we want, we can use it for further processing. Let’s say, we want to print the details of the user:

const fs = require("fs");

fs.readFile("bio.json", "utf8", (err, data) => {
  if (err) {
    console.log(err);
  } else {
    const users = JSON.parse(data);
    users.forEach((user) => {
      console.log(`${user.name} with ID ${user.id} lives at ${user.address}`);
    });
  }
});

Output:

Alice with ID 1 lives at 123 Main St
Bob with ID 2 lives at 456 Elm St
Charlie with ID 3 lives at 789 Oak St

In the above code, we first parse the string data variable to JSON and store it in a users variable. We then loop over the users variable to log the required message.

fs.promises

fs.promises provides a set of asynchronous functions for interacting with the file system in Node.js. These functions are based on promises and offer a more readable and efficient way to handle asynchronous operations compared to callbacks.

With fs.promises, we don't need to add nested callbacks – which means that we can avoid callback hell.

A basic readFile operation with fs.promises looks like this:

const fs = require("fs").promises;

async function readTextFile() {
  try {
    const data = await fs.readFile("data.txt", "utf8");
    console.log(data);
  } catch (err) {
    console.error(err);
  }
}

readTextFile();

Wrapping Up

In this tutorial, we've explored the essential techniques for handling files in Node.js with the help of the fs module.

From understanding synchronous and asynchronous file reading with methods like fs.readFile() and fs.readFileSync() to processing various file formats such as text, HTML, JSON, and even reading files from URLs, we've covered a bunch of functionalities. We also learned about fs.promises, which is a more elegant way to handle file operations using asynchronous functions.

Frequently Asked Questions (FAQs)

1. How do I read a file asynchronously in Node.js?

There are mainly two ways to read a file asynchronously in Node.js: using fs.readFile() and using fs.promises.

The fs.readFile() method uses callbacks to handle the operation. We provide a callback function that will be called with the file data (or an error) when the reading is complete.

But fs.promises offers a promise-based approach. We can use the readFile function with await to wait for the reading to finish and then access the data directly.

2. How can I check if a file exists before reading or writing in Node.js?

There are two methods to check if a file exist: using fs.stat and using fs.promises.access.

The fs.stat method synchronously checks if a file exists and returns information about it like size and access time. The fs.promises.access method asynchronously checks if a file exists and returns a promise that resolves or rejects based on the existence of the file.

3. How can I handle errors when reading or writing files in Node.js?

To handle errors while reading or writing files in Node.js, we can utilize error-first callbacks or promises along with try-catch blocks.

However, when it comes to sorting custom objects, such as instances of user-defined classes, the built-in sorting methods fall short. These methods don't know how to order objects based on custom criteria. This is where Java's Comparable and Comparator interfaces come into play, allowing developers to define and implement custom sorting logic tailored to specific requirements.

In this blog post, we'll explore how to use the Comparable and Comparator interfaces to sort custom objects in Java. I'll provide examples to illustrate the differences and use cases for each approach, helping you master custom sorting in your Java applications.

Table of Contents

• Sorting Methods for Primitive Types
• How to Use the Comparable Interface
• How to Use the Comparator Interface
• Comparable vs Comparator
• Wrapping Up

Sorting Methods for Primitive Types

Java provides a variety of built-in sorting methods that make it easy to sort primitive data types. These methods are highly optimized and efficient, allowing you to sort arrays and collections with minimal code. For primitive types, such as integers, floating-point numbers, and characters, the Arrays.sort() method is commonly used.

How to Use the Arrays.sort() Method

The Arrays.sort() method sorts the specified array into ascending numerical order. This method uses a dual-pivot quicksort algorithm, which is faster and more efficient for most data sets.

Let's look at an example of sorting an array of integers and characters using Arrays.sort():

package tutorial;

import java.util.Arrays;

public class PrimitiveSorting {
    public static void main(String[] args) {
        int[] numbers = { 5, 3, 8, 2, 1 };
        System.out.println("Original array: " + Arrays.toString(numbers));

        Arrays.sort(numbers);
        System.out.println("Sorted array: " + Arrays.toString(numbers));

        char[] characters = { 'o', 'i', 'e', 'u', 'a' };
        System.out.println("Original array: " + Arrays.toString(characters));

        Arrays.sort(characters);
        System.out.println("Sorted array: " + Arrays.toString(characters));
    }
}

Output:

Original array: [5, 3, 8, 2, 1]
Sorted array: [1, 2, 3, 5, 8]
Original array: [o, i, e, u, a]
Sorted array: [a, e, i, o, u]

How to Use the Collections.sort() Method

The Collections.sort() method is used to sort collections such as ArrayList. This method is also based on the natural ordering of the elements or a custom comparator.

package tutorial;

import java.util.ArrayList;
import java.util.Collections;

public class CollectionsSorting {
    public static void main(String[] args) {
        ArrayList<String> wordsList = new ArrayList<>();
        wordsList.add("banana");
        wordsList.add("apple");
        wordsList.add("cherry");
        wordsList.add("date");
        System.out.println("Original list: " + wordsList);

        Collections.sort(wordsList);
        System.out.println("Sorted list: " + wordsList);
    }
}

Output:

Original list: [banana, apple, cherry, date]
Sorted list: [apple, banana, cherry, date]

Limitations with Custom Classes

While Java's built-in sorting methods, such as Arrays.sort() and Collections.sort(), are powerful and efficient for sorting primitive types and objects with natural ordering (like String), they fall short when it comes to sorting custom objects. These methods do not inherently know how to order user-defined objects because there is no natural way for them to compare these objects.

For example, consider a simple Person class that has name, age, and weight attributes:

package tutorial;

public class Person {
    String name;
    int age;
    double weight;

    public Person(String name, int age, double weight) {
        this.name = name;
        this.age = age;
        this.weight = weight;
    }

    @Override
    public String toString() {
        return "Person [name=" + name + ", age=" + age + ", weight=" + weight + " kgs]";
    }
}

If we try to sort a list of Person objects using Arrays.sort() or Collections.sort(), we will encounter a compilation error because these methods do not know how to compare Person objects:

package tutorial;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class CustomClassSorting {
    public static void main(String[] args) {
        List<Person> people = new ArrayList<>(Arrays.asList(
                new Person("Alice", 30, 65.5),
                new Person("Bob", 25, 75.0),
                new Person("Charlie", 35, 80.0)
        ));
        System.out.println("Original people list: " + people);

        Collections.sort(people);
        System.out.println("Sorted people list: " + people);
    }
}

Compilation Error:

java: no suitable method found for sort(java.util.List<tutorial.Person>)
    method java.util.Collections.<T>sort(java.util.List<T>) is not applicable
      (inference variable T has incompatible bounds
        equality constraints: tutorial.Person
        lower bounds: java.lang.Comparable<? super T>)
    method java.util.Collections.<T>sort(java.util.List<T>,java.util.Comparator<? super T>) is not applicable
      (cannot infer type-variable(s) T
        (actual and formal argument lists differ in length))

The error occurs because the Person class does not implement the Comparable interface, and there is no way for the sorting method to know how to compare two Person objects.

To sort custom objects like Person, we need to provide a way to compare these objects. Java offers two main approaches to achieve this:

1. Implementing the Comparable Interface: This allows a class to define its natural ordering by implementing the compareTo method.
2. Using the Comparator Interface: This allows us to create separate classes or lambda expressions to define multiple ways of comparing objects.

We will explore both approaches in the upcoming sections, starting with the Comparable interface.

How to Use the Comparable Interface

Java provides a Comparable interface to define a natural ordering for objects of a user-defined class. By implementing the Comparable interface, a class can provide a single natural ordering that can be used to sort its instances. This is particularly useful when you need a default way to compare and sort objects.

Overview

The Comparable interface contains a single method, compareTo(), which compares the current object with the specified object for order. The method returns:

• A negative integer if the current object is less than the specified object.
• Zero if the current object is equal to the specified object.
• A positive integer if the current object is greater than the specified object.

How Comparable Allows for a Single Natural Ordering of Objects

By implementing the Comparable interface, a class can ensure that its objects have a natural ordering. This allows the objects to be sorted using methods like Arrays.sort() or Collections.sort() without the need for a separate comparator.

Let's implement the Comparable interface in a new PersonV2 class, comparing by age.

package tutorial;

public class PersonV2 implements Comparable<PersonV2> {
    String name;
    int age;
    double weight;

    public PersonV2(String name, int age, double weight) {
        this.name = name;
        this.age = age;
        this.weight = weight;
    }

    @Override
    public String toString() {
        return "PersonV2 [name=" + name + ", age=" + age + ", weight=" + weight + " kgs]";
    }

    @Override
    public int compareTo(PersonV2 other) {
        return this.age - other.age;
    }
}

In this implementation, the compareTo() method compares the age attribute of the current PersonV2 object with the age attribute of the specified PersonV2 object by subtracting one age from the other. By using the expression this.age - other.age, we’re effectively implementing this logic as follows:

• If this.age is less than other.age, the result will be negative.
• If this.age is equal to other.age, the result will be zero.
• If this.age is greater than other.age, the result will be positive.

Note: We can also use Integer.compare(this.age, other.age) instead of performing the arithmetic operation manually.

Now that the PersonV2 class implements the Comparable interface, we can sort a list of PersonV2 objects using Collections.sort():

package tutorial;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class CustomClassSortingV2 {
    public static void main(String[] args) {
        List<PersonV2> people = new ArrayList<>(Arrays.asList(
                new PersonV2("Alice", 30, 65.5),
                new PersonV2("Bob", 25, 75.0),
                new PersonV2("Charlie", 35, 80.0)
        ));
        System.out.println("Original people list: " + people);

        Collections.sort(people);
        System.out.println("Sorted people list: " + people);
    }
}

Output:

Original people list: [PersonV2 [name=Alice, age=30, weight=65.5 kgs], PersonV2 [name=Bob, age=25, weight=75.0 kgs], PersonV2 [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list: [PersonV2 [name=Bob, age=25, weight=75.0 kgs], PersonV2 [name=Alice, age=30, weight=65.5 kgs], PersonV2 [name=Charlie, age=35, weight=80.0 kgs]]

In this example, the PersonV2 objects are sorted in ascending order of age using the Collections.sort() method, which relies on the natural ordering defined by the compareTo() method in the PersonV2 class.

Limitations of Comparable

While the Comparable interface provides a way to define a natural ordering for objects, it has several limitations that can restrict its use in practical applications. Understanding these limitations can help us determine when to use other mechanisms, such as the Comparator interface, to achieve more flexible sorting.

• Single Natural Ordering: The primary limitation of Comparable is that it allows only one natural ordering for the objects of a class. When you implement Comparable, you define a single way to compare objects, which is used whenever the objects are sorted or compared. This can be restrictive if you need to sort objects in multiple ways.
• Inflexibility: If you need to sort objects by different attributes or in different orders, you will have to modify the class or create new implementations of Comparable. This inflexibility can lead to a proliferation of comparison methods and can make the code harder to maintain.
• Non-Adaptable: Once a class implements Comparable, the natural ordering is fixed and cannot be easily changed. For instance, if your PersonV2 class initially sorts by age but later you need to sort by weight or name, you have to either change the compareTo() method or create a new version of the class.

This is where the Comparator interface comes into play. To define multiple ways of comparing objects, we can use the Comparator interface, which we will explore in the next section.

How to Use the Comparator Interface

The Comparator interface in Java provides a way to define multiple ways of comparing and sorting objects. Unlike the Comparable interface, which allows only a single natural ordering, Comparator is designed to offer flexibility by allowing multiple sorting strategies. This makes it particularly useful for scenarios where objects need to be sorted in different ways.

Overview

The Comparator interface defines a single method, compare(), which compares two objects and returns:

• A negative integer if the first object is less than the second object.
• Zero if the first object is equal to the second object.
• A positive integer if the first object is greater than the second object.

This method provides a way to define custom ordering for objects without modifying the class itself.

How Comparator Allows for Multiple Ways of Ordering Objects

The Comparator interface allows you to create multiple Comparator instances, each defining a different ordering for objects. This flexibility means that you can sort objects by various attributes or in different orders without altering the object's class.

Let's implement multiple Comparator instances for the Person class. We'll define comparators for sorting by name, by age, and by weight. First, we need to update the Person class to include getters and ensure that attributes are accessible.

package tutorial;

public class Person {
    String name;
    int age;
    double weight;

    public Person(String name, int age, double weight) {
        this.name = name;
        this.age = age;
        this.weight = weight;
    }

    public String getName() {
        return name;
    }

    public int getAge() {
        return age;
    }

    public double getWeight() {
        return weight;
    }

    @Override
    public String toString() {
        return "Person [name=" + name + ", age=" + age + ", weight=" + weight + " kgs]";
    }
}

Comparator by Name

This comparator sorts Person objects alphabetically by their name.

package tutorial.comparator;

import tutorial.Person;

import java.util.Comparator;

public class PersonNameComparator implements Comparator<Person> {

    @Override
    public int compare(Person p1, Person p2) {
        return p1.getName().compareTo(p2.getName());
    }
}

Comparator by Age

This comparator sorts Person objects by their age, in ascending order.

package tutorial.comparator;

import tutorial.Person;

import java.util.Comparator;

public class PersonAgeComparator implements Comparator<Person> {

    @Override
    public int compare(Person p1, Person p2) {
        return p1.getAge() - p2.getAge();
    }
}

Comparator by Weight

This comparator sorts Person objects by their weight, in ascending order.

package tutorial.comparator;

import tutorial.Person;

import java.util.Comparator;

public class PersonWeightComparator implements Comparator<Person> {

    @Override
    public int compare(Person p1, Person p2) {
        return (int) (p1.getWeight() - p2.getWeight());
    }
}

Now, here’s how you can use these Comparator instances to sort a list of Person objects:

package tutorial;

import tutorial.comparator.PersonAgeComparator;
import tutorial.comparator.PersonNameComparator;
import tutorial.comparator.PersonWeightComparator;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

public class CustomClassSortingV3 {
    public static void main(String[] args) {
        List<Person> people = new ArrayList<>(Arrays.asList(
                new Person("Alice", 30, 65.5),
                new Person("Bob", 25, 75.0),
                new Person("Charlie", 35, 80.0)
        ));
        System.out.println("Original people list: " + people);

        Collections.sort(people, new PersonNameComparator());
        System.out.println("Sorted people list by name: " + people);

        Collections.sort(people, new PersonAgeComparator());
        System.out.println("Sorted people list by age: " + people);

        Collections.sort(people, new PersonWeightComparator());
        System.out.println("Sorted people list by weight: " + people);
    }
}

Output:

Original people list: [Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list by name: [Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list by age: [Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]
Sorted people list by weight: [Person [name=Alice, age=30, weight=65.5 kgs], Person [name=Bob, age=25, weight=75.0 kgs], Person [name=Charlie, age=35, weight=80.0 kgs]]

In this example, the Comparator instances allow sorting the Person objects by different attributes: name, age, and weight. This demonstrates how the Comparator interface enables flexible and versatile sorting strategies for a class.

Comparable vs Comparator

When sorting objects in Java, you have two primary options: the Comparable and Comparator interfaces. Understanding the differences between these two interfaces can help you choose the right approach for your needs. Please note that this is also a very important interview question.

Comparison

Here’s a table comparing and contrasting the Comparable and Comparator interfaces in Java:

Benefits and Drawbacks of Each Approach

Comparable Operator

Benefits:

• Simplicity: Provides a default sorting order that is easy to implement and use.
• Built-in: The natural ordering is part of the class itself, so it is always available and used by default in sorting methods.

Drawbacks:

• Single Ordering: Can only define one way to compare objects. If different sorting orders are needed, the class must be modified or additional Comparator instances must be used.
• Class Modification: Requires altering the class to implement Comparable, which might not be feasible if the class is part of a library or if its natural ordering is not clear.

Comparator

Benefits:

• Flexibility: Allows for multiple sorting orders and criteria, which can be defined externally and used as needed.
• Non-invasive: Does not require modification of the class itself, making it suitable for classes you do not control or when you need different sorting options.

Drawbacks:

• Complexity: Requires creating and managing multiple Comparator instances, which can add complexity to the code.
• Overhead: Might introduce additional overhead if many comparators are used, especially if they are created on the fly.

In summary, Comparable is best used when a class has a natural ordering that makes sense for most use cases.

Comparator, on the other hand, provides flexibility for sorting by multiple criteria and is useful when the class does not have a natural ordering or when different sorting orders are needed.

Choosing between Comparable and Comparator depends on your specific sorting needs and whether you need a single default order or multiple flexible sorting options.

Wrapping Up

Understanding and utilizing both Comparable and Comparator can significantly enhance your ability to manage and manipulate object collections in Java. By applying these concepts, you can create more flexible and powerful sorting mechanisms.

To solidify your understanding, try implementing both Comparable and Comparator in real-world scenarios. Experiment with different classes and sorting criteria to see how each approach works in practice.

Links to Official Java Documentation:

• Java Comparable Interface
• Java Comparator Interface

In this tutorial, you'll learn how to create a simple instant search feature using Flask and HTMX. This will help you build interactive web applications with better user experience.

Table of Contents:

1. How to Set Up the Environment
2. How to Set up the Database
3. How to Set Up Basic Routing and HTML
4. How to Add HTMX for Instant Search
5. Demo
6. Conclusion

Why Use Instant Search?

• Speed: Users get immediate feedback, which helps them refine their search.
• Convenience: It reduces the number of clicks and page loads, leading to a more seamless experience.
• Engagement: Users are more likely to stay on your site if they can find what they need easily.

Technologies Used

To implement this instant search feature, we'll use two main technologies:

• Flask: Flask is a popular web framework for Python. It is simple and lightweight, making it easy to set up and start building web applications quickly. Flask lets you to create routes, handle requests, and serve HTML templates with minimal setup.
• HTMX: This is a powerful JavaScript library that lets you to create dynamic web pages without having to write a lot of JavaScript code. With HTMX, you can update parts of a page based on user actions, like typing in a search box. It makes it easy to load data from the server and display it on the page without a full reload.

How to Set Up the Environment

In this section, we'll set up the environment for our Flask project, including installing the necessary packages and organizing the project structure.

1. How to Install Flask and HTMX

First, you need to install Flask, Flask-SQLAlchemy, and Flask-Migrate. You can do this using pip. Open your terminal and run:

pip install Flask Flask-SQLAlchemy Flask-Migrate

For HTMX, we'll include it in our HTML template directly from a CDN.

2. How to Create a Virtual Environment

It's a good practice to create a virtual environment for your projects to manage dependencies. Here's how to create one:

python -m venv venv

Next, activate the environment:

# On Windows
venv\Scripts\activate

# On macOS/Linux
source venv/bin/activate

3. How to Set Up the Project Structure

Now, set up your project structure as follows:

my_flask_app/
├── core/
│   ├── __init__.py
│   ├── models.py
│   └── routes.py
├── config.py
└── main.py

Let us start with creating the first file: core/init.py. This file is the initialization script for the core module of our Flask application. It sets up the Flask app instance and configures it using the settings from the DevelopmentConfig class, and initializes the database and migration system.

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from config import DevelopmentConfig

# Create the Flask app instance
app = Flask(__name__)

# Load configuration from DevelopmentConfig
app.config.from_object(DevelopmentConfig)

# Initialize SQLAlchemy with the app instance
db = SQLAlchemy(app)

# Initialize Flask-Migrate with the app instance and database
migrate = Migrate(app, db)

# Import routes to register them with the app
from core import routes

Next, we will create the config.py file from where we'll import the DevelopmentConfig class. This file contains configuration settings for different environments (development, testing, production). These settings help manage different behaviors and configurations based on where your app is running.

class Config(object):
    DEBUG = False
    TESTING = False
    CSRF_ENABLED = True
    SECRET_KEY = "guess-me"
    SQLALCHEMY_DATABASE_URI = "sqlite:///db.sqlite"
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    BCRYPT_LOG_ROUNDS = 13
    WTF_CSRF_ENABLED = True
    DEBUG_TB_ENABLED = False
    DEBUG_TB_INTERCEPT_REDIRECTS = False

class DevelopmentConfig(Config):
    DEVELOPMENT = True
    DEBUG = True
    WTF_CSRF_ENABLED = False
    DEBUG_TB_ENABLED = True

class TestingConfig(Config):
    TESTING = True
    DEBUG = True
    SQLALCHEMY_DATABASE_URI = "sqlite:///testdb.sqlite"
    BCRYPT_LOG_ROUNDS = 1
    WTF_CSRF_ENABLED = False

class ProductionConfig(Config):
    DEBUG = False
    DEBUG_TB_ENABLED = False

• Config: The base configuration class with default settings.
• DevelopmentConfig: Inherits from Config and overrides development settings.
• TestingConfig: Inherits from Config and overrides settings for testing.
• ProductionConfig: Inherits from Config and overrides production settings.

Finally, we'll create the main.py file. This is the entry point of our application. When we run this file, it starts the Flask web server.

from core import app

# Start the Flask app
if __name__ == '__main__':
    app.run(debug=True)

• if __name__ == '__main__': This ensures that the Flask app runs only if the script is executed directly (not imported as a module).
• app.run(debug=True): Starts the Flask development server with debug mode enabled, which provides detailed error messages and auto-reloading.

Now that you understand the project files, we can proceed with implementing the instant search functionality. This will involve creating the models and search route, setting up the HTMX-powered front-end, and connecting everything to fetch and display search results dynamically.

How to Set up the Database

In this section, we will set up the database for our Flask application. We will use SQLite for simplicity. We will create a model for the data we want to search and seed the database with sample data.

SQLite is a lightweight, disk-based database that doesn’t require a separate server process. It's an excellent choice for development and small projects because it is easy to set up and use.

How to Create a Model for the Data to Be Searched

We will create a Book model to represent the data in our database. This model will include fields like the book title and author.

Let's create the core/models.py file and add the model there:

from core import db

class Book(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(100), nullable=False)
    author = db.Column(db.String(100), nullable=False)

How to Apply Migrations Using Flask-Migrate

Before we can seed our database, we need to set up database migrations using Flask-Migrate. This tool helps us manage database changes, such as creating tables and altering schemas, systematically.

Initialize the migrations folder by running the following command in your project directory:

flask db init

This command creates a migrations directory in our project, which will store migration scripts.

Generate a migration script that creates the necessary database tables based on your models:

flask db migrate -m "Initial migration"

This command scans your models and generates a new migration script in the migrations folder.

Apply the migration to create the tables in your database:

flask db upgrade

This command executes the migration script, creating the tables defined by your models in the database. Post this step, you will see an instance/db.sqlite file created.

How to Seed Data Into Your Database

Now that we have set up the database and applied the migration, we can proceed with seeding the database. Create a file named seeder.py with the following content:

import csv
from sqlalchemy.exc import IntegrityError

from core import db, app
from core.models import Book


def seed_data():
    with app.app_context():
        # Open the CSV file
        with open("data.csv", newline='', encoding='utf-8') as csvfile:
            reader = csv.DictReader(csvfile)

            # Iterate over the rows in the CSV file
            for row in reader:
                # Create a new Book instance
                book = Book(
                    title=row['Book Name'],
                    author=row['Author Name']
                )

                # Add the book to the session
                db.session.add(book)

            try:
                # Commit the session to write the books to the database
                db.session.commit()
                print("Books added successfully.")
            except IntegrityError as e:
                db.session.rollback()
                print(f"Error occurred: {e}")


if __name__ == "__main__":
    seed_data()

The seeder script is responsible for populating the database with initial data. This is useful for testing and development purposes, allowing you to work with a set of sample data. This script reads data from data.csv, and processes it to insert it into the database.

Note: You can download the data.csv file from here.

To use this script, ensure your data.csv file exists in the same directory as seeder.py. Run the script using Python:

python seeder.py

How to Set Up Basic Routing and HTML

In this section, we'll set up a basic route in Flask to serve an index page (index.html) where users can search and display books.

How to Set Up Flask Route

Let's set up a Flask route (/) to render an index.html template and display books. For that, create a core/routes.py file and add the following route:

from flask import render_template
from core import app
from core.models import Book

@app.route('/')
def index():
    # Fetch the first 20 books to display by default
    books = Book.query.limit(20).all()
    return render_template("index.html", books=books)

The Flask application handles routing through the @app.route('/') decorator, which directs requests to the root URL (/). When a user visits the homepage, the index() function is invoked.

Inside this function, we query the Book model using SQLAlchemy to fetch the first 20 books from the database. These books are then passed as a parameter (books) to the render_template function, which renders the index.html template.

How to Creating the index.html Template

Create a file named index.html inside a templates directory in your project. The templates directory will lie in the core package. This file will contain the HTML structure for our book search page.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Book Search</title>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css" />
</head>
<body>
  <section class="section">
    <div class="columns">
      <div class="column is-one-third is-offset-one-third">
        <input type="text" class="input" placeholder="Search" name="query" />
      </div>
    </div>
    <table class="table is-fullwidth">
      <thead>
        <tr>
          <th>ID</th>
          <th>Book Title</th>
          <th>Book Author</th>
        </tr>
      </thead>
      <tbody id="results">
        {% for book in books %}
        <tr>
          <td>{{ book.id }}</td>
          <td>{{ book.title }}</td>
          <td>{{ book.author }}</td>
        </tr>
        {% endfor %}
      </tbody>
    </table>
  </section>
</body>
</html>

This HTML file uses the Bulma CSS framework for styling and includes elements such as an input field for user searches and a table to display book details fetched from the database.

The index.html template utilizes Jinja2 templating to dynamically populate the table rows (<tr>) with book data retrieved from the Flask backend. Each book's ID, title, and author are displayed in the table rows using {{book.id}}, {{ book.title }}, and {{book.author}} respectively.

How to Run the Application

Let's run the application using the following command:

flask run

Once your application is up and running, this what how it should look like:

web page with ID, book titles, and book authors

How to Add HTMX for Instant Search

Finally, we'll add HTMX to enhance our Flask application with dynamic search capabilities. For this, we'll introduce a new route and modify existing HTML template.

How to Create the Search Route

First, create a new route /search in your Flask application to handle book searches based on user input:

from flask import render_template, request
from core import app
from core.models import Book

@app.route('/search')
def search():
    query = request.args.get("query")
    if query:
        results = Book.query.filter(Book.title.ilike(f"%{query}%") | Book.author.ilike(f"%{query}%")).limit(10).all()
    else:
        results = Book.query.limit(20).all()
    return render_template("search_results.html", results=results)

This route listens for GET requests to /search. It retrieves the search query from the URL parameter using request.args.get("query").

If a query parameter is present, it uses SQLAlchemy's ilike method to perform a case-insensitive search across the title and author columns of the Book table, fetching up to 10 results.

If no query parameter is provided, it defaults to fetching the first 20 books from the database. The results are passed to a new search_results.html template for rendering.

How to Modify index.html to Add HTMX

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Book Search</title>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css" />
  <!-- Include HTMX library -->
  <script src="https://cdn.jsdelivr.net/npm/htmx.org/dist/htmx.min.js"></script>
</head>
<body>
  <section class="section">
    <div class="columns">
      <div class="column is-one-third is-offset-one-third">
        <!-- HTMX-enabled search input -->
        <input
            type="text"
            class="input"
            placeholder="Search"
            name="query"
            hx-get="/search"
            hx-trigger="keyup changed delay:500ms"
            hx-target="#results"
          />
      </div>
    </div>
    <table class="table is-fullwidth">
      <thead>
        <tr>
          <th>ID</th>
          <th>Book Title</th>
          <th>Book Author</th>
        </tr>
      </thead>
      <tbody id="results">
        {% for book in books %}
          <tr>
            <td>{{ book.id }}</td>
            <td>{{ book.title }}</td>
            <td>{{ book.author }}</td>
          </tr>
        {% endfor %}
      </tbody>
    </table>
  </section>
</body>
</html>

The <script> tag imports the HTMX library from a CDN, enabling client-side interactions without requiring complex JavaScript. In addition to that, we enhanced the <input> element with HTMX attributes:

• hx-get="/search": Specifies the endpoint (/search) to send GET requests when the user types in the input field.
• hx-trigger="keyup changed delay:500ms": Triggers the search action after a 500ms delay when the user types (keyup) or changes the input (changed).
• hx-target="#results": Updates the content of the element with id="results" with the response from the /search endpoint.

How to Create the search_results.html Template

Next, we will create a new template search_results.html to display search results:

{% for result in results %}
<tr>
    <td>{{ result.id }}</td>
    <td>{{ result.title }}</td>
    <td>{{ result.author }}</td>
</tr>
{% endfor %}

This template iterates over results, which are passed from the /search route. For each book in results, generates a table row (<tr>) that displays the book's ID, title, and author.

Demo

Finally, we have implemented instant search with HTMX in our Flask application. Here's what our final application should look like:

You'd notice a delay in the search results. This is called debouncing. It is a technique used in programming and web development to limit the rate at which a function or event handler is executed. It ensures that a function is only executed after a certain amount of time has passed since the last invocation of the function.

In our case, we set the delay to 500ms before it calls the /search API again. This ensures that we do not hit the API for every character the user types.

Conclusion

In this tutorial, you learned how to implement instant search using Flask and HTMX, focusing on enhancing user interaction and performance. By integrating HTMX for AJAX interactions, we enabled dynamic updates to search results without refreshing the entire page.

This approach not only improves user experience by providing real-time feedback but also optimizes server load by debouncing search queries.

By mastering these techniques, you're equipped to build responsive web applications that deliver seamless search experiences, combining the flexibility of Flask with the interactivity of HTMX to meet diverse user needs efficiently and effectively.

You can find the code for this tutorial in this repository: https://github.com/ashutoshkrris/instant-search-with-flask-htmx

Magic Link Authentication offers a simple yet secure way for users to log in without passwords. This tutorial will walk you through the implementation of Magic Link Authentication using React for the front end, Flask for the back end, and the authentication service provided by Authsignal.

Table of Contents:

1. Understanding Magic Link Authentication
2. How to Configure Authsignal
3. Application Flow
4. How to Set Up Your Backend Server
5. How to Set Up Environment Variables
6. How to Initialize the Authsignal Client
7. Authsignal Actions
8. How to Create the Required Routes
9. How to Set Up a New Frontend React Project
10. How to Set Up the Components
11. How to Run the Application
12. Wrapping Up

Understanding Magic Link Authentication

Magic link authentication is a convenient and secure authentication method that simplifies users' login process. Instead of entering a username and password, users receive a unique link via email. This link, known as a magic link, grants them access to their account without traditional credentials.

One key difference between magic link authentication and authentication with email verification is the user experience. With magic link authentication, users can authenticate with just a single click. They don't need to remember or enter a password, which can be especially beneficial for users who struggle with password management or find it inconvenient to type in their credentials repeatedly.

While email verification adds an extra layer of security, it may require the user to remember additional credentials or go through multiple steps before accessing their account. If you're interested in learning more about email verification, you can check out my article here to dive deeper into the topic.

How to Configure Authsignal

Authsignal is a service that makes implementing modern authentication methods (like Magic Links and Passkeys) easier. It provides simple tools to integrate secure login methods into your web apps without hassle.

Before proceeding with the tutorial, you need to create an Authsignal account. To do that, you can follow these steps:

First, go to authsignal.com and click on "Create Free Account".

In the next step, create your first tenant. Choose any name for your tenant and select the data storage region.

Creating Tenant on Authsignal

Next, you need to configure the authenticators you want to use for your application. For example, I have enabled Email Magic Link and Authenticator App (TOTP).

Configuring Authenticators

Once you have configured the authenticator, navigate to the API Keys option. Here, you will find your Secret Key, which will be necessary for implementing the authentication.

Finding your secret key

Application Flow

Let's understand the flow of the application:

Initial Visit

• The user visits the application's user interface. On the user interface, they see a login input box and a signup option. Since the user is new, they opt to sign up.

Signup Flow

• Upon clicking the signup link, the user is directed to a page to enter their chosen username.
• After entering the username and clicking the signup button, the front end triggers a POST API call to /api/signup, sending the username in the request body.
• The backend receives the request and communicates with the Authsignal server for user authentication.
• Authsignal prompts the user to set up Magic Link authentication by entering their email address.
• Authsignal sends a magic link to the provided email address.
• After clicking the magic link, the user is authenticated and redirected to the home page, where they receive a welcome message displaying their email address. The page also includes a logout button.

Login Flow

• The user logs out and returns to the login page.
• Here, the user enters their registered username and clicks the Login button.
• Upon clicking Login, the front end triggers a POST API call to /api/login, passing the username in the request body.
• The backend again communicates with Authsignal for user authentication, prompting the setup of Magic Link authentication.
• The user is directed to a page to enter their email address.
• Authsignal sends a magic link to the provided email address.
• After clicking the magic link, the user is authenticated and redirected to the home page, greeted with a welcome message displaying their email address.

This flow ensures users can sign up using a chosen username, and set up Magic Link authentication via email for both signup and login. Here is a video tutorial to visually aid you in understanding the flow:

How to Set Up Your Backend Server

In this section, I will guide you through how to set up your Flask server for implementing Magic Link Authentication. Before we begin, it's recommended to set up a virtual environment to isolate your project's dependencies. Here's how you can do it:

1. Open your terminal or command prompt.
2. Navigate to your project's directory.
3. Run the following command to create a new virtual environment:

python -m venv myenv

Note: Replace myenv with the desired name for your virtual environment.

4. Activate the virtual environment using the appropriate command for your operating system:

5. For Windows:

source myenv/Scripts/activate

• For macOS/Linux:

source myenv/bin/activate

Now that you have your virtual environment set up, let's install the necessary dependencies.

To begin, make sure you have Flask installed, which is a micro web framework for Python. You can install it using the following one-liner:

pip install Flask

Next, we need python-decouple, a library that helps manage configuration settings in separate files. Install it with the following command:

pip install python-decouple

The flask-cors library is a Flask extension that allows for Cross-Origin Resource Sharing (CORS) support in your Flask application.

pip install flask-cors

Finally, we need to install the Python SDK for Authsignal. You can install it with the following command:

pip install authsignal

Now that we have all the necessary dependencies installed, let's create a sample Flask server to get started with Magic Link Authentication. Here's a basic setup to help you get started:

from flask import Flask
from flask_cors import CORS

app = Flask(__name__)
CORS(app, supports_credentials=True)

@app.route("/")
def hello():
    return "Hello, world!"

if __name__ == "__main__":
    app.run(debug=True)

In the next steps, we will integrate AuthSignal and implement the Magic Link Authentication functionality into this server.

How to Set Up Environment Variables

To successfully configure and integrate AuthSignal into your Flask server, you need to set up the following environment variables:

• AUTHSIGNAL_BASE_URL: This variable contains the base URL of the Authsignal server. It allows your server to communicate with Authsignal's authentication service.
• AUTHSIGNAL_SECRET_KEY: This variable contains the secret key associated with your Authsignal project. It is used for secure communication between your server and AuthSignal.
• SECRET_KEY: This variable is a random key used to encrypt the cookies and send them to the browser.

Setting environment variables instead of hardcoding in the code provides improved security by keeping sensitive information, such as API keys and secret keys, separate from the codebase. This reduces the risk of accidental exposure or unauthorized access to these credentials.

To set up these environment variables, you can follow these steps:

1. Open a terminal or command prompt.
2. Navigate to the directory where your Flask server is located.
3. Create a .env file and export the environment variables using the following commands:

export AUTHSIGNAL_BASE_URL=<base_url>
export AUTHSIGNAL_SECRET_KEY=<secret_key>
export SECRET_KEY=<random-secret-key>

Make sure to replace <base_url>, <secret_key>, and <random-secret-key> with the appropriate values for your Authsignal project. You can find the values for these environment variables in the API Keys section of the Authsignal dashboard as explained earlier.

Note: The method for setting environment variables can vary depending on your operating system. The above commands are applicable for Unix-based systems. For Windows, you can use the set command instead of export.

To export the variables added in the .env file, you can use the following command in the terminal:

source .env

By properly setting these environment variables, your Flask server can securely communicate with Authsignal and implement the Magic Link Authentication functionality.

How to Initialize the Authsignal Client

To integrate Authsignal into your Flask server and implement Magic Link Authentication, you need to initialize the Authsignal client. Here's an example of how you can do this:

from flask import Flask
from flask_cors import CORS
import authsignal.client
from decouple import config

app = Flask(__name__)
CORS(app)

AUTHSIGNAL_BASE_URL = config("AUTHSIGNAL_BASE_URL")
AUTHSIGNAL_SECRET_KEY = config("AUTHSIGNAL_SECRET_KEY")
SECRET_KEY = config("SECRET_KEY")

authsignal_client = authsignal.Client(
    api_key=AUTHSIGNAL_SECRET_KEY,
    api_url=AUTHSIGNAL_BASE_URL
)

In this code snippet, we first import the authsignal.client, the Python SDK for Authsignal. We also import config from python-decouple to retrieve the environment variables.

We retrieve the environment variables AUTHSIGNAL_BASE_URL, AUTHSIGNAL_SECRET_KEY and SECRET_KEY using config from python-decouple.

Finally, we initialize the authsignal.Client by passing in the API key AUTHSIGNAL_SECRET_KEY and the base URL of the Authsignal server AUTHSIGNAL_BASE_URL.

By initializing the Authsignal client, we are ready to implement the Magic Link Authentication functionality in our Flask server.

Authsignal Actions

Authsignal allows you to create actions to track and manage user interactions in your application. Actions are events that can be triggered by users, such as signing up or logging in. By creating custom actions, you can have more control over the authentication process and implement specific authentication methods like Magic Link Authentication.

To create an action on your Authsignal dashboard, follow these steps:

1. Click on "Actions" in your Authsignal dashboard.
2. Click on "Configure a new action" to create a new action.
3. Enter a name for the action that describes its purpose or the user interaction it represents.
4. Next, you can configure the rule for the action. In our case, since we want to implement Magic Link Authentication, we will add a rule to challenge users with Email Magic Link. This will send a magic link to the user's email for authentication.
5. Save the action to apply the rule and make it active.

Here is a video demonstrating the process of creating an Authsignal action and configuring it for Magic Link Authentication:

Creating action on Authsignal

You can create two actions – "signUp" and "signIn". In the next steps, we will make use of these actions.

How to Create the Required Routes

Finally, to implement the Magic Link Authentication, we need to create three routes: /api/signup, /api/login, /api/callback, and /api/user.

/api/signup Route

The /api/signup route is responsible for allowing the users to register in our application. Here's how we implement it:

@app.route('/api/signup', methods=['POST'])
def signup():
    username = request.json.get('username')
    if not username:
        return jsonify({'error': 'Missing username parameter'}), 400

    response = authsignal_client.track(
        user_id=username,
        action="signUp",
        payload={
            "user_id": username,
            "redirectUrl": "http://localhost:5000/api/callback"
        }
    )
    return jsonify(response), 200

In this implementation, the route expects a JSON payload containing the username parameter. It then uses the authsignal_client to track the user's signUp action and generate a Magic Link. The track method lets you record actions performed by users and initiate challenges. The redirectUrl specifies the URL where the user will be redirected after they have been authenticated. We will create this API next.

/api/callback Route

The /api/callback route handles the callback URL where the user is redirected after verifying themselves. Here's the implementation for this route:

@app.route('/api/callback', methods=['GET'])
def callback():
    token = request.args.get('token')
    challenge_response = authsignal_client.validate_challenge(token)

    if challenge_response["state"] == 'CHALLENGE_SUCCEEDED':
        encoded_token = jwt.encode(
            payload={"username": challenge_response["user_id"]},
            key=SECRET_KEY,
            algorithm="HS256"
        )
        response = redirect('http://localhost:3000/')
        response.set_cookie(
            key='auth-session',
            value=encoded_token,
            secure=False,
            path='/'
        )
        return response

    return redirect("/")

When the users are redirected, Authsignal adds the JWT token in the URL as a token query parameter.

In this implementation, the route retrieves the token parameter from the query string. It then uses the authsignal_client to validate the challenge and check if the authentication was successful.

If the authentication succeeds, we encode a JSON Web Token (JWT). The token payload includes the username obtained from the challenge response. It uses the SECRET_KEY and the HS256 algorithm for encryption.

Next, the user is redirected to the home page (http://localhost:3000/), and a auth-session cookie is set with the encoded token for further user identification.

Note that the token returned from Authsignal in the redirect is not intended to be used as a session token. It just contains information about the challenge so that we can determine if the challenge was successful.

/api/login Route

The /api/login route is responsible for allowing the users to log into the application. Here’s the implementation for the route:

@app.route('/api/login', methods=['POST'])
def login():
    username = request.json.get('username')
    if not username:
        return jsonify({'error': 'Missing username parameter'}), 400

    response = authsignal_client.track(
        user_id=username,
        action="signIn",
        payload={
            "user_id": username,
            "redirectUrl": "http://localhost:5000/api/callback"
        }
    )
    return jsonify(response), 200

The route is configured to handle POST requests on the /api/login endpoint. Upon receiving a POST request, the route first extracts the provided username from the JSON payload sent with the request. It ensures that the username is present. If not, it promptly returns a 400 error response indicating a missing username parameter.

Similar to the signup flow, it then uses the authsignal_client to track the user's signIn action and generate a Magic Link. The redirectUrl specifies the URL where the user will be redirected after they have been authenticated.

/api/user Route

The /api/user route is responsible for retrieving user information. Here's the implementation for this route:

@app.route("/api/user", methods=['GET'])
def user():
    token = request.cookies.get('auth-session')
    decoded_token = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
    username = decoded_token.get('username')
    response = authsignal_client.get_user(user_id=username)
    return jsonify({"username": username, "email": response["email"]}), 200

In this implementation, the GET endpoint starts by extracting the auth-session cookie from the incoming request. Then it decodes the JWT using the jwt.decode method, utilizing the SECRET_KEY as the secret key for decoding.

The decoded token provides the username of the user. It then uses the authsignal_client to retrieve user information based on the provided userId. It then returns a JSON response with the username and email information.

By implementing these routes, we will be able to handle the basic authentication process and retrieve user information in our Flask server.

How to Set Up a New Frontend React Project

Let's set up our front-end project in this section. This will also include setting up routing in the application.

Start by initializing a new React project using create-react-app or any preferred method (like Vite, for example, which is a more modern way to set up a React app). This command sets up the basic structure for your React application.

npx create-react-app magic-link-auth
cd magic-link-auth

Once the project is created and you're inside the project directory, install the required dependencies. Here, we need react-router-dom for handling routing and bootstrap for easy styling.

npm install react-router-dom bootstrap

Import Bootstrap CSS

Bootstrap provides pre-styled components and utilities for easier and faster styling of your application.

In the index.js file, import Bootstrap:

import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import reportWebVitals from './reportWebVitals';
import "bootstrap/dist/css/bootstrap.min.css"; // Import Bootstrap CSS

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

Additionally, we will write some custom CSS. Replace the code in the index.css file with the following code:

html,
body {
  padding: 0;
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen,
    Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue, sans-serif;
}

* {
  box-sizing: border-box;
}

main {
  padding: 5rem 0;
  flex: 1;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

code {
  background: #fafafa;
  border-radius: 5px;
  padding: 0.75rem;
  font-family: Menlo, Monaco, Lucida Console, Courier New, monospace;
}

input[type="button"] {
  border: none;
  background: cornflowerblue;
  color: white;
  padding: 12px 24px;
  margin: 8px;
  font-size: 18px;
  border-radius: 8px;
  cursor: pointer;
}

Similarly, replace the code in the App.css with the following code:

.mainContainer {
  flex-direction: column;
  display: flex;
  align-items: center;
  justify-content: center;
  height: 100vh;
}

.titleContainer {
  display: flex;
  flex-direction: column;
  font-size: 48px;
  font-weight: bolder;
  align-items: center;
  justify-content: center;
}

.resultContainer,
.historyItem {
  flex-direction: row;
  display: flex;
  width: 400px;
  align-items: center;
  justify-content: space-between;
}

.historyContainer {
  flex-direction: column;
  display: flex;
  height: 200px;
  align-items: center;
  flex-grow: 5;
  justify-content: flex-start;
}

.buttonContainer {
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  height: 260px;
}

.inputContainer {
  display: flex;
  flex-direction: column;
  align-items: flex-start;
  justify-content: center;
}

.inputContainer>.errorLabel {
  color: red;
  font-size: 16px;
  text-align: center;
}

.inputBox {
  height: 48px;
  width: 400px;
  font-size: medium;
  border-radius: 8px;
  border: 1px solid grey;
  padding-left: 8px;
}

.inputButton {
  height: 48px;
  width: 400px;
}

Set Up Routing

Routing in React applications helps navigate between different views or pages. react-router-dom simplifies this process.

In your App.js file, configure the routing:

import React from "react";
import { BrowserRouter, Routes, Route } from "react-router-dom";
import Dashboard from "./pages/Dashboard";
import Register from "./pages/Register";
import Login from "./pages/Login";
import "./App.css";
import "./index.css";

const App = () => {
  return (
    <BrowserRouter>
      <Routes>
        <Route path="/" element={<Dashboard />} />
        <Route path="/login" element={<Login />} />
        <Route path="/signup" element={<Register />} />
      </Routes>
    </BrowserRouter>
  );
};

export default App;

It defines an App component that encapsulates the entire application structure within a <BrowserRouter> component. Inside <Routes>, we define three components: one for the root path / rendering the Dashboard component, and two for the /login and /signup paths, rendering the Login and Register components respectively.

This setup enables navigation between different views based on URL paths, allowing users to access specific components when they visit corresponding routes within the application.

In the upcoming sections, we will set up the above-mentioned three components.

How to Set Up the Components

In the previous section, we imported two components from the src/pages folder. Let's create a pages folder inside the src folder, and then we can start creating the components.

Register Component

Let’s create a Register.jsx file inside the pages folder. The Register component allows users to register within our application.

import React, { useState, useEffect } from "react";
import { useNavigate, Link } from "react-router-dom";

const Register = () => {
  const [username, setUsername] = useState("");
  const [usernameError, setUsernameError] = useState("");

  const navigate = useNavigate();

  useEffect(() => {
    const isAuthenticated = checkCookies();
    if (isAuthenticated) {
      navigate("/");
    }
  }, [navigate]);

  const checkCookies = () => {
    const authSessionCookie = document.cookie.match("auth-session=([^;]+)");

    return !!authSessionCookie;
  };

  const onButtonClick = () => {
    setUsernameError("");

    if ("" === username) {
      setUsernameError("Username is mandatory!");
      return;
    }

    signup();
  };

  const signup = async () => {
    const response = await fetch("http://localhost:5000/api/login", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        username,
      }),
      credentials: "include",
    });

    const { url } = await response.json();

    // Redirect to verification URL
    window.location.href = url;
  };

  return (
    <div className={"mainContainer"}>
      <div className={"titleContainer"}>
        <div>Sign Up</div>
      </div>
      <br />
      <div className={"inputContainer"}>
        <input
          value={username}
          placeholder="Enter your username"
          onChange={(e) => setUsername(e.target.value)}
          className={"inputBox"}
        />
        <label className="errorLabel text-center">{usernameError}</label>
      </div>
      <br />
      <div className={"inputContainer"}>
        <input
          className={"inputButton"}
          type="button"
          onClick={onButtonClick}
          value={"Sign Up"}
        />
      </div>
      <div>
        Existing User? <Link to="/login">Login here</Link>
      </div>
    </div>
  );
};

export default Register;

The component initializes state variables using useState to manage the visibility of the usernameError and store the username input in userName. It also initializes the navigate function from useNavigate to handle navigation within the application.

We use the useEffect hook to check for authentication cookies when the component mounts. It calls the checkCookies function, which checks for the existence of cookies that we had set from the backend server. If auth-session cookie is found, the user is automatically redirected to the root URL using navigate("/").

Clicking the “Sign Up” button triggers the onButtonClick function. It first checks whether the user has entered the username. If not, it shows an error message using the usernameError. If the user has entered the username, it calls the signup function.

The signup performs an asynchronous POST request to the /api/signup endpoint with the provided username. Upon successful response, it redirects the user to the received verification URL by changing window.location.href.

The JSX returned by the component defines the UI layout that looks like the below:

Register UI Component

It includes an input field for the users to enter their username and a "Sign Up" button. Below the button, we have a link to the Login page for the existing users to log in.

Login Component

We have kept the login page similar to the signup page for simplicity. Hence, the Login component is pretty much the same as the Register component.

import React, { useState, useEffect } from "react";
import { useNavigate, Link } from "react-router-dom";

const Login = () => {
  const [username, setUsername] = useState("");
  const [usernameError, setUsernameError] = useState("");

  const navigate = useNavigate();

  useEffect(() => {
    const isAuthenticated = checkCookies();
    if (isAuthenticated) {
      navigate("/");
    }
  }, [navigate]);

  const checkCookies = () => {
    const authSessionCookie = document.cookie.match("auth-session=([^;]+)");

    return !!authSessionCookie;
  };

  const onButtonClick = () => {
    setUsernameError("");

    if ("" === username) {
      setUsernameError("Username is mandatory!");
      return;
    }

    login();
  };

  const login = async () => {
    const response = await fetch("http://localhost:5000/api/login", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        username,
      }),
      credentials: "include",
    });

    const { url } = await response.json();

    // Redirect to verification URL
    window.location.href = url;
  };

  return (
    <div className={"mainContainer"}>
      <div className={"titleContainer"}>
        <div>Login</div>
      </div>
      <br />
      <div className={"inputContainer"}>
        <input
          value={username}
          placeholder="Enter your username"
          onChange={(ev) => setUsername(ev.target.value)}
          className={"inputBox"}
        />
        <label className="errorLabel text-center">{usernameError}</label>
      </div>
      <br />
      <div className={"inputContainer"}>
        <input
          className={"inputButton"}
          type="button"
          onClick={onButtonClick}
          value={"Log in"}
        />
      </div>
      <div>
        New User? <Link to="/signup">Sign up here</Link>
      </div>
    </div>
  );
};

export default Login;

The only significant difference other than the text and button in the UI here is that we will be making the API call to the /api/login route when the users hit the Login button.

The UI looks like below:

Login UI Component

Dashboard Component

The Dashboard component in our application serves as the interface for authenticated users, displaying a welcome message with the user’s email and enabling user logout functionality.

Let’s create a Dashboard.jsx file inside the pages folder.

import React, { useState, useEffect } from "react";
import { useNavigate } from "react-router-dom";

const Dashboard = () => {
  const [userEmail, setUserEmail] = useState("");
  const navigate = useNavigate();

  useEffect(() => {
    const checkCookies = async () => {
      const authSessionCookie = document.cookie.match("auth-session=([^;]+)");

      if (!authSessionCookie) {
        navigate("/auth");
        return false;
      }

      return true;
    };

    const fetchData = async () => {
      const cookiesValid = await checkCookies();
      if (!cookiesValid) return;

      try {
        const response = await fetch("http://localhost:5000/api/user", {
          method: "GET",
          headers: {
            "Content-Type": "application/json",
          },
          credentials: "include"
        });

        if (!response.ok) {
          throw new Error("Failed to fetch user data");
        }

        const data = await response.json();
        setUserEmail(data.email);
      } catch (error) {
        navigate("/auth");
      }
    };

    fetchData();
  }, [navigate]);

  const handleLogout = () => {
    document.cookie = `auth-session=; max-age=0`;
    navigate("/auth");
  };

  return (
    <div className="d-flex justify-content-center align-items-center vh-100">
      <main className="px-3 text-center">
        <h1>Welcome Home!</h1>
        <p className="lead">You're logged in as {userEmail}!</p>
        <div className="d-flex justify-content-center">
          <button
            className="btn btn-lg btn-dark fw-bold border-white bg-dark"
            onClick={handleLogout}
          >
            Log Out
          </button>
        </div>
      </main>
    </div>
  );
};

export default Dashboard;

Upon component mounting or when navigate changes (a dependency of useEffect), the effect runs. It begins by defining two asynchronous functions. The first, checkCookies, verifies the presence of auth-session cookies. If it is missing, it redirects the user to the authentication route.

The second function, fetchData, is responsible for fetching user data. It checks the validity of cookies using checkCookies. Upon verification, it sends a GET request to our back-end API endpoint. Upon successful response, it updates the userEmail state with the user's email fetched from the API data.

If any error occurs during this process, such as failing to fetch user data, it redirects the user back to the authentication route.

The JSX returned by the component renders a simple dashboard layout.

Dashboard UI Component

The displayed content includes a welcoming message and the currently logged-in user's email. There's also a "Log Out" button which, upon clicking, initiates the logout process by triggering the handleLogout function. It removes the auth-session cookies by setting their max-age to 0, effectively expiring them. Afterward, it redirects the user to the authentication route.

How to Run the Application

You can find the code of the final application in this GitHub repository. To run your backend application, run python app.py from your back-end folder in your terminal. This will start your back-end server on port 5000. Next, run the frontend application using the npm start command. This will start your frontend application on the port 3000.

Wrapping Up

In this tutorial, you learned how to use Authsignal to implement basic user authentication with email verification through magic links.

Authsignal makes handling users and keeping things safe easier, letting developers focus on improving apps. It also removes the overhead of remembering another password for the users of the application.

To learn more about Authsignal, visit the Authsignal documentation.

Should you have any issues or questions related to the tutorial, then feel free to reach out to me on Twitter.

In this article, we'll explore 8 essential Python one-liners that every Pythonista should have in their toolkit. From list comprehensions to lambda functions and beyond, these techniques offer elegant solutions to common programming challenges, helping you write cleaner, more efficient code.

List Comprehension

List comprehension is a Pythonic way to create lists with a single line of code. It offers a concise alternative to traditional loops, enabling you to generate lists quickly and efficiently.

Let's say you want to create a list containing squares of numbers from 0 to 9. Using a traditional loop, you'd do it like this:

# Using a traditional loop
squared_numbers = []
for i in range(10):
    squared_numbers.append(i ** 2)
print(squared_numbers)

The traditional loop method requires more lines of code and explicitly defines the iteration process, appending each squared number to the list step by step.

On the other hand, list comprehension can achieve the same result in a single line, making the code more concise and readable. It condenses the loop into a clear, compact structure, generating the squared numbers directly into a list.

# Using list comprehension
squared_numbers = [i ** 2 for i in range(10)]
print(squared_numbers)

You can use list comprehensions when you need to apply a simple operation to every element in a sequence, such as transforming a list of numbers or strings.

You can learn how you can pack and destructure lists in Python here.

Lambda Functions

Lambda functions, also known as anonymous functions, allow you to create small, throwaway functions without explicitly defining them with def. They are particularly useful in scenarios where a function is needed for a short operation.

First, let's look at an example using def:

# Using def
def add_numbers(x, y):
    return x + y

print(add_numbers(2, 3))

In this code, the def keyword is used to define a named function add_numbers explicitly. It takes an argument x and y and returns the sum of them. This traditional approach provides a named function that can be called multiple times.

But when you need a function just for one-time usage, you can just define an anonymous function using the lambda keyword like this:

# Using Lambda
add = lambda x, y: x + y
print(add(2, 3))

It achieves the same result as add_numbers but in a single line without assigning a name explicitly. Lambda functions are useful for short, throwaway functions that are used infrequently or as part of other expressions.

Map and Filter

The map and filter functions are powerful tools for working with iterables, allowing concise manipulation and filtering of data.

Let's say you have a list of strings and you want to convert each item of the list into uppercase.

fruits = ['apple', 'banana', 'cherry']
upper_case_loop = []
for fruit in fruits:
    upper_case_loop.append(fruit.upper())
print(upper_case_loop)

Now, you can achieve the same using the map function:

upper_case = list(map(lambda x: x.upper(), ['apple', 'banana', 'cherry']))

You can utilize map when you need to perform an operation on every element of an iterable. filter is handy for selectively choosing elements based on a condition.

You can learn more about the map, filter and reduce functions here.

Ternary Operator

The ternary operator provides a condensed way to write conditional statements in a single line, enhancing code readability.

Let's say, you have a number and you want to check if it's even or odd. You can do it using the traditional if condition as below:

# Traditional if
result = None
num = 5
if num % 2 == 0:
    result = "Even"
else:
    result = "Odd"

But you can achieve the same results in a single line using the ternary operator:

# Ternary Operator
num = 7
result = "Even" if num % 2 == 0 else "Odd"

When you need to assign values based on conditions, especially in situations requiring simple if-else checks, the ternary operator shines.

Zip Function

The zip function enables you to combine multiple iterables element-wise, forming tuples of corresponding elements.

Let's assume you have two lists: one containing the names of students and the other containing their respective grades for a specific assignment.

students = ['Dilli', 'Vikram', 'Rolex', 'Leo']
grades = [85, 92, 78, 88]

Now, you want to create a report that pairs each student's name with their grade for easy comprehension or further analysis. You can do it by iterating over the list and appending them to a new list as below:

students = ['Dilli', 'Vikram', 'Rolex', 'Leo']
grades = [85, 92, 78, 88]

student_grade_pairs = []
for i in range(len(students)):
    student_grade_pairs.append((students[i], grades[i]))

print(student_grade_pairs)

The above loop method manually pairs elements from two lists by iterating through their indices, accessing elements at the same positions, and appending tuples of those elements into a new list student_grade_pairs.

But, what if I tell you that we can achieve the same pairing effect in one line using the zip function as below:

students = ['Dilli', 'Vikram', 'Rolex', 'Leo']
grades = [85, 92, 78, 88]

student_grade_pairs = list(zip(students, grades))
print(student_grade_pairs)

The zip function elegantly combines elements from both lists, creating pairs of corresponding elements as tuples. The result student_grade_pairs is a list of tuples, where each tuple contains an element from the grades list paired with the corresponding element from the students list.

You can learn more about the zip function here.

Enumerate Function

The enumerate function offers a concise way to iterate over a sequence while keeping track of the index.

Let's say you're developing a feature where users can add items to their shopping list, and you want to display the items along with their position or index in the list for easy reference.

You can do it using a traditional for-loop as below:

# Simulating a grocery list
grocery_list = ['Apples', 'Milk', 'Bread', 'Eggs', 'Cheese']

# Displaying the grocery list with indices
for i in range(len(grocery_list)):
    print(f"{i}. {grocery_list[i]}")

The traditional loop with manual indexing involves using range along with len to generate indices that are then used to access elements in the grocery_list list. This method requires more code and is less readable due to the explicit handling of indices.

The enumerate function simplifies the process by directly providing both indices and elements from the grocery_list list.

# Simulating a grocery list
grocery_list = ['Apples', 'Milk', 'Bread', 'Eggs', 'Cheese']

# Displaying the grocery list with indices
for index, item in enumerate(grocery_list):
    print(f"{index}. {item}")

It's concise, readable, and more Pythonic, eliminating the need for manual index handling and making the code cleaner. This approach is generally preferred for its simplicity and clarity in obtaining indices and elements from an iterable.

String Join

The join method is a clean way to concatenate strings from an iterable into a single string.

Suppose you have a list of words and want to create a sentence by joining these words using traditional concatenation. You'd do it as below:

# Using traditional concatenation
words = ['Python', 'is', 'awesome', 'and', 'powerful']

sentence = ''
for word in words:
    sentence += word + ' '

print(sentence.strip())  # Strip to remove the trailing space

In the traditional concatenation method, a loop iterates through the list of words, and each word is concatenated with a space. But this approach requires creating a new string for each concatenation operation, which might not be efficient for larger strings due to string immutability.

The join method, on the other hand, is more efficient and concise. It joins the elements of the list using the specified separator (in this case, a space), creating the sentence in a single operation.

.# Using join method
words = ['Python', 'is', 'awesome', 'and', 'powerful']

sentence = ' '.join(words)
print(sentence)

This method is generally the preferred way to join strings in Python due to its efficiency and readability.

Unpacking Lists

Python's unpacking feature allows for efficient assignment of elements from iterables to variables.

Suppose you have a list of numbers, and you want to assign each number to separate variables using traditional indexing.

# Using traditional unpacking
numbers = [1, 2, 3]

a = numbers[0]
b = numbers[1]
c = numbers[2]

print(a, b, c)

In the traditional unpacking method, individual elements from the list are accessed and assigned to separate variables by explicitly indexing each element. This method is more verbose and requires knowing the number of elements in advance.

Now, let's accomplish the same using the * operator for unpacking the list into variables.

# Using * operator for unpacking
numbers = [1, 2, 3]

a, b, c = numbers

print(a, b, c)

You can learn more about the * operator and list unpacking in this tutorial.

Should You Always Use One-Liners?

While Python one-liners offer conciseness and elegance, there are considerations to keep in mind before applying them universally:

1. Readability: One-liners might sacrifice readability for crispness. Complex one-liners can be hard to understand, especially for newcomers or when revisiting code after some time.
2. Maintainability: Overuse of one-liners, especially complex ones, can make code maintenance challenging. Debugging and modifying concise code might be more difficult.
3. Performance: In certain scenarios, one-liners might not be the most performant solution. These concise expressions may consume more resources, such as memory or CPU, and their underlying operations might have higher time complexity, affecting efficiency, especially with large datasets or intensive computations.
4. Debugging: Debugging a one-liner can be more challenging due to its compactness. Identifying issues or errors might take longer compared to well-structured, multiple-line code.
5. Context: Not all situations warrant one-liners. Sometimes, a straightforward, explicit approach might be more suitable for code clarity, especially when working in teams.

Ultimately, the decision to use one-liners should consider the trade-offs between conciseness and readability. Strive for a balance that enhances code clarity without compromising maintainability and understanding, especially when collaborating or working on larger projects.

Wrapping Up

Mastering Python's concise techniques like list comprehensions, lambda functions, enumerate, join, zip, and unpacking with the * operator can significantly enhance code readability, efficiency, and simplicity. These methods offer elegant solutions to common programming challenges, reducing verbosity and improving code maintainability.

Understanding when and how to use these Pythonic constructs empowers developers to write cleaner, more expressive code and enhancing overall productivity in various programming scenarios.

Think of it as a shield for your accounts. Passwords can sometimes be guessed or stolen, but with 2FA, even if someone gets your password, they'd still need that extra code or device to get in. It's an extra step that makes your accounts much harder for hackers to break into.

So, let's explore how to set up this extra layer of protection using PyOTP and Google Authenticator in your Flask app.

Table of Contents:

1. Overview of PyOTP and Google Authenticator
2. Two-Factor Authentication Workflow in Our Application
3. Prerequisites
4. Get Your Tools Ready
5. How to Set Up the Project
6. How to Create Blueprints for Accounts and Core
7. How to Create a User Model
8. How to Add Flask-Login
9. How to Add Templates and Static Files
10. How to Create the Homepage
11. How to Implement User Registration
12. [How to Implement User Login](#

    How to Implement User Login

    )
13. How to Log Out the Users
14. How to Add the Setup 2FA Page
15. How to Add a 2FA Verification Page
16. How to Run the Completed App for the First Time
17. Wrapping Up

Overview of PyOTP and Google Authenticator

PyOTP is a Python library that's incredibly handy for generating Time-based One-Time Passwords (TOTP) and HMAC-based One-Time Passwords (HOTP). Its primary role revolves around creating these unique, time-sensitive codes that add an extra layer of security to user accounts.

By integrating PyOTP into your Flask application, you can easily implement Two-Factor Authentication (2FA) by generating and verifying these OTPs.

If you're new to PyOTP or would like a refresher on its functionalities, I recommend reviewing my previous guide on PyOTP. This understanding will be beneficial as we get into the integration of PyOTP within your Flask application for Two-Factor Authentication (2FA).

Google Authenticator, on the other hand, stands out as one of the most widely used OTP generator apps available. It functions as a secure platform for generating time-based OTPs, compatible with various services and applications supporting 2FA. Users can easily set up Google Authenticator on their devices to generate these time-sensitive codes, adding an extra level of security to their accounts.

Two-Factor Authentication Workflow in Our Application

Here's a breakdown of the flow of two-factor authentication in our application:

1. Registration with 2FA Setup: When users sign up on our website, they're prompted to set up an extra layer of security—2FA. This involves scanning a QR code using an authenticator app, such as Google Authenticator, to link their account securely.
2. Login Initiation: When users return to log in, they start by entering their usual email/username and password combo to access their account.
3. Extra Security Check: Before granting access, our website throws in an additional hurdle: users need to provide an OTP (One-Time Password) displayed on their authenticator app. This ensures they're not just entering the password but also confirming their identity with a unique, time-sensitive code.
4. Validation and Authorization: The user inputs the received OTP into our platform. The system then double-checks this OTP against the expected code, validating the information. If the OTP matches, it's like handing over the secret handshake, granting the user access to their account.

This seamless back-and-forth between passwords, authenticator apps, and unique codes ensures that only the rightful account owner can access the precious content behind the digital doors of your website.

If you also enjoy visual learning, here's a fancy video showing how the app does its thing.

Now, let's get to some coding!

Prerequisites

Before you get started with the tutorial, make sure you have the following requirements satisfied:

• Working knowledge of Python
• Python 3.8+ installed on your system
• Basic knowledge of Flask and Flask Blueprints
• Knowledge of basic authentication in Flask (optional)

Get Your Tools Ready

You'll need a few external libraries for this project. Let's learn more about them and install them one by one.

But before we install them, let's create a virtual environment and activate it.

First, start with creating the project directory and navigating to it like this:

mkdir flask-two-factor-auth
ccd flask-two-factor-auth

We are going to create a virtual environment using venv. Python now ships with a pre-installed venv library. So, to create a virtual environment, you can use the below command:

python -m venv env

The above command will create a virtual environment named env. Now, we need to activate the environment using this command:

source env/Scripts/activate

To verify if the environment has been activated or not, you can see (env) in your terminal. Now, we can install the libraries.

• Flask is a simple, easy-to-use microframework for Python that helps you build scalable and secure web applications.
• Flask-Login provides user session management for Flask. It handles the common tasks of logging in, logging out, and remembering your users’ sessions over extended periods of time.
• Flask-Bcrypt is a Flask extension that provides bcrypt hashing utilities for your application.
• Flask-WTF is a simple integration of Flask and WTForms that helps you create forms in Flask.
• Flask-Migrate is an extension that handles SQLAlchemy database migrations for Flask applications using Alembic. The database operations are made available through the Flask command-line interface.
• Flask-SQLAlchemy is an extension for Flask that adds support for SQLAlchemy to your application. It helps you simplify things using SQLAlchemy with Flask by giving you useful defaults and extra helpers that make it easier to perform common tasks.
• PyOTP helps you generate OTPs using Time-based OTP (TOTP) and HMAC-based OTP (HOTP) algorithms effortlessly.
• QRCode helps you generate QR Codes in Python
• Python Decouple helps you use environment variables in your Python project.

To install the above-mentioned libraries all in one go, run the following command:

pip install Flask Flask-Login Flask-Bcrypt Flask-WTF FLask-Migrate Flask-SQLAlchemy pyotp qrcode python-decouple

How to Set Up the Project

Let’s start by creating a src directory:

mkdir src

The first file will be the __init__.py file for the project:

from decouple import config
from flask import Flask
from flask_bcrypt import Bcrypt
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config.from_object(config("APP_SETTINGS"))

bcrypt = Bcrypt(app)
db = SQLAlchemy(app)
migrate = Migrate(app, db)

# Registering blueprints
from src.accounts.views import accounts_bp
from src.core.views import core_bp

app.register_blueprint(accounts_bp)
app.register_blueprint(core_bp)

In the above script, we created a Flask app called app . We use the __name__ argument to indicate the app's module or package so that Flask knows where to find other files such as templates. We also set the configuration of the app using an environment variable called APP_SETTINGS. We'll export it later.

To use Flask-Bcrypt, Flask-SQLAlchemy, and Flask-Migrate in our application, we just need to create objects of the Bcrypt, SQLAlchemy and Migrate classes from the flask_bcrypt, flask_sqlalchemy and, flask_migrate libraries, respectively.

We've also registered blueprints called accounts_bp and core_bp in the application. We'll define them later in the tutorial.

In the root directory of the project (that is, outside the src directory), create a file called config.py. We'll store the configurations for the project in this file. Within the file, add the following content:

from decouple import config

DATABASE_URI = config("DATABASE_URL")
if DATABASE_URI.startswith("postgres://"):
    DATABASE_URI = DATABASE_URI.replace("postgres://", "postgresql://", 1)


class Config(object):
    DEBUG = False
    TESTING = False
    CSRF_ENABLED = True
    SECRET_KEY = config("SECRET_KEY", default="guess-me")
    SQLALCHEMY_DATABASE_URI = DATABASE_URI
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    BCRYPT_LOG_ROUNDS = 13
    WTF_CSRF_ENABLED = True
    DEBUG_TB_ENABLED = False
    DEBUG_TB_INTERCEPT_REDIRECTS = False
    APP_NAME = config("APP_NAME")


class DevelopmentConfig(Config):
    DEVELOPMENT = True
    DEBUG = True
    WTF_CSRF_ENABLED = False
    DEBUG_TB_ENABLED = True


class TestingConfig(Config):
    TESTING = True
    DEBUG = True
    SQLALCHEMY_DATABASE_URI = "sqlite:///testdb.sqlite"
    BCRYPT_LOG_ROUNDS = 1
    WTF_CSRF_ENABLED = False


class ProductionConfig(Config):
    DEBUG = False
    DEBUG_TB_ENABLED = False

In the above script, we have created a Config class and defined various attributes inside that. Also, we have created different child classes (as per different stages of development) that inherit the Config class.

Notice that we're using a few environment variables like SECRET_KEY, DATABASE_URL, and APP_NAME. Create a file named .env in the root directory and add the following content there:

export SECRET_KEY=fdkjshfhjsdfdskfdsfdcbsjdkfdsdf
export DEBUG=True
export APP_SETTINGS=config.DevelopmentConfig
export DATABASE_URL=sqlite:///db.sqlite
export FLASK_APP=src
export FLASK_DEBUG=1
export APP_NAME="Flask User Authentication App"

Apart from the SECRET_KEY , DATABASE_URL and APP_NAME, we've also exported APP_SETTINGS, DEBUG, FLASK_APP, and FLASK_DEBUG.

The APP_SETTINGS refers to one of the classes we created in the config.py file. We set it to the current stage of the project.

The value of FLASK_APP is the name of the package we have created. Since the app is in the development stage, you can set the values of DEBUG and FLASK_DEBUG to True and 1, respectively.

Run the following command to export all the environment variables from the .env file:

source .env

Next, we'll create a CLI application of the app so that we can later add custom commands if required.

Create a manage.py file in the root directory of the application and add the following code:

from flask.cli import FlaskGroup

from src import app

cli = FlaskGroup(app)


if __name__ == "__main__":
    cli()

Now, your basic application is ready. You can run it using the following command:

python manage.py run

Your file structure should look like below as of now:

flask-two-factor-auth/
├── src/
│   └── __init__.py
├── .env
├── config.py
└── manage.py

How to Create Blueprints for Accounts and Core

As mentioned earlier, you'll use the concepts of blueprints in the project. Let's create two blueprints – accounts_bp and core_bp – in this section.

First create a directory called accounts like this:

mkdir accounts
cd accounts

Next, add an empty __init__.py file to covert it into a Python package. Now, create a views.py file inside the package where you'll store all your routes related to user authentication.

touch __init__.py views.py

Add the following code inside the views.py file:

from flask import Blueprint

accounts_bp = Blueprint("accounts", __name__)

In the above script, you have created a blueprint called accounts_bp for the accounts package.

Similarly, you can create a core package in the root directory, and add a views.py file.

mkdir core
cd core
touch __init__.py views.py

Now, add the following code inside the views.py file:

from flask import Blueprint

core_bp = Blueprint("core", __name__)

Note: If you're new to Flask Blueprints, make sure you go through this tutorial to learn more about how it works.

Now, your file structure should look like what you see below:

flask-two-factor-auth/
├── src/
│   ├── accounts/
│   │   ├── __init__.py
│   │   └── views.py
│   ├── core/
│   │   ├── __init__.py
│   │   └── views.py
│   └── __init__.py
├── .env
├── config.py
└── manage.py

How to Create a User Model

Let's create a models.py file inside the accounts package.

touch src/accounts/models.py

Inside the models.py file, add the following code:

from datetime import datetime

import pyotp
from flask_login import UserMixin

from src import bcrypt, db
from config import Config


class User(db.Model):

    __tablename__ = "users"

    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String, unique=True, nullable=False)
    password = db.Column(db.String, nullable=False)
    created_at = db.Column(db.DateTime, nullable=False)
    is_two_factor_authentication_enabled = db.Column(
        db.Boolean, nullable=False, default=False)
    secret_token = db.Column(db.String, unique=True)

    def __init__(self, username, password):
        self.username = username
        self.password = bcrypt.generate_password_hash(password)
        self.created_at = datetime.now()
        self.secret_token = pyotp.random_base32()

    def get_authentication_setup_uri(self):
        return pyotp.totp.TOTP(self.secret_token).provisioning_uri(
            name=self.username, issuer_name=Config.APP_NAME)

    def is_otp_valid(self, user_otp):
        totp = pyotp.parse_uri(self.get_authentication_setup_uri())
        return totp.verify(user_otp)

    def __repr__(self):
        return f"<user {self.username}>"

In the above code, you created a User model by inheriting the db.Model class. The User model consists of the following fields:

• id: stores the primary key for the users table
• username: stores the username of the user
• password: stores the hashed password of the user
• created_at: stores the timestamp when the user was created
• is_two_factor_authentication_enabled: boolean flag that stores whether the user has activated two-factor authentication. Default value is False.
• secret_token: stores a unique token generated for each user, essential for implementing two-factor authentication.

The constructor initializes the User object upon instantiation by accepting username and password parameters. It hashes the provided password using bcrypt.generate_password_hash(password), records the current timestamp as the created_at value, and generates a unique secret_token using pyotp.random_base32() for 2FA setup.

The get_authentication_setup_uri() method generates a setup URI used by authenticator apps like Google Authenticator. It constructs a URI containing the user's username and the application's name (Config.APP_NAME) necessary for setting up two-factor authentication. The basic format of the URI is:

otpauth://totp/Example:alice@google.com?secret=JBSWY3DPEHPK3PXP&issuer=Example

where, alice@google.com is the username of the user and Example is the application's name.

Next up, the is_otp_valid() method verifies the one-time password (OTP) entered by the user during login. It parses the setup URI generated earlier, checks the validity of the provided OTP (user_otp), and returns True if the OTP matches, ensuring secure authentication.

Finally, the __repr__ method provides a string representation of the User object, displaying the associated username when an instance of the class is printed or represented as a string.

How to Add Flask-Login

The most important part of Flask-Login is the LoginManager class that lets your application and Flask-Login work together.

In the src/__init__.py file, add the following code:

from decouple import config
from flask import Flask
from flask_login import LoginManager # Add this line
from flask_migrate import Migrate
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config.from_object(config("APP_SETTINGS"))

login_manager = LoginManager() # Add this line
login_manager.init_app(app) # Add this line
db = SQLAlchemy(app)
migrate = Migrate(app, db)

# Registering blueprints
from src.accounts.views import accounts_bp
from src.core.views import core_bp

app.register_blueprint(accounts_bp)
app.register_blueprint(core_bp)

In the above script, we created and initialized the login manager in our app.

Next, we need to provide a user_loader callback. This callback is used to reload the user object from the user ID stored in the session. It should take the ID of a user, and return the corresponding user object.

from src.accounts.models import User

@login_manager.user_loader
def load_user(user_id):
    return User.query.filter(User.id == int(user_id)).first()

The User model should implement the following properties and methods:

• is_authenticated: This property returns True if the user is authenticated.
• is_active: This property returns True if this is an active user (the account is activated)
• is_anonymous: This property returns True if this is an anonymous user (actual users return False).
• get_id(): This method returns a string that uniquely identifies this user, and can be used to load the user from the user_loader callback.

Now, we don't need to implement these explicitly. Instead, the Flask-Login provides a UserMixin class that contains the default implementations for all of these properties and methods. We just need to inherit it in the following way:

from datetime import datetime

from flask_login import UserMixin # Add this line

from src import bcrypt, db


class User(UserMixin, db.Model): # Change this line
    ....

We can also customize the default login process in the src/__init__.py file.

The name of the login view can be set as LoginManager.login_view. The value refers to the function name that will handle the login process.

login_manager.login_view = "accounts.login"

To customize the message category, set LoginManager.login_message_category:

login_manager.login_message_category = "danger"

How to Add Templates and Static Files

Let's create a CSS file called styles.css inside the src/static folder:

.error {
  color: red;
  margin-bottom: 5px;
  text-align: center;
}

a {
  text-decoration: none;
}

Let's also create the basic templates inside the src/templates folder. Create a _base.html file and add the following code:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Two Factor Authentication</title>
    <!-- meta -->
    <meta name="description" content="">
    <meta name="author" content="">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <!-- styles -->
    <!-- CSS only -->
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-T3c6CoIi6uLrA9TneNEoa7RxnatzjcDSCmG1MXxSR1GAsXEV/Dwwykc2MPK8M2HN" crossorigin="anonymous">
    <link rel="stylesheet" href="{{url_for('static', filename="styles.css")}}">
    {% block css %}{% endblock %}
  </head>
  <body>

    {% include "navigation.html" %}

    <div class="container">

      <br>

      <!-- messages -->
      {% with messages = get_flashed_messages(with_categories=true) %}
      {% if messages %}
      <div class="row">
        <div class="col-md-4"></div>
        <div class="col-md-4">
          {% for category, message in messages %}
          <div class="alert alert-{{ category }} alert-dismissible fade show" role="alert">
           {{message}}
           <button type="button" class="btn-close" data-bs-dismiss="alert" aria-label="Close"></button>
          </div>
          {% endfor %}
        </div>
        <div class="col-md-4"></div>
      </div>
      {% endif %}
      {% endwith %}

      <!-- child template -->
      {% block content %}{% endblock %}

    </div>

    <!-- scripts -->
    <script src="https://code.jquery.com/jquery-3.7.1.min.js" integrity="sha256-/JqT3SQfawRcv/BIHPThkBvs0OEvtFFmqPF/lYI/Cxo=" crossorigin="anonymous"></script>
    <!-- JavaScript Bundle with Popper -->
    <script src="https://cdn.jsdelivr.net/npm/@popperjs/core@2.11.8/dist/umd/popper.min.js" integrity="sha384-I7E8VVD/ismYTF4hNIPjVp/Zjvgyol6VFvRkX/vR+Vc4jQkC+hVqc2pM8ODewa9r" crossorigin="anonymous"></script>
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/js/bootstrap.min.js" integrity="sha384-BBtl+eGJRgqQAUMxJ7pMwbEyER4l1g+O15P+16Ep7Q9Q+zqX6gSbd85u4mG4QzX+" crossorigin="anonymous"></script>
    {% block js %}{% endblock %}
  </body>
</html>

The _base.html is the parent HTML file that will be inherited by the other templates. We have added Bootstrap 5 support in the above file. We are also making use of Flask Flashes to show Bootstrap alerts in the app.

Let's also create a navigation.html file that contains the navbar of the app:

<!-- Navigation -->
<nav class="navbar bg-dark navbar-expand-lg bg-body-tertiary p-3" data-bs-theme="dark">
  <div class="container-fluid">
    <a class="navbar-brand" href="{{ url_for('core.home') }}">Two-Factor Authentication App</a>
    <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
      <span class="navbar-toggler-icon"></span>
    </button>
    <div class="collapse navbar-collapse" id="navbarSupportedContent">
      {% if current_user.is_authenticated %}
      <a href="{{ url_for('accounts.logout') }}"><button type="button" class="btn btn-danger me-2">Logout</button></a>
      {% endif %}
    </div>
  </div>
</nav>

Note that we have not yet created the views used above.

How to Create the Homepage

In this section, we'll first create a view function for the homepage inside the core/views.py file. Add the following code there:

from flask import Blueprint, render_template
from flask_login import login_required

core_bp = Blueprint("core", __name__)


@core_bp.route("/")
@login_required
def home():
    return render_template("core/index.html")

Notice that we have used the blueprint to add the route. We also added a @login_required middleware to prevent access for unauthenticated users.

Next, let's create an index.html file inside the templates/core folder, and add the following code:

{% extends "_base.html" %}
{% block content %}

<h1 class="text-center">Welcome {{current_user.username}}!</h1>

{% endblock %}

The HTML page will just have a welcome message for authenticated users.

Your file structure as of now should look like below:

flask-two-factor-auth/
├── src/
│   ├── accounts/
│   │   ├── __init__.py
│   │   └── views.py
│   ├── core/
│   │   ├── __init__.py
│   │   └── views.py
│   ├── static/
│   │   └── styles.css
│   ├── templates/
│   │   ├── core/
│   │   │   └── index.html
│   │   ├── _base.html
│   │   └── navigation.html
│   └── __init__.py
├── .env
├── config.py
└── manage.py

How to Implement User Registration

First of all, we'll create a registration form using Flask-WTF. Create a forms.py file inside the accounts package and add the following code:

from flask_wtf import FlaskForm
from wtforms import EmailField, PasswordField
from wtforms.validators import DataRequired, Email, EqualTo, Length

from src.accounts.models import User


class RegisterForm(FlaskForm):
    username = StringField(
        "Username", validators=[DataRequired(), Length(min=6, max=40)]
    )
    password = PasswordField(
        "Password", validators=[DataRequired(), Length(min=6, max=25)]
    )
    confirm = PasswordField(
        "Repeat password",
        validators=[
            DataRequired(),
            EqualTo("password", message="Passwords must match."),
        ],
    )

    def validate(self, extra_validators):
        initial_validation = super(RegisterForm, self).validate(extra_validators)
        if not initial_validation:
            return False
        user = User.query.filter_by(username=self.username.data).first()
        if user:
            self.username.errors.append("Username already registered")
            return False
        if self.password.data != self.confirm.data:
            self.password.errors.append("Passwords must match")
            return False
        return True

The RegisterForm extends the FlaskForm class and contains three fields – username, password, and confirm. We have added different validators such as DataRequired, Length, Email, and EqualTo to the respective fields.

We also defined a validate() method which is automatically called when the form is submitted.

Inside the method, we first perform the initial validation provided by FlaskForm. If that is successful, we perform our custom validation such as checking whether user is already registered, and matching the password with the confirmed password. If there are any errors, we append the error message in the respective fields.

Now, let's use this form inside the HTML file. Create an accounts directory inside the templates folder and add a new file called register.html inside it. Add the following code:

{% extends "_base.html" %}

{% block content %}

<div class="row">
  <div class="col-md-4"></div>
  <div class="col-md-4">
    <main class="form-signin w-100 m-auto">
      <form role="form" method="post" action="">
        {{ form.csrf_token }}
        <h1 class="h3 mb-3 fw-normal text-center">Please register</h1>

        <div class="form-floating">
          {{ form.username(placeholder="username", class="form-control mb-2") }}
          {{ form.username.label }}
            {% if form.username.errors %}
              {% for error in form.username.errors %}
                <div class="alert alert-danger" role="alert">
                  {{ error }}
                </div>
              {% endfor %}
            {% endif %}
        </div>
        <div class="form-floating">
          {{ form.password(placeholder="password", class="form-control mb-2") }}
          {{ form.password.label }}
            {% if form.password.errors %}
              {% for error in form.password.errors %}
                <div class="alert alert-danger" role="alert">
                  {{ error }}
                </div>
              {% endfor %}
            {% endif %}
        </div>
        <div class="form-floating">
          {{ form.confirm(placeholder="Confirm Password", class="form-control mb-2") }}
          {{ form.confirm.label }}
            {% if form.confirm.errors %}
              {% for error in form.confirm.errors %}
                <div class="alert alert-danger" role="alert">
                  {{ error }}
                </div>
              {% endfor %}
            {% endif %}
        </div>
        <button class="w-100 btn btn-lg btn-primary" type="submit">Sign up</button>
        <p class="text-center mt-3">Already registered? <a href="{{ url_for('accounts.login') }}">Login now</a></p>
      </form>
    </main>
  </div>
  <div class="col-md-4"></div>
</div>

{% endblock %}

In the above Jinja template, we make use of the form that we created, and add relevant error handling logic checks for validation errors in each field. Users can submit the form by clicking the "Sign up" button, and a link below the form allows already registered users to navigate to the login page for authentication.

Next, let's use this form in the views.py to create a function to handle the registration process.

from .forms import RegisterForm
from src.accounts.models import User
from src import db, bcrypt
from flask_login import current_user
from flask import Blueprint, flash, redirect, render_template, request, url_for

accounts_bp = Blueprint("accounts", __name__)

HOME_URL = "core.home"
SETUP_2FA_URL = "accounts.setup_two_factor_auth"
VERIFY_2FA_URL = "accounts.verify_two_factor_auth"

@accounts_bp.route("/register", methods=["GET", "POST"])
def register():
    if current_user.is_authenticated:
        if current_user.is_two_factor_authentication_enabled:
            flash("You are already registered.", "info")
            return redirect(url_for(HOME_URL))
        else:
            flash("You have not enabled 2-Factor Authentication. Please enable first to login.", "info")
            return redirect(url_for(SETUP_2FA_URL))
    form = RegisterForm(request.form)
    if form.validate_on_submit():
        try:
            user = User(username=form.username.data, password=form.password.data)
            db.session.add(user)
            db.session.commit()

            login_user(user)
            flash("You are registered. You have to enable 2-Factor Authentication first to login.", "success")

            return redirect(url_for(SETUP_2FA_URL))
        except Exception:
            db.session.rollback()
            flash("Registration failed. Please try again.", "danger")

    return render_template("accounts/register.html", form=form)

The route begins by checking if the current user is already authenticated. If so, it verifies whether 2FA is enabled for the user. If 2FA is already enabled, a message informs the user that they're already registered, redirecting them to the home URL. However, if the user is authenticated but 2FA is not enabled, a flash message prompts the user to enable 2FA first before logging in, redirecting them to the 2FA setup URL.

If the user is not authenticated or has not yet registered 2FA, the code initializes a registration form and proceeds to validate the form data on submission. Upon successful form validation, we create a new User object with the provided username and password and save it to the database.

Upon successful user registration, the newly registered user is logged in. A success message flashes, notifying the user of successful registration and prompting them to enable 2FA before logging in. Subsequently, the user is redirected to the 2FA setup URL to enable 2FA.

How to Implement User Login

First, let's create a login form in the accounts/forms.py file:

class LoginForm(FlaskForm):
    username = StringField("Username", validators=[DataRequired()])
    password = PasswordField("Password", validators=[DataRequired()])

The form is similar to the registration form but it has only two fields – username and password.

Now, let's use this form inside new HTML file called login.html created inside the templates/accounts directory. Add the following code:

{% extends "_base.html" %}

{% block content %}

<div class="row">
  <div class="col-md-4"></div>
  <div class="col-md-4">
    <main class="form-signin w-100 m-auto">
      <form role="form" method="post" action="">
        {{ form.csrf_token }}
        <h1 class="h3 mb-3 fw-normal text-center">Please sign in</h1>

        <div class="form-floating">
          {{ form.username(placeholder="username", class="form-control mb-2") }}
          {{ form.username.label }}
            {% if form.username.errors %}
              {% for error in form.username.errors %}
              <div class="alert alert-danger" role="alert">
                {{ error }}
              </div>
              {% endfor %}
            {% endif %}
        </div>
        <div class="form-floating">
          {{ form.password(placeholder="password", class="form-control mb-2") }}
          {{ form.password.label }}
            {% if form.password.errors %}
              {% for error in form.password.errors %}
                <div class="alert alert-danger" role="alert">
                  {{ error }}
                </div>
              {% endfor %}
            {% endif %}
        </div>
        <button class="w-100 btn btn-lg btn-primary" type="submit">Sign in</button>
        <p class="text-center mt-3">New User? <a href="{{ url_for('accounts.register') }}">Register now</a></p>
      </form>
    </main>
  </div>
  <div class="col-md-4"></div>
</div>

{% endblock %}

The above HTML file is also similar to the register.html file but with just two fields for the username and password.

Next, let's create a view function to handle the login process inside the accounts/views.py file:

from .forms import LoginForm, RegisterForm

@accounts_bp.route("/login", methods=["GET", "POST"])
def login():
    if current_user.is_authenticated:
        if current_user.is_two_factor_authentication_enabled:
            flash("You are already logged in.", "info")
            return redirect(url_for(HOME_URL))
        else:
            flash("You have not enabled 2-Factor Authentication. Please enable first to login.", "info")
            return redirect(url_for(SETUP_2FA_URL))

    form = LoginForm(request.form)
    if form.validate_on_submit():
        user = User.query.filter_by(username=form.username.data).first()
        if user and bcrypt.check_password_hash(user.password, request.form["password"]):
            login_user(user)
            if not current_user.is_two_factor_authentication_enabled:
                flash(
                    "You have not enabled 2-Factor Authentication. Please enable first to login.", "info")
                return redirect(url_for(SETUP_2FA_URL))
            return redirect(url_for(VERIFY_2FA_URL))
        elif not user:
            flash("You are not registered. Please register.", "danger")
        else:
            flash("Invalid username and/or password.", "danger")
    return render_template("accounts/login.html", form=form)

The route starts by checking if the current user is already authenticated. If the user is authenticated and 2FA is enabled, a message informs the user they're already logged in, redirecting them to the home URL. If the user is authenticated but 2FA isn't enabled, a flash message prompts the user to enable 2FA before logging in, redirecting them to the 2FA setup URL.

If the user isn't authenticated, the code initializes a login form and validates the form data upon submission. Upon successful validation, it queries the database to find a user matching the provided username. If the user exists and the password matches the hashed password stored in the database, the user is logged in.

Additionally, if 2FA isn't enabled for the current user after successful login, a flash message prompts the user to enable 2FA before proceeding, redirecting them to the 2FA setup URL. If the login is successful and 2FA is enabled, the user is redirected to the 2FA verification URL.

If the user isn't registered, a flash message informs them to register. If there's a mismatch in the provided username or password, another flash message notifies the user of invalid credentials.

How to Log Out the Users

Logging out the user is a very simple process. You just need to create a view function for it inside the accounts/views.py file:

from flask_login import login_required, login_user, logout_user


@accounts_bp.route("/logout")
@login_required
def logout():
    logout_user()
    flash("You were logged out.", "success")
    return redirect(url_for("accounts.login"))

The Flask-Login library contains a logout_user method that removes the user from the session. We used the @login_required decorator so that only authenticated users can logout.

How to Add the Setup 2FA Page

Up until now, we have been redirecting the users to the setup 2FA page whenever the 2FA is not enabled in their accounts, but we haven't implemented it yet. Let's do that in this section.

Let's start with the route for the page:

from src.utils import get_b64encoded_qr_image

@accounts_bp.route("/setup-2fa")
@login_required
def setup_two_factor_auth():
    secret = current_user.secret_token
    uri = current_user.get_authentication_setup_uri()
    base64_qr_image = get_b64encoded_qr_image(uri)
    return render_template("accounts/setup-2fa.html", secret=secret, qr_image=base64_qr_image)

The route, created inside accounts/views.py, ensures that only authenticated users can access it using the @login_required decorator.

Upon accessing this route, the function retrieves the current user's secret_token for 2FA setup and generates a URI through current_user.get_authentication_setup_uri() to configure an authenticator app like Google Authenticator.

It also uses get_b64encoded_qr_image(uri) to obtain a Base64-encoded QR code image representing this setup URI. We will define it below.

Finally, it renders the setup-2fa.html template, passing the user's secret_token and the Base64-encoded QR image to the template for users to scan it.

Next, create a utils.py file in the src directory and add the following code to generate the QR:

from io import BytesIO
import qrcode
from base64 import b64encode


def get_b64encoded_qr_image(data):
    print(data)
    qr = qrcode.QRCode(version=1, box_size=10, border=5)
    qr.add_data(data)
    qr.make(fit=True)
    img = qr.make_image(fill_color='black', back_color='white')
    buffered = BytesIO()
    img.save(buffered)
    return b64encode(buffered.getvalue()).decode("utf-8")

Remember the qrcode library we installed in the beginning of the tutorial? This is where we're going to use it.

Upon receiving data as input, representing the content to be embedded within the QR code, the function initializes a QRCode object using the qrcode library. It adds the provided data to this QR code instance and generates the QR code. The code then converts this QR code into an image representation.

Using a BytesIO object, it stores this image in memory. The function proceeds to encode the content of this in-memory buffer, representing the QR code image, into Base64 format. Finally, it returns this Base64-encoded string, encapsulating the QR code image, ready for transmission or display in various applications.

Next, let's create the setup-2fa.html page inside the templates/accounts folder, and add the following content:

{% extends "_base.html" %}

{% block content %}

<div class="row">
  <div class="col-md-4"></div>
  <div class="col-md-4">
    <main class="form-signin w-100 m-auto">
      <form role="form">
        <h5>Instructions!</h5>
          <ul>
            <li>Download <a href="https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=en&gl=US" target="_blank">Google Authenticator</a> on your mobile.</li>
            <li>Set up a new authenticator.</li>
            <li>Once you have scanned the QR, please click <a href="{{ url_for('accounts.verify_two_factor_auth') }}">here.</li>
          </ul>
          <div class="text-center">
            <img src="data:image/png;base64, {{ qr_image }}" alt="Secret Token" style="width:200px;height:200px"/>
          </div>
        <div class="form-group">
          <label for="secret">Secret Token</label>
          <input type="text" class="form-control" id="secret" value="{{ secret }}" readonly>
        </div>
        <div class="text-center mt-2">
          <button type="button" class="btn btn-primary" onclick="copySecret()">
            Copy Secret
          </button>
        </div>
        <p class="mt-4 text-center">
          Once you have scanned the QR, please click <a href="{{ url_for('accounts.verify_two_factor_auth') }}">here</a>.
        </p>
      </form>
    </main>
  </div>
  <div class="col-md-4"></div>
</div>

{% endblock %}

{% block js %}
<script>
    function copySecret() {
    var copyText = document.getElementById("secret");
    copyText.select();
    copyText.setSelectionRange(0, 99999); /*For mobile devices*/
    document.execCommand("copy");
    alert("Successfully copied TOTP secret token!");
  }
</script>
{% endblock %}

We add some instructions in the page for the users to follow. These instructions provide clear steps for users to enable 2FA: directing them to download the Google Authenticator app via a link, guiding the setup process within the app, and prompting users to proceed by clicking a link after scanning the displayed QR code.

Displaying the QR code is central to the setup process. The template embeds the QR code image using an <img> tag with its source set to a Base64-encoded string ({{ qr_image }}). This image represents the secret key essential for 2FA setup.

We also show the secret key in read-only mode, allowing users to view the key without being able to modify it. We have added a copy button to make it easier for the users to copy the key.

Moreover, we have added a link to the 2FA verification page guiding users to proceed with the setup process after scanning the QR code. We will implement this functionality in the next section.

Here's how your page looks right now:

2FA Setup Page

How to Add a 2FA Verification Page

In this section, let's implement the 2FA verification. To start with, we will require an OTP form where users can enter their OTP. Add the following content in the accounts/forms.py file:

class TwoFactorForm(FlaskForm):
    otp = StringField('Enter OTP', validators=[
                      InputRequired(), Length(min=6, max=6)])

The TwoFactorForm contains just one field (otp) to get the OTP from the users.

Now, let's use this form in the verify-2fa.html file inside the templates/accounts folder:

{% extends "_base.html" %}

{% block content %}

<div class="row">
  <div class="col-md-4"></div>
  <div class="col-md-4">
    <main class="form-signin w-100 m-auto">
      <form role="form" method="post" action="">
        {{ form.csrf_token }}
        <h1 class="h3 mb-3 fw-normal text-center">Enter OTP</h1>

        <div class="form-floating">
          {{ form.otp(placeholder="OTP", class="form-control mb-2") }}
          {{ form.otp.label }}
            {% if form.otp.errors %}
              {% for error in form.otp.errors %}
              <div class="alert alert-danger" role="alert">
                {{ error }}
              </div>
              {% endfor %}
            {% endif %}
        </div>
        <button class="w-100 btn btn-lg btn-primary" type="submit">Verify</button>
      </form>
    </main>
  </div>
  <div class="col-md-4"></div>
</div>

{% endblock %}

The Jinja template essentially contains a form with one field for OTP and a verify button.

Let's create the route which handles the submission of this form inside the accounts/views.py file:

@accounts_bp.route("/verify-2fa", methods=["GET", "POST"])
@login_required
def verify_two_factor_auth():
    form = TwoFactorForm(request.form)
    if form.validate_on_submit():
        if current_user.is_otp_valid(form.otp.data):
            if current_user.is_two_factor_authentication_enabled:
                flash("2FA verification successful. You are logged in!", "success")
                return redirect(url_for(HOME_URL))
            else:
                try:
                    current_user.is_two_factor_authentication_enabled = True
                    db.session.commit()
                    flash("2FA setup successful. You are logged in!", "success")
                    return redirect(url_for(HOME_URL))
                except Exception:
                    db.session.rollback()
                    flash("2FA setup failed. Please try again.", "danger")
                    return redirect(url_for(VERIFY_2FA_URL))
        else:
            flash("Invalid OTP. Please try again.", "danger")
            return redirect(url_for(VERIFY_2FA_URL))
    else:
        if not current_user.is_two_factor_authentication_enabled:
            flash(
                "You have not enabled 2-Factor Authentication. Please enable it first.", "info")
        return render_template("accounts/verify-2fa.html", form=form)

The route starts by initializing a form (TwoFactorForm) meant for 2FA verification using the data obtained from the request. Upon form submission, the code proceeds with several conditional checks to validate the OTP entered by the user.

Once the form has been successfully submitted and validated, the code verifies the authenticity of the OTP using current_user.is_otp_valid(form.otp.data), which checks if the entered OTP is valid for the current user. If the OTP is valid, the code executes the following logic:

• If the provided OTP is valid and 2FA is already enabled for the user, a success message is flashed indicating successful 2FA verification, and the user is redirected to the home URL.
• If the OTP is valid but 2FA isn't enabled for the user, it attempts to enable 2FA for that user. Upon successful activation, a success message flashes, and the user is redirected to the home URL.

Furthermore, if the OTP entered by the user is invalid, the code flashes an error message indicating an invalid OTP and redirects the user back to the 2FA verification URL to retry the verification process.

2FA Verification Page

With this, we have completed the implementation of all the features! 🎉

How to Run the Completed App for the First Time

Now that our application is ready, you can first migrate the database, and then run the app.

To initialize the database (create a migration repository), use the command:

flask db init

To migrate the database changes, use the command:

flask db migrate

To apply the migrations, use the command:

flask db upgrade

Since this is the first time we're running our app, you'll need to run all the above commands. Later, whenever you make changes to the database, you'll just need to run the last two commands.

After that, you can run your application using the command:

python manage.py run

Since we have completed the development, here's how your file structure should look like:

flask-two-factor-auth/
├── migrations/
├── src/
│   ├── accounts/
│   │   ├── __init__.py
│   │   ├── forms.py
│   │   ├── models.py
│   │   └── views.py
│   ├── core/
│   │   ├── __init__.py
│   │   └── views.py
│   ├── static/
│   │   └── styles.css
│   ├── templates/
│   │   ├── accounts/
│   │   │   ├── login.html
│   │   │   ├── register.html
│   │   │   ├── setup-2fa.html
│   │   │   └── verify-2fa.html
│   │   ├── core/
│   │   │   └── index.html
│   │   ├── _base.html
│   │   └── navigation.html
│   ├── __init__.py
│   └── utils.py
├── .env
├── config.py
└── manage.py

Wrapping up

In this tutorial, you learned how to set up two-factor authentication in your Flask app using PyOTP.

Here's the link to the GitHub repository. Feel free to check it out whenever you're stuck.

Here are some other tutorials I wrote about authentication, email verification, and OTPs that you might enjoy:

• How to Set Up Basic User Authentication in a Flask App
• How to Set Up Email Verification in a Flask App
• How To Generate OTPs Using PyOTP in Python

Thank you for reading. I hope you found this article useful. You can follow me on Twitter.

This step-by-step guide will help you grasp the fundamentals of working with databases, data manipulation, and email communication, all while automating these processes with a Python script.

Business Context

Imagine you're a part of an organization where your managers expect a weekly report filled with valuable insights. But creating this report is far from a straightforward task.

To get the information you need, you have to manually run ten different database queries, gather the results, and then meticulously compile them into an Excel spreadsheet. It's a time-consuming and error-prone process that can leave you exhausted.

In this scenario, wouldn't it be a game-changer if Python could take the reins and handle this entire process for you?

Picture this: Every week, without any manual intervention, Python seamlessly extracts the required data, compiles it into a neat Excel sheet, and even sends it off to your managers like clockwork.

This tutorial will help you learn how to do this. I'll walk you through the steps to automate this process, making your weekly or monthly reporting a breeze, and freeing you up to focus on more critical tasks.

Table of Contents

1. Prerequisites
2. How to Set Up Your Virtual Environment
3. How to Set Up Your Sample Database
4. How to Set Up Logging and Environment Variables
5. How to Extract the Data From the Database
6. How to Structure the Booking Data with the BookingInfo Class
7. How to Convert the Data into an Excel Sheet
8. How to Combine the Functionalities
9. How to Send an Email with the Bookings Data Report
10. How to Test the Flow
11. How to Schedule the Application
12. Wrapping Up

Prerequisites

Before you get started, make sure you have the following:

1. Python installed on your computer. You can download Python from Python.org.
2. Basic knowledge of the Python programming language
3. Familiarity with sending emails in Python
4. PostgreSQL installed on your computer. You can download PostgreSQL from here.

How to Set Up Your Virtual Environment

Before you start coding, you'll need to make sure you have all the necessary tools and libraries installed.

To ensure that you have a clean and isolated environment, you'll create a virtual environment using venv.

Create a project directory and navigate to it in the terminal:

mkdir report-automation
cd report-automation

Create a virtual environment named env using the following command:

python -m venv env

Python now ships with the pre-installed venv library to create virtual environments.

Activate the virtual environment like this:

source env/bin/activate

Note: if you're on Windows, you'll need to use source env/Scripts/activate to activate the environment.

You should see (env) in your terminal prompt, indicating that the virtual environment has been activated.

How to Install the Required Libraries

Now that you've created the virtual environment, you can install the following libraries:

• psycopg2: Python adapter for PostgreSQL, enabling Python applications to interact with PostgreSQL databases.
• pandas: A versatile data manipulation and analysis library for Python, ideal for working with structured data.
• xlsxwriter: Python module for creating and formatting Excel (XLSX) files, useful for generating reports and spreadsheets.

To install the libraries, run the following command:

pip install psycopg2 pandas xlsxwriter

How to Set Up Your Sample Database

In this section, I will guide you through setting up a demo database named "airlines" that we'll use throughout this tutorial. The database includes three tables: bookings, flights, and airports_data.

I will provide you with an SQL script file named airlines_db.sql that creates the database and populates it with sample data. To set up the database, you will need PostgreSQL installed on your system.

Download and Install the Database

1. Download the SQL script file "airlines_db.sql" from here.
2. Open your terminal or command prompt.
3. Use the following command to install the database. Make sure you have the PostgreSQL command-line tools installed and that you can access the psql command. Replace postgres with your PostgreSQL username if it's different.

psql -f airlines_db.sql -U postgres

This command will execute the SQL script and create the "airlines" database with the bookings, flights, and airports_data tables.

Schema Description

The main schema in the database is bookings. Let's take a closer look at the tables in the "airlines" database:

Schema Diagram

Table bookings.bookings

The "bookings" table is designed to store crucial information about bookings made for flights. Each booking is uniquely identified by the book_ref, which is a character(6) field. The total_amount field is a numeric(10,2) type and represents the total cost of the booking.

To track the booking date and time, the table includes a book_date field of type bigint. This table serves as the central repository for booking data and is essential for tracking passenger reservations, costs, and booking dates.

Table bookings.flights

The "flights" table is dedicated to capturing comprehensive details about flights, including information about their statuses, scheduled and actual times of departure and arrival, and other important flight-related data.

The primary key for this table is the flight_id, an integer identifier. Each flight is associated with a specific flight number denoted by the flight_no field, a character(6) type.

To understand the flight's origin and destination, the departure_airport and arrival_airport fields store the departure and arrival airport codes as character(3) types, respectively.

The status field is a character varying(20) that records the flight's status, which must be one of 'On Time,' 'Delayed,' 'Departed,' 'Arrived,' 'Scheduled,' or 'Cancelled.' The table also includes fields for scheduled departure and arrival times (scheduled_departure and scheduled_arrival) and actual departure and arrival times (actual_departure and actual_arrival).

Furthermore, this table establishes two essential foreign keys: flights_arrival_airport_fkey and flights_departure_airport_fkey, which link to the airport_code in the "airports_data" table. This establishes connections between flights and their respective departure and arrival airports.

Table bookings.airports_data

The "airports_data" table serves as a repository for data related to airports and their geographic locations. Each airport is identified by a unique character(3) code stored in the airport_code field, which also serves as the primary key.

The timezone field, of type text, records the specific timezone of the airport, providing essential information for scheduling and operational purposes. The airport_name field is a character varying type that holds the name of the airport. Additionally, the table includes the city field as a character varying type, indicating the city in which the airport is situated.

These details enable the "airports_data" table to provide a comprehensive overview of airport locations and information. This serves as a reference for the "flights" table through the flights_arrival_airport_fkey and flights_departure_airport_fkey foreign keys, facilitating the association between flights and their respective departure and arrival airports.

How to Set Up Logging and Environment Variables

In this section, we'll configure logging to provide informative messages and handle errors throughout the code. We'll also set up environment variables to securely store sensitive information and configuration parameters. These practices enhance code readability, maintainability, and security.

Logging Configuration

We will utilize Python's built-in logging module to configure a logging system. Logging is essential for tracking the execution flow of the code and capturing important information or errors.

The logging.basicConfig method is called to define the format of log messages and set the logging level to INFO.

import logging

logging.basicConfig(
    format="%(asctime)s | %(levelname)s : %(message)s", level=logging.INFO
)

• Format: The format parameter specifies the format of log messages. In this case, each log entry includes a timestamp, log level (for example, INFO, ERROR), and the actual log message.
• Log Levels: We set the logging level to INFO, which means the logger will record informational messages. You can also use higher severity levels, such as WARNING or ERROR, for more critical issues.

You can learn more about logging in Python in this tutorial.

How to Manage Environment Variables

We will create a .env file to manage environment variables. Environment variables are used to store sensitive information and configuration settings, allowing us to keep such data separate from the code.

In this case, we set environment variables for email credentials and database connection details.

export EMAIL=
export PASSWORD=
export EMAIL_PORT=587
export SMTP_SERVER=smtp.gmail.com
export DB_HOSTNAME=localhost
export DB_NAME=airlines
export DB_PORT=5432
export DB_USERNAME=postgres
export DB_PASSWORD=postgres

Here's a breakdown of the variables:

• EMAIL: The email address to be used for sending emails.
• PASSWORD: The password associated with the email account.
• EMAIL_PORT: The port for the email server (for example, SMTP server). The default is 587 for secure email transmission (TLS/SSL).
• SMTP_SERVER: The SMTP server address, often specific to the email service provider.
• DB_HOSTNAME: The hostname or IP address of the PostgreSQL database server.
• DB_NAME: The name of the PostgreSQL database.
• DB_PORT: The port number for connecting to the database (default is 5432 for PostgreSQL).
• DB_USERNAME: The username for authenticating with the database.
• DB_PASSWORD: The password for the database user.

Make sure you run source .env to load the environment variables.

By using environment variables, sensitive data like passwords and email credentials can be kept separate from the code, reducing the risk of accidental exposure or unauthorized access. The code can access these variables at runtime, ensuring security and flexibility in configuration.

How to Extract the Data From the Database

Let's start by setting the database configurations.

import logging
import os

logging.basicConfig(
    format="%(asctime)s | %(levelname)s : %(message)s", level=logging.INFO
)

DB_CONFIG = {
    "host": os.environ.get("DB_HOSTNAME"),
    "database": os.environ.get("DB_NAME"),
    "user": os.environ.get("DB_USERNAME"),
    "password": os.environ.get("DB_PASSWORD"),
}

The DB_CONFIG dictionary is used to store the configuration parameters for connecting to the PostgreSQL database. These parameters include the host, database name, username, and password. These values can be set through environment variables.

How to Connect to the Database

Before we extract the data from the database, we need to connect to our database. We will use the psycopg2 library to connect to the PostgreSQL database.

We will start by defining a DataExporter class that will contain methods to extract the database and generate the Excel sheet.

class DataExporter:
    def __init__(self):
        """Initialize the DataExporter with the database configuration."""
        self.db_config = DB_CONFIG

The class constructor initializes the DataExporter with the database configuration stored in the DB_CONFIG dictionary.

Next, let's define a method that connects to the database.

...
import psycopg2

...

class DataExporter:
    def __init__(self):
        """Initialize the DataExporter with the database configuration."""
        self.db_config = DB_CONFIG

    def __connect_to_database(self) -> None:
        """
        Establish a connection to the PostgreSQL database.

        Raises:
            Exception: If a connection to the database cannot be established.
        """
        try:
            self.conn = psycopg2.connect(**self.db_config)
            self.cursor = self.conn.cursor()
            logging.info("Connected to the database")
        except Exception as e:
            logging.error(
                "Failed to connect to the database with error: %s", e)
            raise

The __connect_to_database private method is responsible for establishing a connection to the PostgreSQL database. It uses the psycopg2 library to create a connection and a cursor for executing SQL queries. If the connection fails, it logs an error and raises an exception.

You can learn more about exception handling in Python here.

How to Fetch Data from the Database

Now we'll define another private method that connects to the database and fetches the total number of bookings and the total amount from the database.

from datetime import datetime

class DataExporter:
    ...

    def __fetch_from_database(self, start_timestamp, end_timestamp) -> list | None:
        """
        Fetch booking data from the database for a given time range.

        Args:
            start_timestamp (datetime): The start of the time range.
            end_timestamp (datetime): The end of the time range.

        Returns:
            list: A list containing booking data (num_bookings, total_amount) or None if an error occurs.
        """
        self.__connect_to_database()
        query = f"""
        SELECT COUNT(*) AS num_bookings, SUM(total_amount) AS total_amount
        FROM bookings
        WHERE book_date >= {int(start_timestamp.timestamp()) * 1000} AND book_date <= {int(end_timestamp.timestamp()) * 1000}
        """
        logging.info(
            "Exracting bookings data from database for start timestamp=%s and end_timestamp=%s",
            start_timestamp,
            end_timestamp,
        )
        result = None
        try:
            self.cursor.execute(query)
            result = list(self.cursor.fetchone())
            result.append(
                f'{start_timestamp.strftime("%d %b, %Y")} - {end_timestamp.strftime("%d %b, %Y")}'
            )
            logging.info(
                "Successfully exracted bookings data from database for start timestamp=%s and end_timestamp=%s",
                start_timestamp,
                end_timestamp,
            )
        except Exception as e:
            logging.error(
                "Error occurred while extracting bookings data from database: %s", e
            )
        return result

This private method retrieves booking data from the database for a specified time range.

It takes two datetime objects as arguments, start_timestamp and end_timestamp. It also constructs a SQL query to retrieve the count of bookings and the total booking amount for that time range.

The query is executed, and if it's successful, the method returns the data as a tuple. We convert the tuple into a list and append the timeframe for which data was extracted to the list. If an error occurs during the database interaction, it logs an error and returns None.

Using the above method, you can extract booking data for various timeframes, whether it's for a week, a month, a year, or any custom time range of your choice.

How to Structure the Booking Data with the BookingInfo Class

In this section, we will define a BookingInfo class in booking_info.py, which serves as a structured container for booking data retrieved from the database. The class encapsulates booking-related information, making it easier to work with and present the data.

from decimal import Decimal


class BookingInfo:
    def __init__(self, data_list: list):
        """
        Initialize BookingInfo with data from the database.

        Args:
            data_list (list): A list containing booking data (total_bookings, total_amount, timestamp).

        Note:
            The total_amount is converted to a Decimal type.

        """
        self.__total_bookings, self.__total_amount, self.__timestamp = data_list
        self.__total_amount = Decimal(self.__total_amount) if self.__total_amount else Decimal(0)

    def __str__(self) -> str:
        """
        Return a string representation of BookingInfo.

        Returns:
            str: A string in the format "Total Bookings: X, Total Amount: $Y".

        """
        return f"Total Bookings: {self.__total_bookings}, Total Amount: ${self.__total_amount}"

    def get_total_bookings(self) -> int:
        """
        Get the total number of bookings.

        Returns:
            int: The total number of bookings.

        """
        return self.__total_bookings

    def get_total_amount(self) -> Decimal:
        """
        Get the total booking amount as a Decimal.

        Returns:
            Decimal: The total booking amount.

        """
        return self.__total_amount

    def get_timestamp(self) -> str:
        """
        Get the timestamp associated with the booking data.

        Returns:
            str: The timestamp as a string.

        """
        return self.__timestamp

The BookingInfo class is designed to organize and represent booking data returned from the database. It receives a list of values containing total bookings, total booking amount, and a timestamp as input and converts the total amount to a Decimal type. The class offers methods for accessing and presenting this data in a structured manner.

The constructor of the BookingInfo class takes a data_list as input, which is expected to be a list containing the following elements:

• total_bookings: An integer representing the total number of bookings.
• total_amount: A floating-point value representing the total booking amount.
• timestamp: A timestamp associated with the booking data.

The __init__ method initializes private instance variables (__total_bookings, __total_amount, and __timestamp) with the values from the data_list. It also converts the __total_amount to a decimal type for precise handling of monetary values.

The __str__ method is implemented to provide a string representation of the BookingInfo object. It returns a string in the format "Total Bookings: X, Total Amount: $Y", where X is the total number of bookings and Y is the total booking amount formatted as dollars.

Getter Methods

The class provides three getter methods to access the encapsulated data:

• get_total_bookings(): Returns the total number of bookings as an integer.
• get_total_amount(): Returns the total booking amount as a Decimal type.
• get_timestamp(): Returns the timestamp associated with the booking data as a string.

By encapsulating the booking data within the BookingInfo class, the code is more organized, readable, and reusable. This structured approach simplifies the handling of booking information throughout the application, making it more intuitive to work with and present the data.

How to Convert the Data into an Excel Sheet

Now that you can retrieve data from the database for a specific time range, you can also generate an Excel sheet based on the extracted data.

To do this, let's define another private method to create the Excel sheet.

...
import pandas as pd

from booking_info import BookingInfo


...

class DataExporter:

    ...

    def __convert_to_excelsheet(self, data: list, sheet_name: str):
        """
        Convert the fetched data into an Excel sheet.

        Args:
            data (list): A list containing booking data.
            sheet_name (str): Name of the Excel sheet to be created.

        Raises:
            ValueError: If there is an error in converting data to an Excel sheet.
        """
        try:
            booking_info = BookingInfo(data)
            data = {
                "": ["Total Bookings", "Total Amount ($)"],
                booking_info.get_timestamp(): [
                    booking_info.get_total_bookings(),
                    booking_info.get_total_amount(),
                ],
            }
            logging.info("Converting the data into pandas dataframe")
            df = pd.DataFrame(data)
            logging.info("Inserting the data into the excelsheet")
            with pd.ExcelWriter(sheet_name, engine="xlsxwriter") as writer:
                df.to_excel(writer, sheet_name="Sheet1", index=False)
            logging.info("Successfully inserted data into the excelsheet")
        except ValueError as e:
            logging.error("Error converting data into excel: %s", e)

The __convert_to_excelsheet method within the DataExporter class is responsible for structuring and converting extracted booking data into an Excel sheet.

It accepts two input parameters. The first parameter, data, is expected to be a list containing specific booking data. This data includes the total number of bookings, the total booking amount, and a timestamp for which data was extracted. The second parameter, sheet_name, represents the desired name for the Excel sheet that will contain the formatted data.

A key aspect of the method is the structuring of the data. To achieve this, the method initiates the creation of a BookingInfo object, referred to as booking_info. The BookingInfo object provides a structured representation of the booking data, which simplifies the subsequent formatting and presentation.

Following the creation of the booking_info object, a new dictionary called data is generated. This dictionary is designed to structure the data in a format suitable for conversion into an Excel sheet.

The dictionary consists of two key-value pairs:

• The first pair uses an empty string as the key and contains a list with two header values, "Total Bookings" and "Total Amount ($)".
• The second pair uses the timestamp obtained from booking_info.get_timestamp() as the key and includes a list with two elements: the total number of bookings (booking_info.get_total_bookings()) and the total booking amount (booking_info.get_total_amount()).

This dictionary allows the data to be inserted in the excel sheet as below:

Sample Excel Sheet

Then, the structured data dictionary is converted into a pandas DataFrame, referred to as df. Dataframes are a commonly used data structures for handling tabular data in Python. This step streamlines the manipulation and export of the data for further processing or visualization.

To create the Excel sheet, the code uses the pd.ExcelWriter context manager with the "xlsxwriter" engine. This context manager ensures that the Excel file is appropriately prepared for data insertion. The sheet_name parameter is supplied to specify the name of the sheet within the Excel file.

The data within the DataFrame, df, is then written to the Excel sheet. The to_excel method is used in conjunction with the writer object, and the index parameter is set to False. This specific configuration excludes the default row numbers that are typically included in Excel sheets.

How to Combine the Functionalities

Now let's write a public method that the users can use to extract the data from the database and convert the extracted data into the Excel sheet file.

...


class DataExporter:

    ...

    def generate_excelsheet(
        self,
        start_timestamp: datetime,
        end_timestamp: datetime,
        sheet_name: str = "Bookings Data.xlsx",
    ) -> bool:
        """
        Generate an Excel sheet with booking data for a specified time range.

        Args:
            start_timestamp (datetime): The start of the time range.
            end_timestamp (datetime): The end of the time range.
            sheet_name (str, optional): Name of the Excel sheet to be created. Defaults to "Bookings Data.xlsx".

        Returns:
            bool: True if excelsheet was generated successfully else False

        Note:
            This method logs errors but does not raise exceptions to avoid breaking the workflow.
        """
        data = self.__fetch_from_database(start_timestamp, end_timestamp)
        if data is not None:
            self.__convert_to_excelsheet(data, sheet_name)
            return True
        else:
            logging.error("No data to convert generate excelsheet")
            return False

This method accepts several parameters, including start_timestamp and end_timestamp, which define the beginning and end of the time period for data extraction. There's also an optional sheet_name parameter that allows the user to specify the name of the Excel sheet. By default, the sheet is named "Bookings Data.xlsx" to provide a convenient default option.

Upon execution, the method initiates the data retrieval process by calling the __fetch_from_database method, an internal private method of the class, with the specified time range.

If the data retrieval is successful and data is available, the method proceeds to call the __convert_to_excelsheet method. This structures and formats the data for insertion into the Excel sheet.

If, on the other hand, no data is available for the provided time range, the method logs an error message and returns "False" to indicate that the Excel sheet generation was unsuccessful.

How to Send an Email with the Bookings Data Report

In this section, you will learn how you can use Python to send an email with a bookings data report as an attachment.

Create a mailer.py file and add the following content:

import logging
import os
import smtplib
import ssl

from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

logging.basicConfig(
    format="%(asctime)s | %(levelname)s : %(message)s", level=logging.INFO
)

SMTP_SERVER = os.environ.get("SMTP_SERVER")
PORT = os.environ.get("EMAIL_PORT")
EMAIL = os.environ.get("EMAIL")
PASSWORD = os.environ.get("PASSWORD")


def send_email(to_email: str, subject: str, attachment_name: str):
    """
    Send an email with an attachment to the specified recipient.

    Args:
        to_email (str): The recipient's email address.
        subject (str): The subject of the email.
        attachment_name (str): The filename of the attachment.

    Note:
        This function assumes that the SMTP server requires TLS encryption.

    Raises:
        smtplib.SMTPException: If there is an issue with sending the email.

    """
    message = MIMEMultipart()
    message["From"] = EMAIL
    message["To"] = to_email
    message["Subject"] = subject
    body = "Hi there\n\nPlease find attached your report.\n\nThanks"

    message.attach(MIMEText(body, "plain"))

    with open(attachment_name, "rb") as file:
        part = MIMEBase(
            "application", "vnd.openxmlformats-officedocument.spreadsheetml.sheet"
        )
        part.set_payload(file.read())

    encoders.encode_base64(part)

    part.add_header(
        "Content-Disposition",
        f"attachment; filename= {attachment_name}",
    )

    logging.info(f"Attaching {attachment_name} to the email")
    message.attach(part)
    text = message.as_string()

    context = ssl.create_default_context()
    with smtplib.SMTP(SMTP_SERVER, PORT) as server:
        logging.info(f"Sending email to {to_email}")
        server.starttls(context=context)
        server.login(EMAIL, PASSWORD)
        server.sendmail(EMAIL, to_email, text)
        logging.info(f"Successfully sent the email to {to_email}")

As usual, we have configured the logger and environment variables in our script.

The core functionality is encapsulated within the send_email function. This function takes three parameters:

1. to_email: The recipient's email address.
2. subject: The subject of the email.
3. attachment_name: The filename of the attachment, which should be the bookings data report in this context.

Within the function, we construct an email message using the MIMEMultipart class. This message includes the sender's email address, recipient's email address, subject, and a plain text body with a simple message.

The script allows attaching the bookings data report as an attachment. It reads the attachment file, encodes it, and adds it to the email message. This ensures that the recipient can easily access and download the data report from the email.

You can learn how you can add attachments while sending emails using Python here.

The create_default_context function from the ssl library creates a secure SSL context for email communication. Finally, the script connects to the SMTP server, logs in using the sender's email address and password, sends the email, and logs a success message upon successful transmission.

How to Test the Flow

Let's finally test the flow of the application.

In this section, we will automate the monthly reports. Create a main.py file and add the following content:

from exporter import DataExporter
from datetime import datetime
from mailer import send_email

start_timestamp = datetime(2023, 5, 28, 00, 00, 00)  # May 28 2023 00:00:00
end_timestamp = datetime(2023, 8, 20, 23, 59, 59)  # Aug 20 2023 23:59:59

exporter = DataExporter()
if exporter.generate_excelsheet(
        start_timestamp, end_timestamp, sheet_name="Bookings Data.xlsx"):
    send_email("myemail@gmail.com", "Your Report", "Bookings Data.xlsx")

In the above code, we create two timestamp objects, start_timestamp and end_timestamp, to specify a time range. We have the start date set to May 28, 2023 at midnight and the end date set to August 20, 2023 just before midnight.

Next, we create an instance of the DataExporter class, which handles the data export and Excel sheet generation. The generate_excelsheet method of this instance is called with the previously defined timestamps to create a report related to bookings.

Finally, the code sends an email with the generated Excel sheet as an attachment using the send_email function.

How to Schedule the Application

Next, our goal is to automate the report scheduling process. We aim to schedule report deliveries for two distinct scenarios: on every Monday for the previous week's data, and on the 1st day of every month for the previous month's information.

To schedule the execution, you will need to install the schedule library:

pip install schedule

Once the library is installed, here's how you can do automate the monthly and weekly reports:

import schedule
from exporter import DataExporter
from datetime import datetime, timedelta
from mailer import send_email


def main():
    today = datetime.now()
    sheet_name = "Bookings Data.xlsx"

    if today.weekday() == 0:  # Check if it's Monday (0 means Monday)
        # It's Monday, fetch data for the previous week (Monday to Sunday)
        start_timestamp = (today - timedelta(days=7)
                           ).replace(hour=0, minute=0, second=0, microsecond=0)
        end_timestamp = (today - timedelta(days=1)
                         ).replace(hour=23, minute=59, second=59, microsecond=0)
        sheet_name = "Weekly Report.xlsx"
    elif today.day == 29:
        # It's the 1st day of the month, fetch data for the last month
        start_timestamp = (today.replace(day=1) - timedelta(days=1)
                           ).replace(day=1, hour=0, minute=0, second=0, microsecond=0)
        end_timestamp = (today.replace(day=1) - timedelta(days=1)
                         ).replace(hour=23, minute=59, second=59, microsecond=0)
        sheet_name = "Monthly Report.xlsx"

    exporter = DataExporter()
    exporter.generate_excelsheet(
        start_timestamp, end_timestamp, sheet_name)

    send_email("youremail@gmail.com",
               "Your Report", sheet_name)


schedule.every().day.at("00:00").do(main)

while True:
    schedule.run_pending()

The above script uses the schedule library to run the main function daily at midnight. The main function calculates the timestamps for data extraction and Excel sheet generation. After generating the Excel sheet, the script sends it via email to a specified recipient.

If the script runs on a Monday, it sets up to generate a weekly report. It calculates the start_timestamp and end_timestamp for the previous week. The start_timestamp is set to the previous Monday at midnight (00:00:00), and the end_timestamp is set to the previous Sunday just before midnight (23:59:59). The Excel sheet is named "Weekly Report.xlsx."

On the 1st day of the month, the script shifts its focus to generating a monthly report. It calculates the start_timestamp and end_timestamp to encompass the entire previous month. The start_timestamp is set to the first day of the previous month at midnight (00:00:00), while the end_timestamp is set to the last day of the previous month just before midnight (23:59:59). The Excel sheet is named "Monthly Report.xlsx."

Wrapping Up

In this tutorial, you learned how you can leverage Python to automate generating a report and sending it to email recipients. I hope you found the tutorial helpful!

Future Scope

• You can add the email recipients in a database and fetch their list from there instead of hardcoding them in the code itself. This will make the application more configurable.
• You can also use Cron Jobs to automate the execution of the script every day at midnight. In that case, you won't need the schedule library.

Here's a link to the Github Code Repository.

Throughout the course of the article, we will:

1. Utilize the Medusa Next.js Starter Template along with the Digital Products Recipe build the store.
2. Enhance the product pages to suit digital products. This involves adding a button for previewing media content and displaying essential product details.
3. Refine the checkout process to make it more efficient for delivering digital products.
4. Create Next.js API routes to validate and conceal file paths for product downloads.

Demo of the final application

Table of Contents

1. What is Medusa?
2. Prerequisites
3. Starting Out
4. How to Set Up TypeScript Type Definitions
5. How to Incorporate e-Book Previews into the Product Details
6. How to Offer e-Book Previews
7. How to Adjust the Product and Shipping Details
8. How to Simplify the Checkout
9. How to Deliver Digital Products

Let's get started.

What is Medusa?

Medusa is a suite of tools and modules specifically designed for e-Commerce products.

Using Medusa, you can build modularized commerce logic like carts, products, and order management. It also provide tools that help you orchestrate powerful ecommerce websites, POS applications, commerce-enabled products, and everything in between.

Prerequisites

Before you get started with the tutorial, you should have installed:

• Node.js(V14 or later)
• Git
• Medusa CLI

Starting Out

Using the Next.js starter, you can create a new Medusa app by running the following command:

npx create-medusa-app@latest --with-nextjs-starter

After that, you can opt to create a user account for admin panel access. Then, set up the backend infrastructure following the Medusa Digital Products Recipe.

Once the backend is set, create sample products through your Medusa admin interface. Make sure that these products include digital media files for previews and primary content. Also make sure to incorporate relevant product metadata values using key/value pairs linked to each product.

How to Set Up TypeScript Type Definitions

If you’re using regular JavaScript, you can skip this step.

Before we continue, let's make sure to add in the necessary TypeScript type definitions for digital products in the Next.js storefront.

import { Product } from "@medusajs/medusa"
import { ProductVariant } from "@medusajs/product"

export enum ProductMediaVariantType {
  PREVIEW = "preview",
  MAIN = "main",
}

export type ProductMedia = {
  id: string
  name?: string
  file?: string
  mime_type?: string
  created_at?: Date
  updated_at?: Date
  attachment_type?: ProductMediaVariantType
  variant_id?: string
  variants?: ProductMediaVariant[]
}

export type ProductMediaVariant = {
  id: string
  variant_id: string
  product_media_id: string
  type: string
  created_at: Date
  updated_at: Date
}

export type DigitalProduct = Omit<Product, "variants"> & {
  product_medias?: ProductMedia[]
  variants?: DigitalProductVariant[]
}

export type DigitalProductVariant = ProductVariant & {
  product_medias?: ProductMedia
}

      throw err
    })

  return product_medias[0]
}

This code defines TypeScript types and interfaces for managing digital products and their associated media files in an e-commerce system. It introduces several crucial structures:

1. ProductMedia: This interface describes media files related to a product. These files can include images, documents, or any digital assets. It encompasses properties such as an id (a unique identifier for the media), name (an optional name for the media), file (representing the file path or URL), mime_type (the type of media, e.g., image/jpeg), created_at and updated_at timestamps, and attachment_type that categorizes the media as "preview" or "main." Additionally, a media item can have multiple variants, making it adaptable for various use cases.
2. ProductMediaVariant: This interface represents different variants or versions of a product's media. Each variant has its unique id, variant_id (relating it to a specific product variant), product_media_id (linking it to a particular media item), and timestamps for created_at and updated_at.
3. DigitalProduct: It extends the standard Product type by introducing an array called product_medias. This array enables the association of media files with a digital product, allowing the presentation of images or other media related to the product. The variants property is tailored for digital products, adapting the generic ProductVariant to digital product-specific requirements.
4. DigitalProductVariant: This type, an extension of ProductVariant, allows the linking of media files with a specific variant of a digital product. This is particularly valuable for showcasing different digital assets associated with each variant of the product.

How to Incorporate e-Book Previews into the Product Details

Now, let's move forward by adding e-book previews to our product detail page. To do this, we'll get the media previews linked to the currently selected product variant.

In the src/lib/data/index.ts file, we'll create a function to get these previews based on the chosen variant.

// ... other imports
import { DigitalProduct, ProductMedia } from "types/product-media"

// ... rest of the functions

export async function getProductMediaPreviewByVariant(
  variant: Variant
): Promise<ProductMedia> {
  const { product_medias } = await medusaRequest("GET", `/product-media`, {
    query: {
      variant_ids: variant.id,
      expand: ["variants"],
    },
  })
    .then((res) => res.body)
    .catch((err) => {
      throw err
    })

  return product_medias[0]
}

This function is responsible for fetching information related to a specific product variant. It does this by making an HTTP request to the /product-media endpoint. It takes one argument, variant, which is expected to be of type Variant. The request includes query parameters specifying the variant_ids and requests additional details about related "variants".

The function awaits the response from the HTTP request and extracts the response body, which is assumed to be an array of product media objects. It then returns the first product media object from this array, presuming there is at least one such object. If an error occurs during the request, it catches the error and rethrows it.

How to Offer e-Book Previews

To give customers a glimpse of the e-book's content, we'll provide a preview PDF with the first few pages.

To do this, we'll set up a Next API route to manage file downloads while keeping the file's location private. We'll also create a component for a straightforward "download free preview" button. If a product variant has preview media, it will be shown in the product-actions component.

You can use the newly created DigitalProduct and DigitalProductVariant types to fix any TypeScript errors that you may encounter.

import { NextRequest, NextResponse } from "next/server"

export async function GET(req: NextRequest) {
  // Get the file info from the URL
  const { filepath, filename } = Object.fromEntries(req.nextUrl.searchParams)

  // Fetch the PDF file
  const pdfResponse = await fetch(filepath)

  // Handle the case where the PDF could not be fetched
  if (!pdfResponse.ok) return new NextResponse("PDF not found", { status: 404 })

  // Get the PDF content as a buffer
  const pdfBuffer = await pdfResponse.arrayBuffer()

  // Define response headers
  const headers = {
    "Content-Type": "application/pdf",
    "Content-Disposition": `attachment; filename="${filename}"`, // This sets the file name for the download
  }

  // Create a NextResponse with the PDF content and headers
  const response = new NextResponse(pdfBuffer, {
    status: 200,
    headers,
  })

  return response
}

import Button from "@modules/common/components/button"
import { ProductMedia } from "types/product-media"

type Props = {
  media: ProductMedia
}

const ProductMediaPreview: React.FC<Props> = ({ media }) => {
  const downloadPreview = () => {
    window.location.href = `${process.env.NEXT_PUBLIC_BASE_URL}/api/download/preview?filepath=${media.file}&filename=${media.name}`
  }

  return (
    <div>
      <Button variant="secondary" onClick={downloadPreview}>
        Download free preview
      </Button>
    </div>
  )
}

export default ProductMediaPreview

The GET function is designed to handle incoming HTTP GET requests using the Next.js framework. It first extracts information from the request URL, specifically the filepath and filename, which are expected to be query parameters. It then attempts to fetch a PDF file from the specified filepath. If the PDF is successfully retrieved, it proceeds to convert the PDF content into a buffer.

In case the PDF retrieval fails, for instance, if the file is not found, it returns a response with a "PDF not found" message and a 404 status code, indicating a not found error.

If the PDF is successfully fetched, it defines response headers, specifying that the content type is "application/pdf" and setting the "Content-Disposition" header to control the behavior of file downloads. The Content-Disposition header is set to "attachment," and the filename parameter is used to suggest a filename for the downloaded PDF.

import Button from "@modules/common/components/button"
import { ProductMedia } from "types/product-media"

type Props = {
  media: ProductMedia
}

const ProductMediaPreview: React.FC<Props> = ({ media }) => {
  const downloadPreview = () => {
    window.location.href = `${process.env.NEXT_PUBLIC_BASE_URL}/api/download/preview?filepath=${media.file}&filename=${media.name}`
  }

  return (
    <div>
      <Button variant="secondary" onClick={downloadPreview}>
        Download free preview
      </Button>
    </div>
  )
}

export default ProductMediaPreview

The above component displays a preview of a product's media along with a button to download a free preview of that media. The component receives a prop named media, which is expected to be of type ProductMedia.

Inside the component, there's a downloadPreview function that's called when a user clicks the "Download free preview" button. This function constructs a URL for downloading the preview using the window.location.href property. It combines the base URL from the environment variable NEXT_PUBLIC_BASE_URL with the "/api/download/preview" route and includes query parameters for the file path and file name, which are extracted from the media prop.

// ...other imports
import ProductMediaPreview from "../product-media-preview"
import { getProductMediaPreviewByVariant } from "@lib/data"

const ProductActions: React.FC<ProductActionsProps> = ({ product }) => {
    // ...other code

  const [productMedia, setProductMedia] = useState({} as ProductMedia)

  useEffect(() => {
    const getProductMedia = async () => {
      if (!variant) return
      await getProductMediaPreviewByVariant(variant).then((res) => {
        setProductMedia(res)
      })
    }
    getProductMedia()
  }, [variant])

  return (
            // ...other code

      {productMedia && <ProductMediaPreview media={productMedia} />}

      <Button onClick={addToCart}>
        {!inStock ? "Out of stock" : "Add to cart"}
      </Button>
    </div>
  )
}

export default ProductActions

This component is responsible for displaying product-related actions, such as adding a product to the cart, and showing product media preview if available. It leverages asynchronous operations to fetch the media data based on the provided variant, making it a dynamic and interactive component.

How to Adjust the Product and Shipping Details

Because product and shipping information differs between digital and physical products, we'll make changes to these sections on the product page as needed.

How to Add Product Details

I've added product details to the e-book using the product's metadata section in the Medusa admin. Since we're not using the standard attributes, we'll enhance the ProductInfoTab component to display any additional metadata we include.

By default, metadata is structured as an object. To make it simpler to create our list of attributes, we'll change it into an array.

In this case, we'll feature four attributes from the metadata, splitting them into two columns. If you want to show a different number of attributes, you can easily adjust the values within the slice() function as needed.

// ... other components

const ProductInfoTab = ({ product }: ProductTabsProps) => {
  // map the metadata object to an array
  const metadata = useMemo(() => {
    if (!product.metadata) return []
    return Object.keys(product.metadata).map((key) => {
      return [key, product.metadata?.[key]]
    })
  }, [product])

  return (
    <Tab.Panel className="text-small-regular py-8">
      <div className="grid grid-cols-2 gap-x-8">
        <div className="flex flex-col gap-y-4">
                {/* Map the metadata as product information */}
          {metadata &&
            metadata.slice(0, 2).map(([key, value], i) => (
              <div key={i}>
                <span className="font-semibold">{key}</span>
                <p>{value}</p>
              </div>
            ))}
        </div>
        <div className="flex flex-col gap-y-4">
          {metadata.length > 2 &&
            metadata.slice(2, 4).map(([key, value], i) => {
              return (
                <div key={i}>
                  <span className="font-semibold">{key}</span>
                  <p>{value}</p>
                </div>
              )
            })}
        </div>
      </div>
      {product.tags?.length ? (
        <div>
          <span className="font-semibold">Tags</span>
        </div>
      ) : null}
    </Tab.Panel>
  )
}

// ... other components

How to Adjust the Shipping Details

Shipping information isn't relevant for digital products, so we'll change the content in this tab. You can make any necessary adjustments to the content within the ShippingInfoTab component in the same file to better match your store's requirements.

// ... other components

const ProductTabs = ({ product }: ProductTabsProps) => {
  const tabs = useMemo(() => {
    return [
      {
        label: "Product Information",
        component: <ProductInfoTab product={product} />,
      },
      {
        label: "E-book delivery",
        component: <ShippingInfoTab />,
      },
    ]
  }, [product])
    // ... rest of code
}

// ... other components

const ShippingInfoTab = () => {
  return (
    <Tab.Panel className="text-small-regular py-8">
      <div className="grid grid-cols-1 gap-y-8">
        <div className="flex items-start gap-x-2">
          <FastDelivery />
          <div>
            <span className="font-semibold">Instant delivery</span>
            <p className="max-w-sm">
              Your e-book will be delivered instantly via email. You can also
              download it from your account anytime.
            </p>
          </div>
        </div>
        <div className="flex items-start gap-x-2">
          <Refresh />
          <div>
            <span className="font-semibold">Free previews</span>
            <p className="max-w-sm">
              Get a free preview of the e-book before you buy it. Just click the
              button above to download it.
            </p>
          </div>
        </div>
      </div>
    </Tab.Panel>
  )
}

// ... other components

The ProductTabs component is used for rendering a set of tabs. The component takes a product prop, and it uses the useMemo hook to create an array of tab objects. Each tab object consists of a label and a component to be displayed when that tab is active.

In the above snippet, there are two tabs: "Product Information" and "E-book delivery." The "Product Information" tab displays information about the product using the ProductInfoTab component, which we defined earlier.

The "E-book delivery" tab uses the ShippingInfoTab component to display information related to e-book delivery. Inside the ShippingInfoTab component, it provides details about the delivery process, mentioning instant delivery via email and the option to download from an account, as well as free e-book previews.

Product Page

How to Simplify the Checkout

Selling digital products doesn't require gathering customers' physical addresses. We only need their first name and email address to deliver the e-book, making the checkout process simpler by removing unnecessary input fields.

In this example, we'll keep only the first name, last name, country, and email fields, completely removing the billing address section. Keep in mind that your specific requirements may require different input fields.

To start, we'll adjust the checkout types and context by removing any references to values that are no longer needed.

"use client"

import { medusaClient } from "@lib/config"
import useToggleState, { StateType } from "@lib/hooks/use-toggle-state"
import { Cart, Customer, StorePostCartsCartReq } from "@medusajs/medusa"
import Wrapper from "@modules/checkout/components/payment-wrapper"
import { isEqual } from "lodash"
import {
  formatAmount,
  useCart,
  useCartShippingOptions,
  useMeCustomer,
  useRegions,
  useSetPaymentSession,
  useUpdateCart,
} from "medusa-react"
import { useRouter } from "next/navigation"
import React, { createContext, useContext, useEffect, useMemo } from "react"
import { FormProvider, useForm, useFormContext } from "react-hook-form"
import { useStore } from "./store-context"

type AddressValues = {
  first_name: string
  last_name: string
  country_code: string
}

export type CheckoutFormValues = {
  shipping_address: AddressValues
  billing_address?: AddressValues
  email: string
}

interface CheckoutContext {
  cart?: Omit<Cart, "refundable_amount" | "refunded_total">
  shippingMethods: { label?: string; value?: string; price: string }[]
  isLoading: boolean
  readyToComplete: boolean
  sameAsBilling: StateType
  editAddresses: StateType
  initPayment: () => Promise<void>
  setAddresses: (addresses: CheckoutFormValues) => void
  setSavedAddress: (address: AddressValues) => void
  setShippingOption: (soId: string) => void
  setPaymentSession: (providerId: string) => void
  onPaymentCompleted: () => void
}

const CheckoutContext = createContext<CheckoutContext | null>(null)