In this article, you’ll learn why traditional testing methods fall short for AI systems and how an evaluation flywheel can be used as a practical approach to testing and improving AI applications. The sections below break the evaluation flywheel down step by step, from identifying the problem to implementing a repeatable evaluation loop.

Table of Contents

• Why Does Traditional Testing Fail for AI applications?

• What is the Evaluation Flywheel?

• Drawing Parallels to Familiar Practices

• Why Silent Failures Matter: A Real-World Example

• How to Create an Evaluation Flywheel

• Tools and Frameworks you can use for evaluation

• What a Complete Evaluation Loop Looks Like in Practice

• Key Takeaways

• Conclusion

Why Does Traditional Testing Fail for AI applications?

In standard programming, tests assume deterministic behavior. This means the same input is expected to always produce the same output. For example:

def authenticate_user_age(age: int) -> str:
    limit = 18

    if age >= limit:
        return "Access granted"
    else:
        return "User doesn't meet the age limit"

# Test 
assert authenticate_user_age(20) == "Access granted"
assert authenticate_user_age(16) == "User doesn't meet the age limit"

The response from this function is always predictable. You can write tests once and trust they'll catch errors forever.

However, AI models don’t behave the same way every time, they generate output based on probabilities. A query like “best programming practices” may produce strong guidance one day, and outdated or incomplete advice the next. This shift can happen because of changes in the underlying model, updates to retrieval components, or gradual data drift. Without a structured evaluation process in place, these inconsistencies slip into production unnoticed and can quietly weaken the system’s performance.

What is the Evaluation Flywheel?

The evaluation flywheel is a continuous improvement system where test cases representing real user behavior are passed through multiple evaluation steps to assess the output of AI models. The results don't just tell you whether the system passed or failed, they feed directly into the next cycle of improvement.

┌─────────────┐
│   Collect   │
│ Test Cases  │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│     Run     │
│ Evaluations │
└──────┬──────┘
       │
       ▼
┌─────────────┐      ┌─────────────┐
│  Identify   │─────▶│   Improve   │
│  Failures   │      │   System    │
└─────────────┘      └──────┬──────┘
                            │
                            ▼
                       ┌─────────────┐
                       │   Repeat    │
                       └─────────────┘

Here's how it works in practice:

• Collect test cases — Gather examples from real user interactions or create synthetic scenarios. These should reflect the kind of tasks and input your system needs to handle.

• Run evaluations — Pass each test case through a series of checks. The check can either be programmatic (automated metrics like relevance scores or hallucination detectors) or require manual review (like verifying legal advice accuracy or brand voice consistency).

• Identify failures — Detect where the model goes wrong, this can include hallucinations, irrelevant responses, or mistakes on corner-cases.

• Improve the system — Based on those failures, refine prompts, improve training or retrieval data, or adjust architectural components.

• Repeat the cycle — Re-run the updated system on the existing and newly collected cases. Over time, this grows and strengthens your evaluation suite and boosts system reliability.

Drawing Parallels to Familiar Practices

If you've written software before, the evaluation flywheel will feel familiar. It mirrors patterns that are already used in engineering. For instance,

Unit tests → Evaluation datasets
Unit tests confirm a function returns the right output. Evaluation datasets play the same role for AI: they're ground-truth queries and answers that guard against regressions.

Test-driven development (TDD) → Evaluation-driven development (EDD)
In TDD, you write tests before code. In EDD, you write evaluation cases before shipping prompts or updating models. This replaces assumptions with verifiable results.

CI/CD pipelines → Continuous evaluation pipelines
CI/CD runs checks automatically on every code change. Continuous evaluation does the same for models: it runs automated quality checks every time you tweak a prompt, retrain, or swap out a component.

The key difference is subtle but important. Traditional software tests check whether a function returns the right value or type. AI evaluation tests check whether the system produces the right meaning. That's harder to measure, but the principle is the same: build a safety net that grows stronger with every cycle.

Why Silent Failures Matter: A Real-World Example

AI systems often behave differently in production than they do in development. A model that seems solid in testing can drift, hallucinate, or silently fail when facing real-world input.

Case in point: A fraud detection model passed all monitoring metrics yet missed a spike in fraud. An ML engineer shared how their production monitoring dashboards tracked latency, throughput, and error rates, everything showed green. But fraudulent transactions were slipping through at twice the normal rate. Nobody noticed because existing observability tools focused on pipeline health, not prediction quality.

This silent failure cost the company significant losses. The system seemed fine by traditional metrics. It measured system performance—latency, throughput, uptime—but ignored what mattered most: prediction accuracy. As fraudsters adapted their tactics, the model drifted, and without proper evaluation loops, the degradation went undetected for weeks.

Source: InsightFinder.

Why This Example Matters

• Silent failures aren't always bugs — They often stem from models failing to adapt to shifting patterns in the real world.

• Static evaluation isn't enough — You need continuous, real-world feedback loops to detect when assumptions no longer hold.

• Data drift has business impact — Model degradation isn't just technical, it translates directly into revenue loss, security breaches, or damaged user trust.

How to Create an Evaluation Flywheel

To show how to build a flywheel and how it works, let's create one for a customer support chatbot that answers questions about a SaaS product.

Step 1: Build Your AI System

Create your initial product: prompts, retrieval logic, and integrations. For our chatbot:

def answer_support_question(question: str) -> str:
    # Retrieve relevant docs from knowledge base
    context = retrieve_docs(question, top_k=5)

    # Generate answer using LLM
    prompt = f"""You are a helpful customer support agent.

Context: {context}

Question: {question}

Provide a clear, accurate answer based on the context."""

    response = llm.generate(prompt)
    return response

How this works: This function defines the core chat logic, it takes a customer’s question and returns an AI-generated answer. First, it searches your knowledge base to find the five most relevant documents using retrieve_docs(). These documents provide context about your product or policies. Next, it constructs a prompt that includes this context and the user's question, then sends it to a language model. The LLM reads the context and generates a relevant answer, which the function returns.

Step 2: Identify Test Cases

Build an evaluation set that reflects real user behavior. The more representative your test cases are, including common cases, edge cases, and ambiguous inputs, the better your model can catch failures before they reach production.

Sources for test cases:

• Previous customer support tickets

• Common FAQ topics

• Edge cases discovered in beta testing

• Synthetic scenarios (hypothetical but realistic queries)

Example test cases:

test_cases = [
    {
        "question": "How do I reset my password?",
        "expected_elements": ["settings page", "reset link", "email"],
        "category": "account_management"
    },
    {
        "question": "What's your refund policy?",
        "expected_elements": ["30 days", "full refund", "contact support"],
        "category": "billing"
    },
    {
        "question": "Can I export my data to CSV?",
        "expected_elements": ["yes", "export button", "dashboard"],
        "category": "features"
    },
    {
        "question": "Does your API support webhooks?",
        "expected_elements": ["yes", "webhook endpoints", "documentation"],
        "category": "technical"
    }
]

How this works: Here, we define a set of representative test cases to evaluate the AI system. Each test case includes the user’s question, a list of key elements expected in the answer, and a category for organization. These cases help ensure the chatbot is tested against real-world scenarios, edge cases, and important information that should appear in responses.

Step 3: Evaluate Outputs

Define evaluation criteria based on what matters for your use case: accuracy, faithfulness, safety, relevance, tone. Then measure the output against these criteria.

Evaluation happens in two main ways:

Automated Evaluation

Use programmatic metrics and LLM-as-judge patterns:

def evaluate_response(question: str, response: str, expected_elements: list) -> dict:
    scores = {}

    # 1. Faithfulness: Does response contain expected elements?
    scores['contains_key_info'] = all(
        elem.lower() in response.lower() 
        for elem in expected_elements
    )

    # 2. Relevance: Semantic similarity to question
    scores['relevance'] = calculate_semantic_similarity(question, response)

    # 3. Safety: Check for problematic content
    scores['is_safe'] = not contains_harmful_content(response)

    # 4. Tone: Use LLM-as-judge
    judge_prompt = f"""Rate the helpfulness of this support response on a scale of 1-5.

Question: {question}
Response: {response}

Score (1-5):"""

    scores['helpfulness'] = int(llm.generate(judge_prompt))

    return scores

# Run evaluation
for test_case in test_cases:
    response = answer_support_question(test_case['question'])
    scores = evaluate_response(
        test_case['question'],
        response,
        test_case['expected_elements']
    )
    test_case['scores'] = scores
    test_case['response'] = response

How this works: The evaluate_response() function applies four different checks to each AI response:

• First, it verifies faithfulness by checking if all expected elements appear in the response using simple string matching.

• Second, it calculates semantic similarity, a measure of how closely the responses meaning match the intent of the questions, using embeddings.

• Third, it runs a safety check to flag any problematic content.

• Fourth, it uses an LLM as a judge by asking a more powerful model (like GPT-4) to rate the helpfulness of the response on a 1-5 scale.

The loop then runs the evaluation for every test case. It generates a response for each question, evaluates it using the evaluate_response function, and then stores both the scores and the response back in the test case. This creates a complete dataset of test results for analysis and further improvements.

Common Automated Metrics:

• Semantic similarity (0.0–1.0): This is measured by converting the question and response into vector embeddings and calculating cosine similarity. The score shows how closely the response matches the intent of the question, even if the wording differs.

• ROUGE / BLEU scores: The model’s output is compared to reference answers by checking n-gram overlap. These metrics help spot regressions, though scores can be modest for open-ended answers.

• LLM-as-judge: A stronger model (like GPT-4 or Claude) can rate the response on a fixed scale, such as 1–5. These ratings give a sense of quality and are useful for tracking improvements or drops over time.

• Retrieval metrics (Precision@k, Recall@k): For retrieval-based systems, these metrics calculate how many relevant documents appear in the top-k results. Precision shows accuracy of the retrieved set, and recall indicates completeness.

• Custom validators: Simple rule-based checks, like regex patterns, keywords, or length limits, ensure responses meet hard requirements. These help catch issues automated metrics might miss.

Manual Evaluation

Automated metrics can't capture everything. Subjective qualities like tone, empathy, and brand voice require human judgment, as do small factual errors that slip past keyword checks and similarity scores.

# Flag cases for human review
needs_review = [
    case for case in test_cases 
    if case['scores']['helpfulness'] < 3 
    or not case['scores']['contains_key_info']
]

# SMEs review and annotate
for case in needs_review:
    annotation = get_sme_feedback(case)
    case['human_rating'] = annotation['rating']
    case['improvement_notes'] = annotation['notes']

This code filters test cases to find responses that need human attention, those scoring below 3 for helpfulness or missing important information. Subject matter experts review these flagged cases and provide ratings with helpful feedback. Their input helps you spot patterns that automated metrics miss and shows you where to improve your prompts, retrieval setup, or system settings.

When to use manual evaluation:

• Assessing tone, empathy, or brand voice

• Detecting subtle hallucinations automated checks miss

• Validating edge cases with domain-specific nuance

• Creating ground truth labels for training evaluation models

Step 4: Learn and Improve

Once you've identified failures, adjust the controllable parts of your AI system (the "configs"):

Common configuration levers:

• Prompts — Add instructions, examples, constraints

• Retrieval — Change chunk size, top-k, reranking strategy

• Model — Switch models, adjust temperature, max tokens

• Context — Modify system instructions, add memory

• Post-processing — Add validation, formatting, safety filters

Example improvement cycle:

# Problem discovered: Chatbot missing key details
failing_case = {
    "question": "What's your refund policy?",
    "response": "We offer refunds in certain cases.",
    "issue": "Too vague, missing 30-day window and process"
}

# Root cause: Retrieval returning wrong docs
retrieved_docs = retrieve_docs(failing_case['question'], top_k=5)
# Docs about "payment processing" ranked higher than "refund policy"

# Solution 1: Improve retrieval with reranking
def retrieve_docs_v2(question: str, top_k: int) -> str:
    # Initial retrieval
    candidates = vector_search(question, top_k=20)

    # Rerank by relevance
    reranked = rerank_by_relevance(question, candidates)

    return reranked[:top_k]

# Solution 2: Update prompt to require specificity
prompt_v2 = f"""You are a helpful customer support agent.

Context: {context}

Question: {question}

Provide a clear, accurate answer based on the context. Include specific details like:
- Time windows (e.g., "within 30 days")
- Step-by-step processes
- Relevant links or contact methods

Answer:"""

# Re-evaluate
new_response = answer_support_question_v2(failing_case['question'])
new_scores = evaluate_response(
    failing_case['question'],
    new_response,
    ["30 days", "full refund", "contact support"]
)

# Verify improvement
assert new_scores['contains_key_info'] == True
assert new_scores['helpfulness'] >= 4

How this works: In this example, the chatbot's refund answer was too vague. After checking what went wrong, the problem was that the system retrieved docs about payment processing instead of the refund policy.

To resolve this, two changes can be made. First, retrieval is improved by grabbing twenty documents, then picking the best five. Second, the prompt is updated to ask for specific details like dates and steps.

After making these changes, the test runs again to confirm it works: the response now has all the key info and scores at least 4 out of 5. This process turns problems into fixes you can measure.

Step 5: Automate and Repeat

Integrate evaluation into your development workflow using CI/CD:

# .github/workflows/eval.yml
name: Continuous Evaluation

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Run evaluation suite
        run: python run_evals.py

      - name: Check pass rate
        run: |
          PASS_RATE=$(python calculate_pass_rate.py)
          if (( $(echo "$PASS_RATE < 0.85" | bc -l) )); then
            echo "Pass rate $PASS_RATE below threshold"
            exit 1
          fi

      - name: Upload results
        uses: actions/upload-artifact@v2
        with:
          name: eval-results
          path: results/

Explanation: This GitHub Actions workflow automates your evaluation process so it runs automatically on every code change. The workflow triggers whenever someone opens a pull request or pushes code to the main branch. It checks out your code, runs your full evaluation suite using run_evals.py, then calculates what percentage of test cases passed. If the pass rate drops below 85%, the workflow fails and blocks the code from being merged, preventing quality regressions from reaching production.

Key practices for automation:

• Version your test cases — Track them in Git alongside code

• Set quality gates — Block deployments if pass rate drops below threshold

• Monitor trends — Track metrics over time to catch gradual drift

• Alert on regressions — Notify team when specific test cases start failing

• Sample production traffic — Continuously add real queries to eval dataset

Tools and Frameworks you can use for evaluation

Several platforms can help implement continuous evaluation. The one you choose depends on your stack and needs:

If you're building with LLMs: Try LangSmith or Braintrust first. Both handle prompt versioning, evaluation datasets, and tracing out of the box.

If you're doing traditional ML: Weights & Biases is the industry standard. If you're in the Microsoft ecosystem, PromptFlow integrates well with Azure.

If you want full control: Build custom with pytest for test execution and MLflow for tracking results. More setup, but you own the entire pipeline

What a Complete Evaluation Loop Looks Like in Practice

This walkthrough shows how a support chatbot improves after running a single cycle of evaluations. Each stage shows how evaluation signals guide improvements and lock in quality for the next release.

Next cycle:

• Add this test case to permanent suite

• Look for similar issues (other plan-related questions)

• Monitor if this pattern appears in production queries

Key Takeaways

• AI systems need continuous evaluation, not one-time testing — Models drift, data changes, and silent failures accumulate without ongoing checks.

• Build evaluation into your workflow from day one — Don't wait until production failures force you to retrofit evaluation.

• Start simple, then scale — Begin with 10-20 test cases and basic metrics. Grow your suite as you encounter edge cases.

• Automate what you can, involve humans for what you can't — Use programmatic checks for speed, SME review for nuance.

• Treat evaluation datasets as first-class artifacts — Version control them, review changes, and grow them over time.

• Make evaluation a team sport — Product, engineering, and domain experts should all contribute test cases and evaluation criteria.

Conclusion

Every developer has felt the relief of seeing "all tests passing." In AI systems, that reassurance is often misleading. A model can deploy successfully, meet performance benchmarks, and still produce incorrect, incomplete, or misleading outputs in ways traditional tests miss.

The evaluation flywheel addresses this gap by making model behavior testable in practice. Instead of assuming correctness, it forces the system to answer real questions, measures the quality of those answers, and highlights where performance degrades over time. This shifts evaluation from a one-off validation step into an ongoing part of development.

Evaluation won't eliminate uncertainty completely, but it makes failures visible before they reach users. With failures clearly exposed, teams stop guessing and start fixing based on results. This might mean adjusting prompts, improving retrieval logic, or refining evaluation criteria. Over time, this leads to AI systems that evolve in controlled ways rather than breaking silently.

Resources for further reading

• Anthropic's eval guide: https://docs.anthropic.com/en/docs/build-with-claude/develop-tests

• OpenAI's evals framework: https://github.com/openai/evals

• LangChain evaluation: https://python.langchain.com/docs/guides/evaluation

• Arize AI blog: Comprehensive resources on ML observability

As developers, we are tasked with safeguarding these sensitive bits of info, and it's important to implement strong secure measures in our applications.

Now, there are many authentication mechanisms available for securing data, like OAuth, OpenID Connect, and JSON Web Tokens (JWTs).

In this article, I'll show you how to use JWTs when securing information in APIs by integrating JWT-based authentication in a Flask application.

Here's what this article will cover:

• What is a JSON Web Token?
• How Do JWTs Work?
• How to Use JSON Web Tokens to Authenticate Flask Applications
• Install dependencies
• Create a database and user model
• Configure the application for JWT authentication
• Create protected routes
• Create a Login Function
• Conclusion

Prerequisites

To follow along with this tutorial you will need:

• An understanding of HTTP Methods
• An understanding of how to create APIs in Flask
• VS Code (or other similar) editor
• A terminal

What is a JSON Web Token?

JSON Web Tokens, or JWTs, are an authentication mechanism used to securely transmit information between a client and a server in JSON format.

This information can be verified and trusted because it is digitally signed with the HMAC algorithm or a public/private key pair using RSA or ECDSA.

The tokens are encoded into three parts, each divided by a period, like this:

Header.Payload.Signature

• Header: This defines the type of token (JWT) and the signing algorithm used.
• Payload: This carries user-specific data like user ID, username, roles, and any additional claims you want to include. This payload is encoded in Base64 for maximum security.
• Signature: This is a hashed combination of the header, the payload, and the server's secret key. It ensures the token's integrity and that any modifications to the token will be detected.

How Do JWTs Work?

To understand how JWTs work, you need to know what the tokens are meant to do. JWTs are not created to hide data but to ensure that the data being sent is authenticated. This is why the JWT is signed and encoded, not encrypted.

A JWT acts as a stateless means of transmitting data from a client to a server. This means that it doesn't store any session object in the browser, so the browser doesn't maintain a session state between requests.

Instead, JWTs use a token that is sent in a request header each time a request is made. This token confirms that the token sent is authenticated and is allowed access to make that request.

Let's look at how this happens:

1. A user attempts to log in and sends a username and password to be verified by the server.
2. The verification function carries out a check to see if there's a match in the database.
3. A JWT is then generated by the server once the user is successfully authenticated (logged in) using their information (payload), such as user ID or username, and signs it using a secret key.
4. The generated JWT is sent along as a bearer token with every request header to check if the user is authenticated to make that request.

How to Use JSON Web Tokens to Authenticate Flask Applications

To demonstrate how you can implement JWT authentication in Flask, we'll create a simple application that uses JWT for handling login functions and accessing protected routes.

1. Install the dependencies

Run this command to install the dependencies we'll need

pip install flask flask-bcrypt Flask-JWT-Extended

Next, make sure you import the dependencies and initialize your Flask application with this code:

from flask import Flask, jsonify, session, request, redirect, url_for
from flask_jwt_extended import JWTManager, create_access_token, jwt_required, get_jwt_identity, get_jwt


app = Flask(__name__)

////WRITE MAIN CODE HERE


if __name__ == "__main__":
    with app.app_context():
        app.run(debug=True)

2. Create a database and User Model

To do this, we'll use SQL-Alchemy, which is a Python SQL toolkit that makes it less complex to use SQL in Python scripts.

To set up SQL Alchemy in your application, follow these steps:

First, open your terminal or command prompt and enter the following command:

pip install sqlalchemy

This command installs SQLAlchemy in your Python environment, making it available in your project directory.

Next, configure your application to make use of your preferred Database Management System (DBMS). This tutorial will use the SQlite3 DBMS as it doesn't require a separate server:

app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///site.db'

This code snippet instructs Flask-SQLAlchemy to create and use the site.db file in your project directory as the SQLite database for the application.

Then initialize the database in your application:

db = SQLAlchemy(app)

This instance of the database acts as a bridge between the application and the database.

Now create the User Model where we'll store the user's details in this tutorial:

class User(db.Model, UserMixin):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(20), unique=True, nullable=False)
    password = db.Column(db.String(80), nullable=False)
    is_active = db.Column(db.Boolean(), default=True)
    cart = db.Column(JSON, nullable=True, default=list)  # Make cart nullable

    # Define the relationship between User and CartProducts
    cart_products = relationship('CartProducts', backref="user", lazy="dynamic")
    # Define the relationship between User and Wishlists
    wishlists = db.relationship('Wishlists', backref='user', lazy=True)

    def __repr__(self):
        return f'<User {self.username}>'

Note: You can create other models using the same syntax to represent different data entities in your application.

3. Configure the application for JWT Authentication

To implement JWT authentication in your Flask application, import the necessary libraries and set up the appropriate configurations with this code:

from flask import Flask, jsonify, request
from flask_sqlalchemy import SQLAlchemy
from flask_jwt_extended import JWTManager, create_access_token, jwt_required

app = Flask(__name__)

# Configuration
app.config['SECRET_KEY'] = 'your_strong_secret_key'
app.config["JWT_SECRET_KEY"] = 'your_jwt_secret_key'
app.config['JWT_TOKEN_LOCATION'] = ['headers']

# Database Initialization
db = SQLAlchemy(app)

# JWT Initialization
jwt = JWTManager(app)

# Rest of the application code (routes, etc.)

This code snippet imports the following components needed for our application:

• app.config['SECRET_KEY'] sets the Flask application's secret key which is used to securely sign session cookies and other security-related needs.
• app.config['JWT_SECRET_KEY'] sets the secret key used to encode and decode JWTs in for Flask-JWT operations.
• app.config['JWT_TOKEN_LOCATION'] specifies where the application should look for the JWT. Here it's set to look in the HTTP headers.

Once we've set this up, we can create the endpoints and routes that we intend to protect.

4. Create protected routes

Protected routes are the pages that we intend to keep hidden from unauthorized users.

For example, let's assume we are trying to get into a venue that's exclusive to members of a society. But a guard is securing the venue from "unauthorized users" like us. In this situation, we are the application's users, the venue is the URL we are protecting, and the guard protecting the venue is a @jwt_required decorator.

The @jwt_required decorator is used to protect specific routes that require authentication. This decorator will confirm that there's a JWT access token in the request headers before allowing access to the page:

@app.route('/get_name', methods=['GET'])
@jwt_required()
def get_name():
    # Extract the user ID from the JWT
    user_id = get_jwt_identity()
    user = User.query.filter_by(id=user_id).first()

    # Check if user exists
    if user:
        return jsonify({'message': 'User found', 'name': user.name})
    else:
        return jsonify({'message': 'User not found'}), 404

In this code snippet, we created a function that returns the name of the user after authentication. If the token is missing, invalid, or expired, the request will be denied, and typically, the server will return an HTTP 401 Unauthorized status.

5. Create a Login Page

In this endpoint, we'll create a function that accepts username and password credentials from the client request (for example, form data) and compares the credentials gotten from the user with the user data in the database. If there's a match, a JWT access token will be generated containing the user's information.

@app.route('/login', methods=['POST'])
def login():
    data = request.get_json()
    username = data['username']
    password = data['password']
    print('Received data:', username , password)

    user = User.query.filter_by(username=username).first()

    if user and bcrypt.check_password_hash(user.password, password):
        access_token = create_access_token(identity=user.id)
        return jsonify({'message': 'Login Success', 'access_token': access_token})
    else:
        return jsonify({'message': 'Login Failed'}), 401

In this example, the function checks the user's credentials against the database using bcrypt for secure password verification when a POST request is received. If the credentials are valid, the server generates a JWT for the user, allowing them to access protected routes.

Here's an example of a React form sending a POST request to the login endpoint:

import React from "react";
import axios from "axios";
import { useState } from "react";
import Footer from "./Footer";
// import "./Login.css";

function Login() {
  const [password, setPassword] = useState("");
  const [username, setUsername] = useState("");

  const handleLogin = async (event) => {
    event.preventDefault();
    const data = {
      username: username,
      Password: password,
    };

    try {
      const response = await axios.post("http://localhost:5000/login", {
        username,
        password,
      });
      localStorage.setItem("access_token", response.data.access_token);
      // Redirect to protected route
      alert("Login successful");
    } catch (error) {
      console.error(error);
      // Display error message to user
    }
  };

  const handleUsernameChange = (event) => {
    setUsername(event.target.value);
  };

  const handlePasswordChange = (event) => {
    setPassword(event.target.value);
  };

  return (
    <div >

          <form method="post" >
              <input
                type="text"
                id=""
                placeholder="Username"
                name="username"
                required
                value={username}
                onChange={handleUsernameChange}
              />
              <input
                type="text"
                id=""
                placeholder="Your email"
              />
              <input
                type="password"
                required
                placeholder="Your Password"
                name="password"
                value={password}
                onChange={handlePasswordChange}
              />
          </form>
          <button type="submit" onClick={handleLogin}>
            Log In
          </button>

    </div>
  );

}

export default Login;

In this React component, we provided a login form that uses Axios to send a POST request to the login endpoint. It manages username and password inputs using React's useState hook and submits these values once the form is submitted.

If the login is successful, it stores a JWT in local Storage. This allows the client-side application to retrieve the token easily when making authenticated requests to the server.

Conclusion

In this article, we've learned how to secure APIs with JSON Web Tokens in Flask. We covered the basics of JWTs, how they work, and provided a step-by-step process for implementing this method of authentication. This included everything from installing necessary dependencies to creating user models and protecting routes.

You can build upon this foundation, such as by adding refresh tokens, integrating with third-party OAuth providers, or handling more complex user permissions.

For example, let's say you have a button that only opens a door when pressed. This button does not have the ability to close the door, so it stays open even when it's pressed repeatedly. It simply remains in the state it was changed to by the first press.

This same logic applies to HTTP methods that are idempotent. Operating on idempotent HTTP methods repeatedly won't have any additional effect beyond the initial execution.

Understanding idempotence is important for maintaining the consistency of HTTP methods and API design. Idempotence has a significant impact on API design, as it influences how API endpoints should behave when processing requests from clients.

In this tutorial, I'll explain the concept of idempotence and the role it plays in building robust and functional APIs. You'll also learn about what safe methods are, how they relate to idempotence, and how to implement idempotency in non-idempotent methods.

Prerequisites

Before understanding and implementing idempotence in API design, it's essential to have a solid foundation in the following areas:

• RESTful Principles
• Fundamentals of HTTP methods
• API Development
• HTTP Status codes
• Basics of Web development.

Idempotence Example

Let's start off with an example of idempotence in action. We'll create a function that uses the DELETE method to delete data from a web page:

from flask import Flask, jsonify, abort

app = Flask(__name__)

web_page_data = [
   {"id": 1, "content": "Row 1 data"},
   {"id": 2, "content": "Row 2 data"},
   # Add more rows as needed
]

@app.route('/delete_row/<int:row_id>', methods=['DELETE'])
def delete_row(row_id):
   # Find the row to delete
   row_to_delete = next((row for row in web_page_data if row["id"] == row_id), None)

   if row_to_delete:
       # Simulate deletion
       web_page_data.remove(row_to_delete)
       return jsonify({"message": f"Row {row_id} deleted successfully."}), 200
   else:
       abort(404, description=f"Row {row_id} not found.")

if __name__ == '__main__':
   app.run(debug=True)

This function is expected to delete the rows chosen by the user. Now because of the idempotent nature of the DELETE method, the data will be deleted once, even when called repeatedly. But subsequent calls will return a 404 error since the data has already been deleted by the first call.

Let’s look at another example with the GET method. The GET method is used to retrieve data from a resource. Let’s create a function that uses the GET method to retrieve a username:

import requests

def get_username():
    url = 'https://api.example.com/get_username'
    try:
        response = requests.get(url)
        if response.status_code == 200:
            return response.json()['username']
        else:
            return None
    except requests.RequestException as e:
        print(f"Error occurred: {e}")
        return None

# Usage
username = get_username()
if username:
    print(f"The username is: {username}")
else:
    print("Failed to retrieve the username.")

In this example, we define the get_username() function, which sends a GET request to the API endpoint to retrieve the username. If the request is successful, we extract the username from the JSON response and return it. But if any error occurs during the request, we handle it and return None.

Now the idempotent nature of the GET method ensures that even if you call get_username() multiple times, the same username will be fetched from the API each time. The result will always be the same which is to fetch the username from the resource.

Idempotent vs. Non-Idempotent HTTP Methods:

HTTP methods play crucial roles in determining how data is fetched, modified, or created when interacting with APIs. And Idempotency is one of the important concepts that influences data consistency and reliability in the methods used .

Here's a breakdown of the different methods based on their idempotency.

Idempotent methods:

• GET
• HEAD
• PUT
• DELETE
• OPTIONS
• TRACE

Non-idempotent methods:

• POST
• PATCH
• CONNECT

Safe Methods

In our previous example, we used the GET method to retrieve a username and this had no side effect on the server. This is because it is a safe method.

A safe method is a type of method that doesn’t modify the server’s state or the resource being accessed. In other words, they perform read-only operations used to retrieve data or for resource representation.

When you make a request using a safe method, the server does not perform any operations that modify the resource's state. Like in our previous example, we retrieved the username from the webpage which is the resource without changing anything in the server.

All safe methods are automatically idempotent, but not all idempotent methods are safe. This is because while idempotent methods produce consistent results when called repeatedly, some of them may still modify the server's state or the resource being accessed.

Like in our first example, the DELETE method is idempotent, because deleting a resource multiple times will have the same effect. But it's not safe, as it changes the server's state by removing the resource.

Here’s a classification of HTTP methods based on their safe status:

Safe methods:

• GET
• OPTIONS
• HEAD

Unsafe methods:

• DELETE
• POST
• PUT
• PATCH

Why is POST not idempotent?

POST is an HTTP method that sends information to a server. When you make a POST request, you typically submit data to create a new resource or trigger a server-side action. Therefore, making the same request multiple times can result in different outcomes and side effects on the server. This can lead to duplicated data, starting server resources, and reducing performance because of the repeated action.

Unlike idempotent methods like GET, PUT, and DELETE, which have consistent results regardless of repetition, POST requests can cause changes to the server's state with each invocation.

POST requests often create new resources on the server. Repeating the same POST request will generate multiple identical resources, potentially leading to duplication.

This is similar to DELETE which is an idempotent method but not a safe method. Deleting the last entry in a collection using a single DELETE request would be considered idempotent. But if a developer creates a function that deletes the last entry, that would trigger DELETE multiple times. Subsequent DELETE calls would have different effects, as each one removes a unique entry. This would be considered non-idempotent.

How to Achieve Idempotency with Non-Idempotent Methods

Idempotency isn't only a property inherent to certain methods – it can also be implemented as a feature of a non-idempotent method.

Here are some techniques to achieve idempotency even with non-idempotent methods.

Unique Identifiers

Adding unique identifiers to every request is one of the most common techniques used to implement idempotency. It works by tracking whether the operation has already been performed or not. If it's a duplicate (a repeat request), the server knows it's already dealt with that request and simply ignores it, ensuring that no side effects occur.

Here's an example of how it works:

from uuid import uuid4

def process_order(unique_id, order_data):
    if Order.objects.filter(unique_id=unique_id).exists():
        return HttpResponse(status=409)  # Conflict
    order = Order.objects.create(unique_id=unique_id, **order_data)
    return HttpResponse(status=201, content_type="application/json")

# Example usage
post_data = {"products": [...]}
headers = {"X-Unique-ID": str(uuid4())}
requests.post("https://api.example.com/orders", data=post_data, headers=headers)

In this code snippet, we define a function called process_order that creates orders in an API, using unique identifiers to implement idempotency.

Here's a breakdown of the code:

Importing the Unique Identifier Generator:

from uuid import uuid4: The code snippet starts by importing the uuid4 function from the uuid module. This function generates unique identifiers, which are used to achieve idempotency in this code.

Defining the process_order Function:

def process_order(unique_id, order_data): This line defines a function named process_order that takes two arguments:

• unique_id: This is a string representing a unique identifier for the request. This ensures no duplicate orders are created with the same identifier.
• order_data: This is a dictionary containing the actual order data, like product information and customer details.

Checking for Existing Orders:

if Order.objects.filter(unique_id=unique_id).exists(): This line checks if an order with the same unique_id already exists in the database.

Order.objects.filter(unique_id=unique_id).exists() queries the Order model for orders with the matching unique_id and checks if any orders were found in the query result. If an order is found, it means the same request was already processed.

Handling existing orders:

return HttpResponse(status=409): If an order with the same unique_id already exists, the function immediately returns an HTTP response with status code 409 indicating a conflict. This prevents duplicate orders from being created.

Creating a new order (if unique):

order = Order.objects.create(unique_id=unique_id, **order_data ): This line only runs if no existing order is found.

Order.objects.create: creates a new object in the Order model.

unique_id=unique_id sets the unique_id attribute of the new order to the provided unique_id.

order_data: spreads the dictionary order_data as keyword arguments to the order model's constructor, setting other relevant attributes like products and customer information.

Sending a success response:

return HttpResponse(status=201, content_type="application/json"): If the order creation is successful, the function will return an HTTP response with status code 201 which shows a successful creation. It also specifies the response content type as JSON, assuming the order data might be returned in JSON format.

post_data = {"products": [...]}: an example request, defines a dictionary containing the actual order data, like a list of products.

headers = {"X-Unique-ID": str(uuid4())}: This line creates a dictionary containing a custom header named X-Unique-ID. It generates a unique identifier string using uuid4() and adds it to the header.

requests.post("https://api.example.com/orders", data=post_data, headers=headers): This line sends a POST request to the API endpoint https://api.example.com/orders with the provided post_data and headers.

How does this implement idempotence?

It does so by using a unique identifier (unique_id) for each order.

It checks if an order with the same identifier already exists in the database. If it returns true, it returns a 409 Conflict status. Otherwise, it creates a new order and responds with a 201 Created status. The unique identifier prevents duplicate orders, making the system idempotent.

Token-based Authorization

Token-based authorization is a form of authorization that assigns temporary tokens for each non-idempotent action. Once the action is completed, the token is invalidated. If the same request comes again with the same token, the server recognizes it as invalid and refuses the request, thereby preventing duplicate actions.

// Generate a unique token for this action
const token = generateToken();

fetch("https://api.example.com/create-user", {
    method: "POST",
    body: JSON.stringify({ username, password }),
    headers: {
        Authorization: `Bearer ${token}`,
        "Content-Type": "application/json",
    },
})
    .then(response => {
        // Handle successful response
        if (response.ok) {
            // Do something with the successful response
        } else {
            // Handle non-successful response
        }
    })
    .catch(error => {
        // Handle error
        console.error("Error occurred:", error);
    })
    .finally(() => {
        // Invalidate token after successful action or in case of an error
        invalidateToken(token);
    });

// Simple implementation for generating a token
function generateToken() {
    return Math.random().toString(36).substr(2);
}

// Simple implementation for invalidating a token
function invalidateToken(token) {
    // Add your logic to invalidate the token, e.g., remove it from storage
}

Here's a breakdown of the code:

Generating a unique token:

const token = generateToken(): This line calls a function named generateToken() (which is assumed to be defined elsewhere) that generates a unique token string. This token will be used for authorization and idempotency.

Sending the POST request:

fetch("https://api.example.com/create-user", { ... }): This line uses the fetch API to send a POST request to the API endpoint https://api.example.com/create-user.

method: "POST": This specifies the HTTP method as POST, indicating the intention to create a new user.

body: JSON.stringify({ username, password }): This defines the request body with user details like username and password. The data is converted to JSON format before sending.

headers: { Authorization:Bearer ${token}}: This sets the Authorization header in the request. The header value includes the generated token prefixed with "Bearer ".

Handling the Response:

.then(response => { ... }): This block defines the code to execute if the request is successful. You would handle things like storing user information or redirecting the user upon successful user creation.

.catch(error => { ... }): This block defines the code to execute if the request encounters an error. You would handle any error messages or handle specific error scenarios here.

Invalidating the Token:

invalidateToken(token): This line calls a function named invalidateToken(token) ( which is assumed to be defined elsewhere) which would likely mark the used token as invalid. This ensures the same token cannot be used for subsequent requests, adding to the idempotency guarantee.

How does this implement Idempotence?

This code snippet uses token-based authorization to implement idempotency in a POST request to create a user on an API. If a user creation request is accidentally sent multiple times, a new unique token is generated each time and used in the Authorization header.

The API server can recognize and verify the unique token, and since the user creation action has already been performed (assuming it's successful the first time), it won't create duplicate users due to subsequent identical requests.

ETag Header:

An ETag header (Entity Tag) is an HTTP header used for web cache validation and conditional requests. It is mainly used for PUT requests, that only update resources if they haven't changed since the last check.

When you want to update a resource, the server sends you its ETag which is then included in your PUT request along with the updated data. If the ETag hasn't changed (meaning the resource remains the same), the server accepts the update. But if the ETag has changed, the server rejects the update, preventing it from overwriting someone else's changes.

def update_article(article_id, content):
    # Get existing article and its ETag
    article = Article.objects.get(pk=article_id)
    etag = article.etag

    # Check if ETag matches with request header
    if request.headers.get("If-Match") != etag:
        return HttpResponse(status=409)  # Conflict

    # Update article content and generate new ETag
    article.content = content
    article.save()
    new_etag = article.etag

    # Return success response with updated ETag
    return HttpResponse(status=200, content_type="text/plain", content=new_etag)

In this code snippet, we define a function called update_article that allows you to update the content of an existing article based on its ID and new content. It implements idempotency using the ETag header technique.

Here's a step-by-step explanation of how it works;

Getting the Existing Article and its ETag:

article = Article.objects.get(pk=article_id): This line fetches the article with the provided article_id from the database using the Article model.

etag = article.etag: This line extracts the ETag value from the retrieved article object. The ETag serves as a unique identifier for the article's current state.

Checking for a Match:

if request.headers.get("If-Match") != etag: This line checks if the ETag header provided in the request matches the ETag of the retrieved article.

return HttpResponse(status=409): If the ETag doesn't match, it indicates that the article might have been updated by another request since the client retrieved its information. The function returns a 409 Conflict response, which prevents accidental data corruption.

Updating the Article Content and generating a new ETag:

article.content = content: This line updates the article's content with the new content received in the request.

article.save(): This line saves the updated article back to the database.

new_etag = article.etag: This line retrieves the new ETag generated for the updated article after saving it.

Returning the Success Response with the new ETag:

return HttpResponse(status=200, content_type="text/plain", content=new_etag): returns a successful 200 OK response, including the content type ("text/plain") and the updated ETag of the article in the response body.

How does this implement idempotence?

This code ensures that if the same update request is sent multiple times with the same ETag, the update will only be performed once, preventing duplicate updates and maintaining data consistency. The new ETag is then provided in the response to help the client keep track of the article's state for future interactions.

Conclusion

In this tutorial, we highlighted the difference between safe methods like GET, which retrieves data without side effects, and non-idempotent methods like POST, which can have different outcomes with each repetition.

We also explored techniques you can apply to achieve idempotence in non-idempotent methods, emphasizing the importance of designing APIs that prioritize consistency and reliability.