Performance Optimization - freeCodeCamp.org

How to Optimize Django REST APIs for Performance: Profiling, Caching, and Scaling.

Mari — Tue, 17 Feb 2026 18:22:09 +0000

Performance problems in APIs rarely start as performance problems. They usually start as small design decisions that worked perfectly when the application had ten users, ten records, or a single developer testing locally. Over time, as traffic increases and data grows, those same decisions begin to slow everything down.

In this article, we’ll walk step by step through how performance issues arise in Django REST APIs, how to see them clearly using profiling tools, and how to fix them using query optimization, caching, pagination, and basic scaling strategies.

This article will be most useful for developers who already understand Django, the Django REST Framework, and REST concepts, but are new to performance optimization.

Why Django REST APIs Become Slow

Before optimizing anything, it’s important to understand why APIs become slow in the first place.

Most performance issues in Django REST APIs come from three main sources:

Too many database queries
Doing expensive work repeatedly
Returning more data than necessary

Django is fast by default, but it does exactly what you ask it to do. If your API endpoint triggers 300 database queries, Django will happily run all 300.

Now let’s look at some common causes of performance issues in Django REST APIs.

1. N+1 Query Problems in Serializers

This happens when you loop over objects and access related fields, causing a separate query for each object.

# models.py
class Author(models.Model):
    name = models.CharField(max_length=100)

class Post(models.Model):
    title = models.CharField(max_length=200)
    author = models.ForeignKey(Author, on_delete=models.CASCADE)

# views.py (naive approach)
posts = Post.objects.all()
for post in posts:
    # This triggers a query per post to fetch the author
    print(post.author.name)

If you have 100 posts, this runs 101 queries: 1 for posts and 100 for authors. Django lazily loads related objects by default, so without intervention, your API performs repetitive database work that slows response times.

# Naive queryset fetching all related objects separately
posts = Post.objects.all()
authors = [post.author for post in posts]  # triggers extra queries per post

Each access to post.author triggers a new query. Even though you already fetched all posts, Django lazily loads related objects by default. This creates many extra queries, slowing down your API.

3. Serializing Large Datasets Without Pagination

Returning large query sets all at once can slow down your API and increase memory usage.

# views.py
from rest_framework.response import Response
from rest_framework.decorators import api_view
from .models import Post
from .serializers import PostSerializer

@api_view(['GET'])
def all_posts(request):
    posts = Post.objects.all()  # retrieves all posts at once
    serializer = PostSerializer(posts, many=True)
    return Response(serializer.data)

If your database has thousands of posts, this endpoint fetches everything in memory, serializes it, and sends it over the network. It’s slow and can crash under load. Later, we’ll learn to paginate results efficiently.

4. Recomputing Expensive Work Repeatedly

Some endpoints calculate the same values on every request instead of caching or precomputing.

def expensive_view(request):
    # Simulate expensive computation
    result = sum([i**2 for i in range(1000000)])
    return JsonResponse({"result": result})

Even if the data doesn’t change often, this computation happens on every request, consuming CPU time unnecessarily.

Performance optimization is about reducing unnecessary work.

At this point, it might be tempting to jump straight into fixes like caching responses or optimizing database queries. But doing that without evidence often leads to wasted effort or even new problems.

Before changing anything, you need to understand where your API is actually spending time. Is it the database? Is it serialization? Is it Python code running repeatedly on every request? This is where profiling becomes essential.

Profiling: Finding the Real Bottlenecks

Optimizing without profiling is guessing. Profiling helps you answer one question:

Where is my API actually spending time?

In practice, profiling means observing an API while it runs and collecting data about what it’s doing. This includes how many database queries are executed, how long those queries take, and how much time is spent in Python code, such as serializers or business logic.

By profiling first, you avoid making assumptions and can focus on fixing the parts of your API that are truly slowing things down.

Measuring Query Count in a View

During development, Django keeps track of all executed queries. You can inspect them directly:

from django.db import connection
from rest_framework.decorators import api_view
from rest_framework.response import Response
from .models import Post
from .serializers import PostSerializer

@api_view(["GET"])
def post_list(request):
    posts = Post.objects.all()
    serializer = PostSerializer(posts, many=True)

    response = Response(serializer.data)

    print(f"Total queries executed: {len(connection.queries)}")

    return response

If this prints 101 queries for 100 posts, you likely have an N+1 problem. This simple check confirms whether the database layer is the bottleneck.

One of the easiest ways to profile Django applications during development is by using tools that expose this information directly while requests are being processed.

The Django Debug Toolbar is one of the simplest ways to understand performance during development. It acts as a lightweight profiling tool that shows what happens behind the scenes when a request is handled.

It shows you:

How many SQL queries were executed
How long each query took
whether queries are duplicated
Which parts of the request lifecycle are slow

First, install it:

pip install django-debug-toolbar

In settings.py:

INSTALLED_APPS = [
    ...
    "debug_toolbar",
]

MIDDLEWARE = [
    ...
    "debug_toolbar.middleware.DebugToolbarMiddleware",
]

INTERNAL_IPS = [
    "127.0.0.1",
]

In urls.py:

import debug_toolbar
from django.urls import path, include

urlpatterns = [
    ...
    path("__debug__/", include(debug_toolbar.urls)),
]

When you load an endpoint in the browser during development, the toolbar displays total SQL queries, execution time, and duplicate queries. This makes inefficiencies immediately visible.

When you load an API endpoint and see 150 SQL queries for a single request, that’s a strong signal that something is wrong, often an N+1 query problem or inefficient serializer behavior.

Logging SQL Queries

Django allows you to log all executed SQL queries. This is especially useful when debugging API views.

Seeing the raw SQL makes inefficiencies obvious, such as repeated SELECT statements for the same table.

How to Enable SQL Query Logging

You can configure Django to log all SQL queries in settings.py:

LOGGING = {
    "version": 1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
    },
    "loggers": {
        "django.db.backends": {
            "handlers": ["console"],
            "level": "DEBUG",
        },
    },
}

With this configuration, every SQL query will be printed to the console when your API runs. Repeated SELECT statements or unexpected queries become obvious.

Profiling API Response Time

Database queries are only one part of API performance. Beyond queries, it’s also important to measure the total response time of an endpoint.

Profiling response time helps you understand whether delays are caused by database access or by other parts of the request lifecycle. For example, if an endpoint takes 1.2 seconds to respond but only 50 milliseconds are spent on database queries, the bottleneck is likely in serialization, business logic, or repeated computations in Python.

By comparing query time and total response time, profiling helps you identify what to fix first instead of optimizing the wrong layer of the system.

How to Measure Total Response Time

import time
from rest_framework.decorators import api_view
from rest_framework.response import Response

@api_view(["GET"])
def example_view(request):
    start_time = time.time()

    # Simulate work
    data = {"message": "Hello world"}

    response = Response(data)

    end_time = time.time()
    print(f"Response time: {end_time - start_time:.4f} seconds")

    return response

If database queries are fast but the total response time is high, the bottleneck may be serialization or expensive Python logic.

Once you’ve identified that database access is a significant contributor to slow response times, the next step is to look more closely at how Django retrieves related data.

SQL Query Optimization in Django REST APIs

One of the most common reasons Django REST APIs become slow is inefficient access to related objects. This often manifests as the N+1 query problem, where fetching related objects triggers a separate database query for each item. Identifying and fixing this problem can significantly reduce the number of queries and improve API performance.

Understanding the N+1 Query Problem

Consider a simple example:

You fetch a list of posts
Each post has an author
For every post, Django fetches the author separately

If you have 100 posts, this results in 101 queries: 1 for the posts and 100 for the authors. This happens because Django lazily loads related objects by default. Without intervention, your API performs repetitive database work that slows down response times.

Solving the Problem with `select_related` and `prefetch_related`

Django provides built-in tools to control how related objects are loaded efficiently: select_related and prefetch_related.

1. Using select_related

select_related is designed for foreign key and one-to-one relationships. It performs an SQL join and retrieves related objects in a single query.

Use it when:

You know you will access related objects
The relationship is one-to-one or many-to-one

posts = Post.objects.select_related("author")

for post in posts:
    print(post.author.name)  # No additional queries

This performs a SQL JOIN and retrieves posts and authors in a single query, eliminating the N+1 problem.

It reduces multiple queries into just one, avoiding repeated database hits.

2. Using prefetch_related

prefetch_related is used for many-to-many and reverse foreign key relationships. It performs separate queries for each related table but combines the results in Python.

Use it when:

A SQL join would produce too much duplicated data
You are dealing with collections of related objects

Example: How to Optimize a Many-to-Many Relationship

Consider a blog application where posts can have multiple tags:

# models.py
class Tag(models.Model):
    name = models.CharField(max_length=50)

class Post(models.Model):
    title = models.CharField(max_length=200)
    tags = models.ManyToManyField(Tag)

Now imagine fetching posts and accessing their tags:

posts = Post.objects.all()

for post in posts:
    print(post.tags.all())  # Triggers additional queries

If you have 100 posts, Django may execute:

1 query to fetch posts
1 query per post to fetch related tags

This results in many unnecessary database hits.

You can optimize this using prefetch_related:

posts = Post.objects.prefetch_related("tags")

for post in posts:
    print(post.tags.all())  # Uses prefetched data

With this approach, Django performs one query for posts and one query for all related tags. It then matches them in Python, eliminating repeated database queries.

Together, these tools allow you to optimize your queries and eliminate the N+1 problem efficiently.

Common Beginner Mistakes

Even after applying these optimizations, it’s easy to make mistakes. Watch out for:

Forgetting that serializers can trigger additional queries
Using select_related on many-to-many relationships
Assuming Django automatically optimizes queries
Not checking the query count after adding serializers

Paying attention to these pitfalls ensures your API remains fast and scalable.

Caching in Django REST APIs

Even after optimizing database queries, API performance can still suffer if the same computations or database lookups are performed repeatedly. This is where caching comes in. Caching is a technique for storing the results of expensive operations so they can be retrieved more quickly the next time they are needed.

At its core, caching exists because computers have multiple layers of memory with different speeds:

CPU registers (fastest)
L1, L2, L3 caches
Main memory (RAM)
SSD storage
HDD storage (slowest)

Each layer trades speed for size: the closer the data is to the CPU, the faster it can be accessed. Software systems use the same principle; by storing frequently accessed data in a “closer” or faster location, applications can respond more quickly.

Cache Eviction

Caches are limited in size, so when a cache is full, some data must be removed to make room for new data. This process is called cache eviction.

Common eviction strategies include:

Least Recently Used (LRU): removes the data that hasn’t been accessed for the longest time
Random Replacement: removes a random item from the cache

The goal is to keep the data that is most likely to be requested again while freeing space for new data. Understanding this helps developers use caching effectively.

Caching in Application Architectures

Caching exists at several levels in modern software systems:

Client-side caching: Web browsers cache HTTP responses to reduce the need for repeated network requests. This is controlled with HTTP headers like Cache-Control.
CDN caching: Content Delivery Networks store static assets closer to users, reducing latency and server load.
Backend caching: Backend services cache results from database queries, computed values, or API responses. This is where Django caching is most commonly applied.

By applying caching strategically at the backend, APIs can serve data faster while reducing computation and database load.

Caching in Django

Django provides a flexible caching framework that supports multiple backends, including in-memory, file-based, database-backed, and third-party stores like Redis. The main types of caching in Django are:

Per-view caching: caches the entire output of a view. Ideal for endpoints where responses rarely change.
```
 from django.views.decorators.cache import cache_page

 @cache_page(60 * 15)  # cache for 15 minutes
 def my_view(request):
```
1. Template fragment caching: caches specific parts of a template to avoid repeated rendering.
2. Low-level caching: gives full control over what is cached and for how long, making it ideal for API responses.

By combining these approaches, you can reduce repeated work in your API, lower database load, and speed up response times.

When to Use Redis

While Django’s built-in caching backends are sufficient for many projects, high-traffic APIs often require a shared, in-memory cache. This is where Redis excels. Redis is designed for fast access, low latency, and can handle frequent reads across multiple servers.

You should consider using Redis when:

Data is read frequently but changes infrequently
Low latency is important for API responses
You need cache expiration and eviction policies
You want a shared cache across multiple servers or services

Redis is particularly effective for API endpoints that serve the same data to many users, such as frequently accessed lists or computed results.

Common Beginner Mistakes

Caching is powerful, but it’s easy to misuse. Some common pitfalls include:

Caching everything blindly: not all data benefits from caching
Forgetting cache invalidation: stale data can lead to incorrect responses
Using cache where query optimization would suffice: sometimes optimizing database queries is a better solution than caching.

Remember: caching should complement good database design, not replace it.

Pagination and Limiting Expensive Datasets

Even with caching, returning large datasets in a single request can slow down your API and increase memory usage. Pagination is a simple and effective way to limit the amount of data returned at once.

Pagination helps by reducing:

Database load
Memory usage
Serialization time
Network transfer size

Django REST Framework provides built-in pagination classes that make it easy to paginate endpoints. As a rule of thumb, always paginate list endpoints unless there is a strong reason not to.

Load Testing and Measuring Improvement

Optimizations are only meaningful if you can measure their impact. Load testing simulates multiple users accessing your API simultaneously, helping you answer key questions:

How many requests per second can my API handle?
Where does the API start to break under load?
Did caching, query optimization, and pagination actually improve performance?

By running load tests before and after optimization, you can validate that your changes have the desired effect and avoid optimizing the wrong parts of your system.

Summary and Next Steps

Optimizing Django REST APIs isn’t about chasing every tiny micro-optimization. It’s about reducing unnecessary work and focusing on the parts of your API that actually slow down performance.

Key Takeaways

Profile before optimizing: Identify the real bottlenecks before making changes.
Reduce database queries: Use techniques like select_related, prefetch_related, and avoid N+1 queries.
Cache frequently accessed data: Use Django caching and Redis to reduce repeated computations.
Paginate large datasets: Limit memory usage and network load by returning data in chunks.
Measure performance changes: Always verify that your optimizations have a real impact.

Next Steps for Your APIs

Add profiling to your existing APIs to understand where time is spent.
Identify one slow endpoint and focus on optimizing it first.
Optimize database queries using Django ORM best practices.
Introduce caching carefully; avoid caching everything blindly.
Measure the results with load testing and performance metrics.

Remember: Performance optimization is not a one-time task. It’s a habit built by continuously observing how your system works, testing improvements, and applying changes where they make the most impact.

How to Use Python's Built-in Profiling Tools: Examples and Best Practices

Vivek Sahu — Tue, 25 Mar 2025 16:11:21 +0000

Python is known for its simplicity and readability, making it a favorite among developers. But this simplicity sometimes comes at the cost of performance. When your Python application grows or needs to handle larger workloads, understanding what's happening under the hood becomes crucial.

While many developers reach for third-party profiling tools, Python's standard library already comes packed with powerful profiling capabilities that are often overlooked or underutilized.

In this article, you'll learn how to use these built-in profiling tools beyond their basic usage. You'll discover how to combine and leverage them to gain deep insights into your code's performance without installing additional packages.

Prerequi site s
The Bui l t-in Profiling A r senal
- The timeit M odule
- The cProfile Module
- T h e pstats M odule
- The profile Module
P ractical Ex periments
Best Practic es
Conclu s ion
References a nd Further R e ading

Prerequisites

Before diving into the profiling techniques, make sure you have:

Python 3.6+: All examples in this article are compatible with Python 3.6 and newer versions.
Basic Python Knowledge: You should be comfortable with Python fundamentals like functions, modules, and basic data structures.
A Test Environment: Either a local Python environment or a virtual environment where you can run the code examples.

No external libraries are required for this tutorial as we'll be focusing exclusively on Python's built-in profiling tools. You can verify your Python version this way:

# Verify your Python version
import sys
print(f"Python version: {sys.version}")

The Built-in Profiling Arsenal

Python ships with several profiling tools in its standard library. Let's explore each one and understand their various strengths.

The `timeit` Module

Most Python developers are familiar with the basic timeit usage:

import timeit

# Basic usage
execution_time = timeit.timeit('"-".join(str(n) for n in range(100))', number=1000)
print(f"Execution time: {execution_time} seconds")

# Sample output:
# Execution time: 0.006027 seconds

This basic example measures how long it takes to join 100 numbers into a string with hyphens. The number=1000 parameter tells Python to run this operation 1,000 times and return the total execution time, which helps average out any random fluctuations.

However, timeit offers much more flexibility than most developers realize. Let's explore some powerful ways to use it:

Setup Code Separation:

setup_code = """
data = [i for i in range(1000)]
"""

test_code = """
result = [x * 2 for x in data]
"""

execution_time = timeit.timeit(stmt=test_code, setup=setup_code, number=100)
print(f"Execution time: {execution_time} seconds")

# Sample output:
# Execution time: 0.001420 seconds

In this example, we separate the setup code from the code being timed. This is extremely useful when:

You need to create test data but don't want that time included in your measurement
You're timing a function that relies on imports or variable definitions
You want to reuse the same setup for multiple timing tests

The advantage is that only the code in test_code is timed, while the setup runs just once before the timing begins.

Comparing Functions:

def approach_1(data):
    return [x * 2 for x in data]

def approach_2(data):
    return list(map(lambda x: x * 2, data))

data = list(range(1000))

time1 = timeit.timeit(lambda: approach_1(data), number=100)
time2 = timeit.timeit(lambda: approach_2(data), number=100)

print(f"Approach 1: {time1} seconds")
print(f"Approach 2: {time2} seconds")
print(f"Ratio: {time2/time1:.2f}x")

# Sample output:
# Approach 1: 0.001406 seconds
# Approach 2: 0.003049 seconds
# Ratio: 2.17x

This example demonstrates how to compare two different implementations of the same functionality. Here we're comparing:

A list comprehension approach
A map() with lambda approach

By using lambda functions, we can pass existing data to our functions when timing them. This directly measures real-world scenarios where your functions are working with existing data. The ratio calculation makes it easy to understand exactly how much faster one approach is than the other.

In this case, we can see the list comprehension is about 2.17 times faster than the map approach for this specific operation.

The `cProfile` Module

cProfile is Python's C-based profiler that provides detailed statistics about function calls. Many developers use it with its default settings:

import cProfile

def my_function():
    total = 0
    for i in range(100000):  # Reduced for faster execution
        total += i
    return total

cProfile.run('my_function()')

# Sample output:
#          4 function calls in 0.002 seconds
#
#    Ordered by: standard name
#
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#         1    0.000    0.000    0.002    0.002 :1()
#         1    0.002    0.002    0.002    0.002 :1(my_function)
#         1    0.000    0.000    0.002    0.002 {built-in method builtins.exec}
#         1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This basic example runs the profiler on a simple function that sums the numbers from 0 to 99,999. The output provides several key pieces of information:

ncalls: How many times each function was called
tottime: The total time spent in the function (excluding time in subfunctions)
percall: Time per call (tottime divided by ncalls)
cumtime: Cumulative time spent in this function and all subfunctions
filename:lineno(function): Where the function is defined

This gives you a comprehensive view of where time is being spent in your code, but there's much more you can do with cProfile.

The real power comes from advanced usage techniques:

Sorting Results:

import cProfile
import pstats

# Profile the function
profiler = cProfile.Profile()
profiler.enable()
my_function()
profiler.disable()

# Create stats object
stats = pstats.Stats(profiler)

# Sort by different metrics
stats.sort_stats('cumulative').print_stats(10)  # Top 10 functions by cumulative time
stats.sort_stats('calls').print_stats(10)       # Top 10 functions by call count
stats.sort_stats('time').print_stats(10)        # Top 10 functions by time

# Sample output for cumulative sorting:
#          2 function calls in 0.002 seconds
#
#    Ordered by: cumulative time
#
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#         1    0.002    0.002    0.002    0.002 :1(my_function)
#         1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This example demonstrates how to control the profiling process and sort the results in different ways. The advantages are:

You can enable/disable profiling around specific sections of code
You can sort results by different metrics to identify different types of bottlenecks:
- cumulative: Find functions that consume the most time overall (including subfunctions)
- calls: Find functions called most frequently
- time: Find functions with the highest self-time (excluding subfunctions)
Limit output to only the top N results with print_stats(N)

This flexibility lets you focus on specific performance aspects of your code.

Filtering Results:

stats.strip_dirs().print_stats()  # Remove directory paths for cleaner output
stats.print_stats('my_module')   # Only show results from my_module

# Sample output with strip_dirs():
#          2 function calls in 0.002 seconds
#
#    Random listing order was used
#
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#         1    0.002    0.002    0.002    0.002 :1(my_function)
#         1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

These filtering techniques are invaluable when working with larger applications:

strip_dirs() removes directory paths, making the output much more readable
print_stats('my_module') filters results to only show functions from a specific module, letting you focus on your code rather than library code

This is particularly useful when profiling large applications where the full output might include hundreds or thousands of function calls.

The `pstats` Module

The pstats module is often overlooked but provides powerful ways to analyze profiling data:

Saving and Loading Profile Data:

import cProfile
import pstats

# Save profile data to a file
cProfile.run('my_function()', 'my_profile.stats')

# Load and analyze later
stats = pstats.Stats('my_profile.stats')
stats.strip_dirs().sort_stats('cumulative').print_stats(10)

# Sample output:
# Wed Mar 20 14:30:00 2024    my_profile.stats
#
#          4 function calls in 0.002 seconds
#
#    Ordered by: cumulative time
#
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#         1    0.000    0.000    0.002    0.002 {built-in method builtins.exec}
#         1    0.000    0.000    0.002    0.002 :1()
#         1    0.002    0.002    0.002    0.002 :1(my_function)
#         1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

This example shows how to save profiling data to a file and load it later for analysis. The key advantages are:

You can collect profiling data in one session or environment and analyze it in another
You can share profiling data with team members without them needing to run the code
You can save profiling data from production environments where interactive analysis might not be possible
You can compare different runs over time to track performance improvements

This approach separates data collection from analysis, making it more flexible for real-world applications.

Combining Multiple Profiles:

stats = pstats.Stats('profile1.stats')
stats.add('profile2.stats')
stats.add('profile3.stats')
stats.sort_stats('time').print_stats()

# This allows you to combine results from multiple profiling runs,
# useful for aggregating data from different test cases or scenarios

This powerful feature lets you combine results from multiple profiling runs. This is useful for:

Comparing performance across different inputs
Aggregating data from multiple test scenarios
Combining data from different parts of your application
Building a more comprehensive performance picture across multiple runs

By combining stats from multiple runs, you can identify patterns that might not be apparent from a single profiling session.

The `profile` Module

The profile module is a pure Python implementation of the profiler interface. While it's slower than cProfile, it can be more flexible for specific cases:

import profile

def my_function():
    total = 0
    for i in range(100000):  # Using 100000 for faster execution
        total += i
    return total

profile.run('my_function()')

# Sample output:
#          5 function calls in 0.011 seconds
#
#    Ordered by: standard name
#
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#         1    0.000    0.000    0.002    0.002 :0(exec)
#         1    0.009    0.009    0.009    0.009 :0(setprofile)
#         1    0.000    0.000    0.002    0.002 :1()
#         1    0.000    0.000    0.011    0.011 profile:0(my_function())
#         0    0.000             0.000          profile:0(profiler)
#         1    0.002    0.002    0.002    0.002 :1(my_function)

The profile module works similarly to cProfile but offers advantages in specific scenarios:

It's implemented in pure Python, making it easier to modify if you need custom profiling behavior
You can subclass and extend it to implement custom profiling logic
It allows more fine-grained control over the profiling process
It's useful for profiling in environments where the C extension might not be available

While it's slower than cProfile (because it's implemented in Python rather than C), its flexibility makes it valuable for specialized profiling needs.

The profile module follows the same API as cProfile, so you can use all the same techniques for analyzing results.

Practical Experiments

Let's put these tools to practical use with some experiments.

Setup

First, create a simple Python module with various functions to profile:

# profiling_example.py

import time
import random

def process_data(data):
    result = []
    for item in data:
        result.append(process_item(item))
    return result

def process_item(item):
    # Simulate processing time
    time.sleep(0.0001)  # Small delay for demonstration purposes
    return item * 2

def generate_data(size):
    return [random.randint(1, 100) for _ in range(size)]

def process_data_optimized(data):
    return [process_item(item) for item in data]

def main():
    data = generate_data(50)
    result1 = process_data(data)
    result2 = process_data_optimized(data)
    assert result1 == result2
    return result1

if __name__ == "__main__":
    main()

Experiment 1: Basic vs Advanced `timeit` Usage

Let's compare different ways of timing our functions:

import timeit
from profiling_example import generate_data, process_data, process_data_optimized

# Method 1: Basic string evaluation (limited but simple)
setup1 = """
from profiling_example import generate_data, process_data
data = generate_data(5)  # Using a small size for demonstration
"""
basic_time = timeit.timeit('process_data(data)', setup=setup1, number=5)
print(f"Basic timing: {basic_time:.4f} seconds")

# Method 2: Using lambda for better control
data = generate_data(5)
advanced_time = timeit.timeit(lambda: process_data(data), number=5)
print(f"Advanced timing: {advanced_time:.4f} seconds")

# Method 3: Comparing implementations
data = generate_data(5)
original_time = timeit.timeit(lambda: process_data(data), number=5)
optimized_time = timeit.timeit(lambda: process_data_optimized(data), number=5)
print(f"Original implementation: {original_time:.4f} seconds")
print(f"Optimized implementation: {optimized_time:.4f} seconds")
print(f"Improvement ratio: {original_time/optimized_time:.2f}x")

# Sample output:
# Basic timing: 0.0032 seconds
# Advanced timing: 0.0034 seconds
# Original implementation: 0.0033 seconds
# Optimized implementation: 0.0034 seconds
# Improvement ratio: 0.98x

This experiment demonstrates three different approaches to timing code with the timeit module:

Method 1: Basic string evaluation – This approach evaluates a string of code after running the setup code. The advantages include:

Simple syntax for basic timing needs
Setup code runs only once, not during each timing run
Good for timing simple expressions

Method 2: Lambda functions – This more advanced approach uses lambda functions to call our functions directly. Benefits include:

Direct access to functions and variables in the current scope
No need to import functions in setup code
Better for timing functions that take arguments
More intuitive for complex timing scenarios

Method 3: Implementation comparison – This practical approach compares two different implementations of the same functionality. This is valuable when:

Deciding between alternative implementations
Measuring the impact of optimizations
Quantifying performance differences with a ratio

In this example, the list comprehension isn't significantly faster because the dominant cost is the time.sleep() call in both implementations. In real-world cases with actual computation instead of sleep, the difference is often more pronounced.

Experiment 2: Effective `cProfile` Analysis

Now let's use cProfile to identify bottlenecks:

import cProfile
import pstats
import io
from profiling_example import main

# Method 1: Basic profiling
cProfile.run('main()')

# Sample output snippet:
#         679 function calls in 0.014 seconds
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#      100    0.000    0.000    0.014    0.000 profiling_example.py:10(process_item)
#      100    0.014    0.000    0.014    0.000 {built-in method time.sleep}
#        1    0.000    0.000    0.007    0.007 profiling_example.py:4(process_data)
#        1    0.000    0.000    0.007    0.007 profiling_example.py:18(process_data_optimized)

# Method 2: Capturing and analyzing results
profiler = cProfile.Profile()
profiler.enable()
main()
profiler.disable()

# Redirect output to string for analysis
s = io.StringIO()
stats = pstats.Stats(profiler, stream=s).sort_stats('cumulative')
stats.print_stats(10)  # Print top 10 functions by cumulative time
print(s.getvalue())

# Method 3: Focus on specific functions
s = io.StringIO()
stats = pstats.Stats(profiler, stream=s).sort_stats('cumulative')
stats.print_callers('process_item')  # Show what's calling this function
print(s.getvalue())

# Sample output for the callers analysis:
# Function was called by...
#       ncalls  tottime  cumtime
# profiling_example.py:10(process_item)  <-  
#     50    0.000    0.007  profiling_example.py:4(process_data)
#     50    0.000    0.007  profiling_example.py:18(process_data_optimized)

This experiment demonstrates three powerful cProfile techniques for identifying bottlenecks:

Method 1: Basic profiling – Using cProfile.run() to profile a function call provides an immediate overview of performance. This technique:

Gives you a quick snapshot of all function calls
Shows precisely where time is being spent
Is easy to use with minimal setup
Identifies the most time-consuming operations

In our example, we can immediately see that time.sleep() is consuming most of the execution time.

Method 2: Programmatic profiling and analysis – This approach gives you more control:

You can enable/disable profiling for specific sections of code
You can save the results to a variable for further analysis
You can customize how results are sorted and displayed
You can redirect output to a string or file for post-processing

This method is particularly useful for profiling specific parts of a larger application.

Method 3: Caller analysis – The print_callers() method is extremely valuable because it:

Shows which functions are calling your bottleneck functions
Helps identify which code paths are contributing to performance issues
Reveals how many times each caller invokes a particular function
Provides context that's crucial for understanding performance patterns

In our example, we can see that both process_data and process_data_optimized are calling process_item 50 times each, confirming they're contributing equally to the bottleneck.

This immediate shows us that process_item is the bottleneck, specifically the time.sleep() call inside it, and it's being called equally by both implementations.

Experiment 3: Combining Tools for Real-world Profiling

In real-world scenarios, combining profiling tools gives the most complete picture:

import cProfile
import pstats
import timeit
from profiling_example import main, process_data, process_data_optimized, generate_data

# First, use timeit to get baseline performance of main components
data = generate_data(50)
time_process = timeit.timeit(lambda: process_data(data), number=3)
time_process_opt = timeit.timeit(lambda: process_data_optimized(data), number=3)
print(f"Process data: {time_process:.4f}s")
print(f"Process data optimized: {time_process_opt:.4f}s")

# Sample output:
# Process data: 0.0196s
# Process data optimized: 0.0194s

# Then, use cProfile for deeper insights
profiler = cProfile.Profile()
profiler.enable()
main()
profiler.disable()

# Save stats for later analysis
profiler.dump_stats('profile_results.stats')

# Load and analyze
stats = pstats.Stats('profile_results.stats')
stats.strip_dirs().sort_stats('cumulative').print_stats(10)

# Sample output:
# Wed Mar 20 14:30:00 2024    profile_results.stats
#          659 function calls in 0.013 seconds
#    Ordered by: cumulative time
#    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
#         1    0.000    0.000    0.013    0.013 profiling_example.py:21(main)
#       100    0.000    0.000    0.013    0.000 profiling_example.py:10(process_item)
#       100    0.013    0.000    0.013    0.000 {built-in method time.sleep}

This experiment demonstrates a comprehensive real-world profiling strategy that combines multiple tools:

First step: High-level timing with timeit – We start with timeit to:

Get baseline performance metrics for specific functions
Compare different implementations directly
Measure overall execution time
Identify which high-level components might need optimization

This gives us a quick overview that both implementations take about the same time, confirming our earlier findings.

Second step: Detailed profiling with cProfile – Next, we use cProfile to:

Get a function-by-function breakdown of execution time
Identify specific bottlenecks at a granular level
See the number of calls to each function
Understand the call hierarchy

Third step: Saving and analyzing with pstats – Finally, we:

Save profiling data to a file for persistence
Load the data and apply filtering/sorting
Focus on the most time-consuming functions
Get a clean, readable output

This multi-tool approach provides several advantages:

You get both high-level and detailed insights
You can save profiling data for later comparison
You can share results with team members
You can track performance changes over time

In our example, we confirm that our main bottleneck is the time.sleep() call inside process_item, which accounts for most of the execution time. Without this combined approach, we might have missed important details or wasted time optimizing the wrong parts of our code.

This approach gives you both high-level timing information and detailed profiling data, allowing for a comprehensive performance analysis.

Best Practices

Based on our experiments, here are some best practices for effective profiling:

Start with the right tool for the job:
- Use timeit for quick, targeted measurements of specific functions or code blocks
- Use cProfile for comprehensive program analysis when you need to understand how all parts of your code interact
- Use pstats for in-depth analysis of profiling data when you need to filter, sort, and interpret complex profiling results

For example, if you're just trying to decide between two implementations of a sorting algorithm, timeit is sufficient. But if you're trying to understand why your entire web application is slow, start with cProfile.

Profile realistic workloads:
- Synthetic benchmarks often mislead because they don't reflect real-world usage patterns
- Use production-like data sizes to see how your code scales with realistic inputs
- Run multiple iterations to account for variance and ensure your results are reliable

A function that's fast with 10 items might be painfully slow with 10,000. Always test with data sizes that match your production needs.

Focus on the right metrics:
- cumulative time shows the total time spent in a function and all its calls. It’s useful for finding the overall most expensive operations.
- tottime shows time spent only in the function itself. It’s useful for finding inefficient implementations.
- ncalls helps identify functions called excessively. It’s useful for finding redundant operations.

For example, a function with a small tottime but large cumulative time might be efficient itself but is calling expensive subfunctions.

Save profiling data for comparison:
- Use profiler.dump_stats() to save data from different versions of your code
- Compare before and after optimization to quantify improvements
- Track performance over time to catch regressions early

This practice helps you prove that your optimizations are actually working and prevents performance from degrading over time.

Look for the 80/20 rule:
- 80% of time is often spent in 20% of the code. Focus optimization efforts on these "hot spots"
- Focus optimization efforts on the functions with the highest cumulative time.
- Don't optimize what isn't slow – premature optimization wastes time and can make code more complex.

For example, in our experiments, the time.sleep() call was the clear bottleneck. Optimizing anything else would be pointless until that's addressed.

By following these practices, you'll make the most efficient use of your profiling tools and focus your optimization efforts where they'll have the greatest impact.

Conclusion

Python's built-in profiling tools offer a powerful arsenal for identifying and resolving performance bottlenecks in your code. By leveraging the timeit, cProfile, and pstats modules effectively, you can get deep insights into your application's performance without relying on third-party tools.

Each tool serves a specific purpose:

timeit helps you measure execution time of specific code snippets
cProfile gives you a comprehensive view of function calls and execution time
pstats lets you analyze, filter, and interpret profiling data
profile provides a customizable profiling interface for special cases

The greatest power comes from combining these tools, as we demonstrated in our practical experiments. This allows you to approach performance optimization systematically:

Identify high-level performance concerns with timeit
Drill down into specific bottlenecks with cProfile
Analyze and interpret results with pstats
Make targeted optimizations based on data, not guesswork

Remember that profiling is as much an art as it is a science. The goal isn't just to make code faster, but to understand why it's slow in the first place. With the techniques demonstrated in this article, you're well-equipped to tackle performance challenges in your Python applications.

Apply these profiling techniques to your own code, and you'll be surprised at what you discover. Often, the bottlenecks aren't where you expect them to be!

References and Further Reading

Python timeit documentation: The official documentation for the timeit module, with detailed explanations of all parameters and functions.
Python cProfile documentation: Comprehensive guide to profiling modules, including both cProfile and profile.
Python pstats documentation: Detailed reference for the pstats module, explaining all methods for analyzing profiling data.
Python Profilers - Official Documentation: The complete official documentation on Python's profiling capabilities.
Python Speed - Performance Tips: A collection of practical tips for optimizing Python code once you've identified bottlenecks.
The Python Profilers: In-depth explanation of profiling in Python, including details on overhead and accuracy.
Django Optimization Techniques: Practical advice on optimizing Django ORM code in real-world applications.
How Python Magic Methods Work: A Practical Guide: My previous article on FreeCodeCamp covering practical deep-dive on Python Magic methods.

How to Optimize Next.js Web Apps for Better Performance

Ayantunji Timilehin — Thu, 02 Jan 2025 14:30:59 +0000

As engineers, we often get so carried away with other aspects of development that we overlook how users perceive and interact with our applications. This oversight can result in users leaving the app almost as soon as they arrive, leading to higher bounce rates and minimal engagement.

At its core, every business thrives on delivering value to its users. When users are unable to access this value due to poor performance, it ultimately impacts the business's success. Slow load times, among other factors, frustrate users and drive them away before they even get a chance to engage.

Optimizing performance is more than just a technical detail – it’s also a critical part of creating a successful application. Without it, even the best features can go unnoticed if users don’t stick around long enough to see them.

In this article, we’ll explore key approaches to optimize your Next.js application, making it faster and more efficient.

Building a Performant Application
How to Optimize Your Applications
Key Techniques To Optimize Performance
Conclusion

Building a Performant Application

Making your apps more performant means striking the right balance between speed, responsiveness, and efficient use of resources. You should strive to create an application that delivers value and keeps users satisfied.

Building a performant app is about making sure the app feels smooth and intuitive so that there are no frustrating lags when a user clicks buttons, scrolls, or navigates around. You’ll also want to make sure that data loads or updates without unnecessary delays.

How to Optimize Your Applications

The first step in optimizing your application is identifying problem areas. A number of tools and packages can help you analyze your application's performance effectively. Here's how you can use them:

Using `npm run build`

When you run npm run build, Next.js creates a production-ready version of your application and gives a detailed breakdown of your pages. This includes:

Size: The size of the JavaScript files for each route. Highlighting any routes that are too large and could slow things down. Smaller page sizes generally result in faster load times while large pages might take longer to download, especially for users with slower network connections.
First Load Js: This column provides information about the total amount of JavaScript the browser needs to download and execute to fully render the page for the first time. Large First Load JS values

cause Slower Time-to-Interactive (TTI).

Running this command produces an analysis like below:

Using `@next/bundle-analyzer`

The bundle analyzer is a package provided by Next.js to analyze the size of JavaScript bundles by providing a visual representation of the application’s module and dependencies. Here’s how to use the package:

First, install the package by running this command:

npm install @next/bundle-analyzer

Or you can use yarn:

yarn add @next/bundle-analyzer

Then add the @next/bundle-analyzer configuration to your next.config.js file:

const withBundleAnalyzer = require('@next/bundle-analyzer')({
  enabled: process.env.ANALYZE === 'true',
});

module.exports = withBundleAnalyzer({
  // other Next.js config options here
});

To analyze your application bundles while generating a production build, run the following command:

ANALYZE=true npm run build

For a step-by-step guide on how to use the bundle analyzer effectively, check out this detailed video tutorial

Browser tools

Finally, modern browsers, including Google Chrome, Firefox, and Edge, offer powerful tools to analyze and improve your application's performance. Features like the Performance Tab help you record and visualize how your application runs, pinpointing issues like slow rendering or long tasks.

You can also use tools like Lighthouse (available in Chrome and Edge) to generate automated audits, highlighting problems such as large assets and unoptimized resources.

To access the Lighthouse and Performance tabs:

Open your browser's developer tools by right-clicking anywhere on the browser and selecting the Inspect option or pressing Command + Option + I (on Mac) or Ctrl + Shift + I (on Windows).
Look at the top menu in the developer tools.
If you don’t see the Lighthouse or Performance tabs right away, click the double right arrow (>>) to reveal hidden tabs.
Select the desired tab to start analyzing performance or generating a Lighthouse report.

Here is an example of a generated audit in the Performance tab on Chrome

Here’s another image showing the generated audit by lighthouse

Key Techniques to Optimize Performance

1.) Using The Next.js `Image` Component

Images often account for the largest portion of page weight, directly affecting load times and user experience. Large images slow down rendering and ultimately, increase bandwidth usage.

Next.js has a built-in Image component that automatically optimizes images, making it very useful for web performance. It takes care of resizing, lazy loading, and format optimization, so images are served in the most performant format (like .WebP) when the browser supports it.

import Image from 'next/image';

    <Image
      src="/house.jpg"
      alt="House Image"
      width={700}
      height={500}
      priority={false} // Lazy loads the image by default
    />

In the snippet above,

src="/house.jpg": This points to the image file's location, which is in the public folder. Images in the /public directory are served statically, so you don’t need extra configuration.
alt="House Image": The alt text (just like in the native HTML image element) provides a description of the image, which is great for accessibility (like screen readers) and also helps with SEO.
width & height: By explicitly setting the width and height, Next.js can calculate the space the image will occupy on the page before it loads. This prevents the page layout from shifting as the image loads, which improves user experience and boosts performance metrics like Cumulative Layout Shift (as shown in the image above).
priority={false}: This ensures the image will only load when it's near the user's viewport conserving the bandwidth and improving page load times for non-critical images. However, for important images that should load immediately (like those visible as soon as the page opens), you can set priority={true} to bypass lazy loading and ensure the image loads as quickly as possible.

One of the key advantages of the Next.js Image component is its built-in lazy loading feature. This means that images won’t be loaded until they are actually needed (when they enter the viewport). By only loading images that are about to be viewed, performance is improved and pages can load faster, even with many high-quality images.

2.) Optimizing Third-Party Scripts with the Next.js Script Component

Third-party scripts, such as analytics tools or advertising networks, can heavily affect your application's performance if not properly managed. Next.js has a Script component that makes it easy to load scripts efficiently, giving you control over how and when they load.

The Script component allows you to define a loading strategy for scripts, determining when and how they are fetched and executed. By prioritizing or deferring scripts based on their importance, you can improve the overall performance and user experience of your application.

beforeInteractive: Use this strategy for scripts that must load before the page becomes interactive, like essential analytics or monitoring tools.
afterInteractive: When you use this strategy, the script loads after the page becomes interactive, which is the default behavior. This is ideal for scripts that add functionality but aren’t essential for initial rendering.
lazyOnload: Defers loading the script until all other page resources have finished loading. This is perfect for non-essential scripts like ads or social media widgets.

Performance Optimization - freeCodeCamp.org

How to Optimize Django REST APIs for Performance: Profiling, Caching, and Scaling.

What we’ll cover:

Why Django REST APIs Become Slow

1. N+1 Query Problems in Serializers

2. Fetching Related Objects Inefficiently

3. Serializing Large Datasets Without Pagination

4. Recomputing Expensive Work Repeatedly

Profiling: Finding the Real Bottlenecks

Measuring Query Count in a View

Using the Django Debug Toolbar

How to Install and Enable the Django Debug Toolbar

Logging SQL Queries

How to Enable SQL Query Logging

Profiling API Response Time

How to Measure Total Response Time

SQL Query Optimization in Django REST APIs

Understanding the N+1 Query Problem

Solving the Problem with select_related and prefetch_related

Example: How to Optimize a Many-to-Many Relationship

Common Beginner Mistakes

Caching in Django REST APIs

Cache Eviction

Caching in Application Architectures

Caching in Django

When to Use Redis

Common Beginner Mistakes

Pagination and Limiting Expensive Datasets

Load Testing and Measuring Improvement

Summary and Next Steps

Key Takeaways

Next Steps for Your APIs

Read More

How to Use Python's Built-in Profiling Tools: Examples and Best Practices

Table of Contents

Prerequisites

The Built-in Profiling Arsenal

The timeit Module

The cProfile Module

The pstats Module

The profile Module

Practical Experiments

Setup

Experiment 1: Basic vs Advanced timeit Usage

Experiment 2: Effective cProfile Analysis

Experiment 3: Combining Tools for Real-world Profiling

Best Practices

Conclusion

References and Further Reading

How to Optimize Next.js Web Apps for Better Performance

Table of Contents

Building a Performant Application

How to Optimize Your Applications

Using npm run build

Using @next/bundle-analyzer

Browser tools

Key Techniques to Optimize Performance

1.) Using The Next.js Image Component

2.) Optimizing Third-Party Scripts with the Next.js Script Component

Solving the Problem with `select_related` and `prefetch_related`

The `timeit` Module

The `cProfile` Module

The `pstats` Module

The `profile` Module

Experiment 1: Basic vs Advanced `timeit` Usage

Experiment 2: Effective `cProfile` Analysis

Using `npm run build`

Using `@next/bundle-analyzer`

1.) Using The Next.js `Image` Component