<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/"
    xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" version="2.0">
    <channel>
        
        <title>
            <![CDATA[ Mene-Ejegi Ogbemi - freeCodeCamp.org ]]>
        </title>
        <description>
            <![CDATA[ Browse thousands of programming tutorials written by experts. Learn Web Development, Data Science, DevOps, Security, and get developer career advice. ]]>
        </description>
        <link>https://www.freecodecamp.org/news/</link>
        <image>
            <url>https://cdn.freecodecamp.org/universal/favicons/favicon.png</url>
            <title>
                <![CDATA[ Mene-Ejegi Ogbemi - freeCodeCamp.org ]]>
            </title>
            <link>https://www.freecodecamp.org/news/</link>
        </image>
        <generator>Eleventy</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:37:24 +0000</lastBuildDate>
        <atom:link href="https://www.freecodecamp.org/news/author/ogbemi-ejegi/rss.xml" rel="self" type="application/rss+xml" />
        <ttl>60</ttl>
        
            <item>
                <title>
                    <![CDATA[ What Is Model Distillation? A Guide for Developers ]]>
                </title>
                <description>
                    <![CDATA[ Over time, knowledge has been passed down in various forms. Before writing was invented in ancient Mesopotamia (modern-day Iraq), laws, history, myths, songs, and other information were all transferre ]]>
                </description>
                <link>https://www.freecodecamp.org/news/what-is-model-distillation-a-guide-for-developers/</link>
                <guid isPermaLink="false">6ad9087ef5ddaeeade835143</guid>
                
                    <category>
                        <![CDATA[ llm ]]>
                    </category>
                
                    <category>
                        <![CDATA[ AI models ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                    <category>
                        <![CDATA[ pytorch ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Machine Learning ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Mene-Ejegi Ogbemi ]]>
                </dc:creator>
                <pubDate>Wed, 21 Oct 2026 18:46:22 +0000</pubDate>
                <media:content url="https://cdn.hashnode.com/uploads/covers/5e1e335a7a1d3fcc59028c64/8d829d33-57ca-4b85-84b8-13f86798afb7.png" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Over time, knowledge has been passed down in various forms. Before writing was invented in ancient Mesopotamia (modern-day Iraq), laws, history, myths, songs, and other information were all transferred from generation to generation orally. This was done either via instructions or storytelling (especially folktales and events that had happened over time).</p>
<p>But another common method of knowledge transfer is from teacher to student, whether formally or informally. We’ve seen this throughout time as well: the Bible recorded that Jesus had twelve disciples whom he taught daily from the start of his mission to his death. Socrates taught Plato. Plato went on to establish an academy in Athens, where he taught Aristotle for twenty years. Aristotle revolutionized science, logic, and politics. And the list goes on.</p>
<p>A 21st-century example of this is deep learning AI lineage. During the 20th-century AI winter, when we thought neural networks were a dead end, this teacher-student lineage kept the dream alive.</p>
<p>Geoffrey Hinton, regarded as the father of AI, ran a lab where he taught Yann LeCun, Ilya Sutskever, and others. Yann LeCun went on to invent the convolutional neural network (CNN), while Ilya Sutskever co-founded OpenAI before leaving for Safe Superintelligence Inc.</p>
<p>While the concept of knowledge transfer between humans is quite familiar, how about knowledge transfer from machine to machine? This is known as "model distillation."</p>
<p>In this article, I'm going to break down what model distillation is. We'll explore the techniques it uses, and I'll walk you through how to implement it from scratch using PyTorch.</p>
<p>By the end, you'll understand how to train smaller, more efficient models and how distillation compares to other compression techniques like quantization and pruning.</p>
<h2 id="heading-what-well-cover">What We'll Cover</h2>
<ul>
<li><p><a href="#heading-prerequisites">Prerequisites</a></p>
</li>
<li><p><a href="#heading-what-is-model-distillation">What is Model Distillation?</a></p>
<ul>
<li><a href="#heading-why-is-model-distillation-necessary">Why is Model Distillation Necessary?</a></li>
</ul>
</li>
<li><p><a href="#heading-model-distillation-techniques">Model Distillation Techniques</a></p>
<ul>
<li><p><a href="#heading-response-based-distillation">Response-Based Distillation</a></p>
</li>
<li><p><a href="#heading-feature-based-distillation">Feature-Based Distillation</a></p>
</li>
<li><p><a href="#heading-attention-distillation">Attention Distillation</a></p>
</li>
<li><p><a href="#heading-self-distillation">Self-Distillation</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-how-to-implement-knowledge-distillation-in-pytorch">How to Implement Knowledge Distillation in PyTorch.</a></p>
</li>
<li><p><a href="#heading-how-to-evaluate-the-results">How to Evaluate the Results</a></p>
</li>
<li><p><a href="#heading-other-approaches-for-reducing-model-size">Other Approaches for Reducing Model Size</a></p>
<ul>
<li><p><a href="#heading-quantization">Quantization</a></p>
</li>
<li><p><a href="#heading-pruning">Pruning</a></p>
</li>
<li><p><a href="#heading-low-rank-factorization">Low-Rank Factorization</a></p>
</li>
</ul>
</li>
<li><p><a href="#heading-conclusion">Conclusion</a></p>
<ul>
<li><a href="#heading-references">References</a></li>
</ul>
</li>
</ul>
<h2 id="heading-prerequisites">Prerequisites</h2>
<p>To follow along, you should have the following:</p>
<ul>
<li><p>A GitHub Repository: The complete implementation of the project is available <a href="https://github.com/ogbemi-ejegi/knowledge-distillation/blob/main/README.md">here</a>, and you should clone it to follow along.</p>
</li>
<li><p>Basic familiarity with Python and object-oriented programming concepts (for example, working with classes).</p>
</li>
<li><p>An overview of deep learning fundamentals. You should understand concepts like epochs, forward propagation, loss function, and so on.</p>
</li>
<li><p>PyTorch Basics: While I'll explain each code section for you, basic familiarity with PyTorch tensors and training loops will be helpful.</p>
</li>
<li><p>A development environment: You should have Python installed on your machine. I'll be using Visual Studio Code as my editor, but you can use any editor or IDE of your choice.</p>
</li>
<li><p>Hardware: For this project, we'll be using a relatively small dataset so a standard CPU will work well.</p>
</li>
</ul>
<h2 id="heading-what-is-model-distillation">What is Model Distillation?</h2>
<p>Model distillation (or knowledge distillation) is an approach that aims to pass on the learned behavior of a large, highly accurate model to a smaller, computationally efficient model that's better suited for practical deployment.</p>
<p>Distillation is a compression technique where a smaller model (known as the student) learns to mimic the output of a larger, more complex model (the teacher). The goal is for the student model to achieve a level of performance close to the teacher, but with significantly fewer parameters, making it more efficient for deployment. <a href="https://www.gov.uk/government/publications/ai-insights/ai-insights-model-distillation-html">(Source)</a></p>
<h3 id="heading-why-is-model-distillation-necessary">Why is Model Distillation Necessary?</h3>
<p>At the time of this writing, both closed-source and open-source models are trained on trillions of parameters. These are the internal configuration variables that models learn automatically from training data. This immense number of parameters makes it extremely difficult and expensive to run without proper funding. Models consume large amounts of energy and require expensive hardware.</p>
<p>With distillation, we can transfer the learnings of these frontier models to smaller, more efficient models, making them easier and cheaper to use.</p>
<h2 id="heading-model-distillation-techniques">Model Distillation Techniques</h2>
<p>The distillation technique used depends on the knowledge that's being transferred. Some methods transfer only the final prediction. Others transfer internal representation, attention maps, reasoning steps, or even the entire behavior of the teacher model.</p>
<p>These are some of the techniques:</p>
<h3 id="heading-response-based-distillation">Response-Based Distillation</h3>
<p>Originally introduced by Geoffrey Hinton in his paper <a href="https://arxiv.org/abs/1503.02531">Distilling the knowledge in a neural network</a>, this is the classic teacher-to-student distillation. Before diving deep into how it works, there are a few concepts you need to understand about model training: hard labels and soft labels.</p>
<p><strong>Hard labels</strong> deal strictly with fixed categorical targets. When a picture of a dog is fed into a model and it's asked to predict if it's a dog, cat, or horse, it predicts it's a dog, given the result as Dog(1), horse(0), and cat(0). There's no probability as to what percentage or features of the picture look like a cat or a horse.</p>
<p>But we also have <strong>soft labels</strong>, which deal with probability. When the same picture is fed to a model using soft labels, it can predict that the picture has a 0.85 probability of being a dog, a 0.10 probability of being a cat, and a 0.5 probability of being a horse.</p>
<p>In response-based distillation, instead of training the student model with hard labels, it also learns from the probability of the teacher model's output (soft labels). The student can learn that the picture is a dog, but it also learns that the picture also slightly resembles a cat and barely resembles a horse.</p>
<p>This additional information is called dark knowledge. It enables the smaller model to copy the behavior of a larger, more accurate teacher while maintaining high performance.</p>
<p>Some of the advantages of response-based distillation are that it's easy to implement, it's very stable, and it works well with image classification. But there are some drawbacks, the major one being that it ignores what happens inside the network.</p>
<p>That is, response base distillation only looks at the output layer (the probabilities), so the student has a "black-box" relationship with the teacher model. It learns what the teacher decides, but not how it comes to that conclusion. Using the previous example, it learns the probability of what feature is a dog, cat and horse but not the underlying principle or workings of how that conclusion was made.</p>
<h3 id="heading-feature-based-distillation">Feature-Based Distillation</h3>
<p>This technique transfers the internal feature representation of the larger model rather than transferring only the final output.</p>
<p>An easy way to think about this is to imagine that you're a student of a great math tutor. The tutor gives you a math problem and shows you their final result. You might eventually guess the right formula, but you're mostly just learning to memorize output.</p>
<p>But if the tutor requires you to look at their notebook and see their work, you're learning the tutor's exact thought process.</p>
<p>In feature-based distillation, during training, the student model is required to produce hidden layer activations that match those of the tutor using <a href="https://en.wikipedia.org/wiki/Mean_squared_error">Mean Squared Error (MSE)</a>. With mean squared error, the student model isn't just rewarded for correctly guessing the final output. It's also penalised if its internal function system doesn't match that of the teacher model.</p>
<p>This method is heavily used in convolutional neural networks, vision transformers, object detection, and so on.</p>
<p>The drawbacks of this method are that since the student has to extract and store internal feature map of the teacher model at every training iteration, memory and power consumption can quickly escalate. This makes it very expensive to run.</p>
<h3 id="heading-attention-distillation">Attention Distillation</h3>
<p>In this approach, instead of the student model copying all the neurons, it observes where the larger model pays attention and copies those neurons.</p>
<p>In transformer models, which are models that try to make sense of a sentence instead of paying attention to individual words, attention maps indicate which input tokens or regions the model considers most important when making predictions. The student tries to reproduce those maps or the distribution of the teacher model. This shows the student where to "look" when trying to solve a problem.</p>
<p>The drawback of attention distillation is substantial overhead in both training time and hardware resource. This is because metrics like Mean Squared Error (MSE) and Kullback-Leibler (KL) divergence need to be calculated. These metrics act as a loss function that tells a model exactly how much it's prediction or internal representation differs from the teacher model. Making this calculation across massive attention tensors contribute significantly to resource consumption.</p>
<h3 id="heading-self-distillation">Self-Distillation</h3>
<p>A model is also capable of training itself. This involves knowledge transfer from an earlier version to a later version. It can also involve the transfer of knowledge between different layers in the network.</p>
<p>An advantage of this is that the model can become better without necessarily growing in size, reducing the cost of using a larger model to train the student model.</p>
<h2 id="heading-how-to-implement-knowledge-distillation-in-pytorch">How to Implement Knowledge Distillation in PyTorch.</h2>
<p>To showcase how model distillation works in real time, we'll be training a teacher and then a student model using the <a href="https://www.kaggle.com/datasets/hojjatk/mnist-dataset">MNIST dataset</a> and classifier with PyTorch.</p>
<p>The dataset contains about 70,000 greyscale images of handwritten digits. We'll train the teacher, then freeze it while we train a student using hard labels and soft targets from the teacher.</p>
<p>Note that the MNIST dataset is a relatively small dataset, so our distillation gains might not be much. But this project showcases the distillation workflow, and it will give you a good background to run larger datasets.</p>
<p>To run this project, you'll need the following Python packages:</p>
<table>
<thead>
<tr>
<th>Package</th>
<th>Purpose</th>
</tr>
</thead>
<tbody><tr>
<td><code>torch</code></td>
<td>The core deep learning framework we'll use to build and train neural networks.</td>
</tr>
<tr>
<td><code>torchvision</code></td>
<td>Provides datasets like MNIST and image utilities.</td>
</tr>
<tr>
<td><code>numpy</code></td>
<td>Numerical computations and array operations.</td>
</tr>
<tr>
<td><code>matplotlib</code></td>
<td>Plotting graphs and visualizing the visual images/results.</td>
</tr>
<tr>
<td><code>tqdm</code></td>
<td>Displays progress bars during training.</td>
</tr>
</tbody></table>
<p>For this project, I'll be working with the Visual Studio Code editor, but you can use any other code editor of your choice.</p>
<p>This is our project structure. I'll be explaining the folders and files below alongside the code for our project.</p>
<pre><code class="language-plaintext">knowledge-distillation/
│
├── data/
│
├── models.py
│
├── engine.py
│
├── losses.py
│
├── train_teacher.py
│
├── train_student.py
│
├── train_distilled_student.py
│
├── evaluate.py
│
├── visualize.py
│
├── utils.py
│
├── plots.py
│
├── requirements.txt
│
└── README.md
</code></pre>
<p>First in our folder structure is our <code>data</code> folder. This contains our MNIST dataset.</p>
<p>Our <code>models.py</code> file defines the architecture of every neural network used in the project. It contains the teacher network class and student network class. Each class defines layers, forward propagation, and how data flow through the network.</p>
<pre><code class="language-python">import torch
import torch.nn as nn
import torch.nn.functional as F

class TeacherNet(nn.Module):

    def __init__(self):
        super().__init__()

        self.flatten = nn.Flatten()

        self.fc1 = nn.Linear(28 * 28, 1024)
        self.fc2 = nn.Linear(1024, 512)
        self.fc3 = nn.Linear(512, 256)
        self.fc4 = nn.Linear(256, 128)
        self.fc5 = nn.Linear(128, 10)

    def forward(self, x):

        x = self.flatten(x)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))

        logits = self.fc5(x)

        return logits


class StudentNet(nn.Module):
    """
    Smaller neural network (Student)
    """

    def __init__(self):
        super().__init__()

        self.flatten = nn.Flatten()

        self.fc1 = nn.Linear(28 * 28, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):

        x = self.flatten(x)

        x = F.relu(self.fc1(x))

        logits = self.fc2(x)

        return logits
</code></pre>
<p>The <code>utils.py</code> file prepares the dataset and answers the question, "Where does the dataset come from?" Here, we download the MNIST dataset, normalize the images, and create our training and testing loader.</p>
<pre><code class="language-python">from torchvision import datasets, transforms
from torch.utils.data import DataLoader


def get_dataloaders(batch_size=64):
    """
    Downloads MNIST (if needed) and returns
    training and testing DataLoaders.
    """

    transform = transforms.ToTensor()

    train_dataset = datasets.MNIST(
        root="data",
        train=True,
        download=True,
        transform=transform
    )

    test_dataset = datasets.MNIST(
        root="data",
        train=False,
        download=True,
        transform=transform
    )

    train_loader = DataLoader(
        train_dataset,
        batch_size=batch_size,
        shuffle=True
    )

    test_loader = DataLoader(
        test_dataset,
        batch_size=batch_size,
        shuffle=False
    )

    return train_loader, test_loader
</code></pre>
<p>The <code>engine.py</code> file contains the reusable learning algorithm and helps with how training happens. It receives the model.py code and trains it.</p>
<pre><code class="language-python">import torch
from tqdm import tqdm


def train_one_epoch(
    model,
    dataloader,
    criterion,
    optimizer,
    device,
):
    model.train()

    running_loss = 0

    progress = tqdm(dataloader)

    for images, labels in progress:

        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        outputs = model(images)

        loss = criterion(outputs, labels)

        loss.backward()

        optimizer.step()

        running_loss += loss.item()

        progress.set_description(
            f"Loss {loss.item():.4f}"
        )

    return running_loss / len(dataloader)


def evaluate(model, dataloader, device):

    model.eval()

    correct = 0
    total = 0

    with torch.no_grad():

        for images, labels in dataloader:

            images = images.to(device)
            labels = labels.to(device)

            outputs = model(images)

            _, predicted = torch.max(outputs, dim=1)

            total += labels.size(0)

            correct += (predicted == labels).sum().item()

    accuracy = 100 * correct / total

    return accuracy


def predict(model, images, device):

    model.eval()

    with torch.no_grad():

        images = images.to(device)

        outputs = model(images)

        _, predictions = torch.max(outputs, dim=1)

    return predictions

def train_distillation_epoch(
    teacher,
    student,
    dataloader,
    optimizer,
    device,
    loss_fn,
    temperature=4.0,
    alpha=0.5,
):
    """
    Train the student using a frozen teacher.
    """

    teacher.eval()
    student.train()

    running_loss = 0

    progress = tqdm(dataloader)

    for images, labels in progress:

        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        # Teacher predictions (NO gradients)
        with torch.no_grad():
            teacher_logits = teacher(images)

        # Student predictions
        student_logits = student(images)

        # Combined loss
        loss = loss_fn(
            student_logits=student_logits,
            teacher_logits=teacher_logits,
            labels=labels,
            temperature=temperature,
            alpha=alpha,
        )

        loss.backward()

        optimizer.step()

        running_loss += loss.item()

        progress.set_description(
            f"Distill Loss {loss.item():.4f}"
        )

    return running_loss / len(dataloader)
</code></pre>
<p>The <code>losses.py</code> file defines how the model measures its mistake. Since a neural network learns by minimizing loss, for baseline visualize used <code>CrossEntropyLoss</code>, and for knowledge distillation, we introduced <code>distillation_loss()</code>.</p>
<p>Inside this file, we combined hard labels and soft targets using cross entropy and k1 divergence. This is where Geoffrey Hinton's algorithm lives.</p>
<pre><code class="language-python">import torch
import torch.nn as nn
import torch.nn.functional as F

cross_entropy = nn.CrossEntropyLoss()


def distillation_loss(
    student_logits,
    teacher_logits,
    labels,
    temperature=4.0,
    alpha=0.5,
):
    """
    Geoffrey Hinton's Knowledge Distillation Loss

    Parameters
    ----------
    student_logits : output from student model
    teacher_logits : output from teacher model
    labels         : ground-truth labels
    temperature    : controls softness of probability distribution
    alpha          : weight between hard and soft losses
    """

    # Hard Label Loss
    hard_loss = cross_entropy(student_logits, labels)

    # Teacher Soft Targets
    teacher_probs = F.softmax(
        teacher_logits / temperature,
        dim=1
    )

    # Student Soft Predictions
    student_log_probs = F.log_softmax(
        student_logits / temperature,
        dim=1
    )

    # KL Divergence
    soft_loss = F.kl_div(
        student_log_probs,
        teacher_probs,
        reduction="batchmean"
    )

    # Final Distillation Loss
    loss = (
        alpha * hard_loss
        +
        (1 - alpha)
        * (temperature ** 2)
        * soft_loss
    )

    return loss
</code></pre>
<p>The <code>train_teacher.py</code> file is to train the teacher network. After running this for the first time, we don't train the teacher again.</p>
<pre><code class="language-python">import torch
import torch.optim as optim

from models import TeacherNet
from utils import get_dataloaders
from engine import train_one_epoch, evaluate
from losses import cross_entropy

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

train_loader, test_loader = get_dataloaders()

teacher = TeacherNet().to(device)

optimizer = optim.Adam(
    teacher.parameters(),
    lr=0.001,
)

epochs = 10

for epoch in range(epochs):

    loss = train_one_epoch(
        teacher,
        train_loader,
        cross_entropy,
        optimizer,
        device,
    )

    accuracy = evaluate(
        teacher,
        test_loader,
        device,
    )

    print(
        f"Epoch {epoch+1}/{epochs}"
        f" | Loss: {loss:.4f}"
        f" | Accuracy: {accuracy:.2f}%"
    )

torch.save(
    teacher.state_dict(),
    "teacher.pth",
)

print("\nTeacher saved successfully.")
</code></pre>
<p>The <code>train_student.p</code>y file is where we train the student. It's identical to that of the train_teacher except it's a smaller network and no teacher guidance is implemented here. This gives the baseline accuracy of how the student will perform without the teacher.</p>
<pre><code class="language-python">import torch
import torch.optim as optim

from models import StudentNet
from utils import get_dataloaders
from engine import train_one_epoch, evaluate
from losses import cross_entropy

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

train_loader, test_loader = get_dataloaders()

student = StudentNet().to(device)

optimizer = optim.Adam(
    student.parameters(),
    lr=0.001,
)

epochs = 10

for epoch in range(epochs):

    loss = train_one_epoch(
        student,
        train_loader,
        cross_entropy,
        optimizer,
        device,
    )

    accuracy = evaluate(
        student,
        test_loader,
        device,
    )

    print(
        f"Epoch {epoch+1}/{epochs}"
        f" | Loss: {loss:.4f}"
        f" | Accuracy: {accuracy:.2f}%"
    )

torch.save(
    student.state_dict(),
    "student_baseline.pth",
)

print("\nBaseline student saved.")
</code></pre>
<p>The <code>train_distilled_student.py</code> is the heart of this code. In this file, we teach the student using both labels and the teacher's knowledge. Here, we implement response-based knowledge distillation.</p>
<pre><code class="language-python">import torch
import torch.optim as optim

from models import TeacherNet, StudentNet
from utils import get_dataloaders
from engine import (
    train_distillation_epoch,
    evaluate,
)
from losses import distillation_loss

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

train_loader, test_loader = get_dataloaders()

# Load Teacher
teacher = TeacherNet().to(device)
teacher.load_state_dict(
    torch.load(
        "teacher.pth",
        map_location=device,
    )
)

teacher.eval()

# Student
student = StudentNet().to(device)

optimizer = optim.Adam(
    student.parameters(),
    lr=0.001,
)

epochs = 10

for epoch in range(epochs):

    loss = train_distillation_epoch(
        teacher,
        student,
        train_loader,
        optimizer,
        device,
        distillation_loss,
        temperature=3,
        alpha=0.7,
    )

    accuracy = evaluate(
        student,
        test_loader,
        device,
    )

    print(
        f"Epoch {epoch+1}/{epochs}"
        f" | Loss: {loss:.4f}"
        f" | Accuracy: {accuracy:.2f}%"
    )

torch.save(
    student.state_dict(),
    "student_distilled.pth",
)

print("\nDistilled student saved.")
</code></pre>
<p>The <code>evaluate.py</code> measures how well a trained model performs. Here, we simply load the model, load test data, predict, and compute the accuracy.</p>
<pre><code class="language-python">import torch

from models import TeacherNet
from utils import get_dataloaders
from engine import evaluate

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

_, test_loader = get_dataloaders()

teacher = TeacherNet().to(device)

teacher.load_state_dict(
    torch.load(
        "teacher.pth",
        map_location=device,
    )
)

accuracy = evaluate(
    teacher,
    test_loader,
    device,
)

print(
    f"Teacher Accuracy: {accuracy:.2f}%"
)
</code></pre>
<p>The <code>visualize.py</code> file is to present our findings visually. We use the Python package matplotlib to visualise the results of our teacher, baseline student, and distilled student.</p>
<pre><code class="language-python">import matplotlib.pyplot as plt

models = [
    "Teacher",
    "Baseline\nStudent",
    "Distilled\nStudent"
]

accuracy = [98.20, 97.86, 97.91]

plt.figure(figsize=(7,5))
plt.bar(models, accuracy)

plt.title("Knowledge Distillation Performance Comparison")
plt.ylabel("Accuracy (%)")

for i, v in enumerate(accuracy):
    plt.text(i, v + 0.02, f"{v:.2f}%", ha="center")

plt.ylim(97.5, 98.5)

plt.tight_layout()
plt.savefig("accuracy_comparison.png", dpi=300)
plt.show()
</code></pre>
<p>You can see the entire workflow of our program below so you understand how everything comes together:</p>
<pre><code class="language-plaintext">                       utils.py
                          │
                 Load MNIST Dataset
                          │
                          ▼
                     models.py
             TeacherNet / StudentNet
                          │
                          ▼
                     losses.py
  Cross Entropy + Distillation Loss Function
                          │
                          ▼
                     engine.py
    Generic Training and Evaluation Functions
             ┌────────────┼────────────┐
             │            │            │
             ▼            ▼            ▼
  train_teacher.py  train_student.py  distilled_student.py
             │            │            │
             ▼            ▼            ▼
   teacher.pth  student_baseline.pth  distilled.pth
                          │
                          ▼
                     evaluate.py
              Compute Final Accuracy
                          │
                          ▼
                    visualize.py
     Generate Charts and Experimental Figures
</code></pre>
<h2 id="heading-how-to-evaluate-the-results">How to Evaluate the Results</h2>
<p>Now, let's check and evaluate our results. These are screenshots from running ten epochs for each of the networks: the teacher, student, and distilled student network.</p>
<p>Teacher Network: After 10 complete cycles, we attained an accuracy of 98.20%.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6030b75372a689446db3d6d8/66c06c7e-79d4-4689-be44-51aea6c637af.png" alt="Teacher model training phase of 10 epoch, it achieved accuracy of 98.20%" style="display: block;" width="600" height="400" loading="lazy">

<p>Student Network: After 10 complete cycles, we attained an accuracy of 97.74%. This is a pretty good performance for a smaller network without the teacher model.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6030b75372a689446db3d6d8/eddaa86b-8ff8-4faa-aa30-329d1e60e75d.png" alt="Baseline student model training phase of 10 epoch, it achieved accuracy of 97.74%" style="display: block;" width="600" height="400" loading="lazy">

<p>Distilled Student Network: After 10 complete cycle, we attained an accuracy of 98.20%. We had an improvement from 97.74% achieved without the teacher's model knowledge.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6030b75372a689446db3d6d8/6c782cd8-3691-4954-acc6-e430560d4921.png" alt="Distilled Student model training phase of 10 epoch, it achieved accuracy of 97.82%" style="display: block;" width="600" height="400" loading="lazy">

<p>Check out the chart below that shows how each model performed:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6030b75372a689446db3d6d8/5c0543cf-2c16-42e1-938f-9cf683537263.png" alt="A graph of Teacher model achieving 98.20%, Baseline Student achieved 97.86% and Distilled Student achieved 97.91%" style="display: block;" width="600" height="400" loading="lazy">

<h2 id="heading-other-approaches-for-reducing-model-size">Other Approaches for Reducing Model Size</h2>
<p>Now that we've seen what model distillation is with real-world code examples. Let's look at other approaches that developers are taking to reduce the size of large language models to save on latency, memory footprints, and compute cost and how they differ from distillation.</p>
<p>We have three other major approaches for this:</p>
<h3 id="heading-quantization">Quantization</h3>
<p>Quantization achieves efficiency by strictly reducing the bit-width of the underlying data types used to store numerical weight and activations of the neural network. It targets the memory bandwidth bottleneck rather than the model architecture.</p>
<p>In simple terms, quantization can change a 32-bit number to 8-bit numbers, which saves about 75% of the file size.</p>
<p>A real-world example of how quantization differs from distillation is trying to save resources like ink and paper when printing a textbook. Distillation summarizes the textbook, while quantization prints the exact thing but uses a lower-resolution font to save ink and paper.</p>
<h3 id="heading-pruning">Pruning</h3>
<p>Pruning operates on the principle that large language models often have too many redundant and low-impact parameters on the network. By removing weak connections, models can become smaller and faster.</p>
<p>For the textbook analogy, the pruning process is like going through the textbook and ripping off pages or chapters that you feel aren't needed.</p>
<h3 id="heading-low-rank-factorization">Low-Rank Factorization</h3>
<p>This is a mathematical process that uses linear algebra to simplify the matrix multiplication that dominates the deep learning architecture.</p>
<p>With the textbook example, this process involves rewriting complex sentences into simpler, shorter formulas that mean the same thing.</p>
<p>Choosing which method to use depends on the bottleneck you're trying to solve. Quantization generally works well for memory challenges, while pruning and distillation are designed to reduce computation time.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Knowledge transfer is as fundamental to artificial intelligence as it is to human history. Just as Aristotle built upon the teachings of Plato, today's efficient, deployable neural networks rely on the foundational "dark knowledge" extracted from massive, compute-heavy teacher models.</p>
<p>While training trillion-parameter frontier models requires immense hardware and financial resources, knowledge distillation ensures that the practical benefits of these advancements aren't locked behind massive computational bottlenecks.</p>
<p>As we demonstrated with the PyTorch MNIST experiment, even a simple response-based distillation can help a smaller network punch above its weight class by learning from the soft probabilities of a larger teacher.</p>
<p>When combined with other optimization techniques like quantization and pruning, distillation provides a clear pathway for deploying highly capable AI on edge devices, mobile phones, and consumer-grade hardware.</p>
<p>Thank you for reading! Please check out my <a href="https://www.youtube.com/@EjegiOgbemi">YouTube</a> channel, where I post videos on AI-related content.</p>
<h3 id="heading-references">References:</h3>
<ol>
<li><p><a href="https://workos.com/guide/distill-a-machine-learning-model-or-llm">What does it mean to distill a machine learning model or LLM?</a></p>
</li>
<li><p>AI Insights: Model Distillation from the <a href="https://www.gov.uk/government/publications/ai-insights/ai-insights-model-distillation-html">Department for Science, Innovation and Technology</a></p>
</li>
<li><p><a href="https://redis.io/blog/model-distillation-llm-guide/">Model distillation for LLMs: A practical guide to smaller, faster AI</a></p>
</li>
<li><p><a href="https://arxiv.org/abs/1503.02531">Distilling knowledge in a neural network</a> by Geoffrey Hinton</p>
</li>
<li><p><a href="https://huggingface.co/blog/Kseniase/kd">Everything you need to know about knowledge distillation</a></p>
</li>
</ol>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ What is Overfitting in Machine Learning? ]]>
                </title>
                <description>
                    <![CDATA[ Have you ever performed some task without really thinking about the process involved? For example, making coffee, tying your shoes, or walking through your neighborhood.  In these types of activities, you've done these things so many times that you'v... ]]>
                </description>
                <link>https://www.freecodecamp.org/news/what-is-overfitting-machine-learning/</link>
                <guid isPermaLink="false">66c71f7acdd5b7ce3d9c37da</guid>
                
                    <category>
                        <![CDATA[ Artificial Intelligence ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Machine Learning ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Overfitting ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Mene-Ejegi Ogbemi ]]>
                </dc:creator>
                <pubDate>Mon, 16 Oct 2023 20:21:22 +0000</pubDate>
                <media:content url="https://www.freecodecamp.org/news/content/images/2023/09/Banner---Tech-writing---Overfitting.jpg" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Have you ever performed some task without really thinking about the process involved? For example, making coffee, tying your shoes, or walking through your neighborhood. </p>
<p>In these types of activities, you've done these things so many times that you've mastered the process. You can be thinking about something unrelated, yet you perform these activities all the same. This phenomenon is called procedural memory in psychology.</p>
<p>We have this kind of thing with machine learning models as well, but it's not as positive as it is with humans. This is known as overfitting in machine learning. </p>
<h2 id="heading-what-is-overfitting">What is Overfitting?</h2>
<p>In overfitting, a model becomes so good at our training data that it has mastered every pattern, including noise. This makes the model perform well with training data but poorly with test or validation data.</p>
<p>The illustration below depicts how an optimal model fits into the data compared to overfitting. </p>
<p>In the graph, we have our features on the x-axis. In datasets, features are data that can be used to predict an outcome. The output variable is the outcome based on those features. The blue dots represent the data points where the features determine output variables. </p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/overfitting-illustration.jpg" alt="Image" width="600" height="400" loading="lazy"></p>
<p>In the optimal graph, our model tries to find the generalized trend. But in our overfitted chart, the model tries to master each data point, resulting in an asymmetrical curve.</p>
<p>An example of a case study would be to predict if a customer would default on a bank loan. Assuming we have a dataset of 100,000 customers containing features such as demographics, income, loan amount, credit history, employment record, and default status, we split our data into training and test data. </p>
<p>Our training dataset contains 80,000 customers, while our test dataset contains 20,000 customers. In the training the dataset, we observe that our model has a 97% accuracy, but in prediction, we only get 50% accuracy. This shows that we have an overfitting problem.</p>
<p>Can you tell why overfitting is a problem? Yes! It produces an incorrect prediction. It is the purpose of machine learning models to make predictions to help business decision-making. We waste time and resources when our model makes incorrect predictions. </p>
<p>Imagine predicting that a customer will pay back a loan, and the customer defaults. Not just one customer but thousands of customers. This can cause a crisis for any financial institution.</p>
<h2 id="heading-causes-of-overfitting">Causes of Overfitting</h2>
<h3 id="heading-noisy-data">Noisy data</h3>
<p>Noise in data often appears as errors, fluctuations, or outliers in the data. This can be caused by data entry errors, data aging, data transmission errors, and so on. </p>
<p>Too much noise in data can cause the model to think these are valid data points. Fitting the noise pattern in the training dataset will cause poor performance on the new dataset.  </p>
<p>For example, let's say that we are building a machine-learning model to classify images of cats and dogs. But some of the images in the dataset are blurry or poorly lit. While the model may perform well on the training data, it might struggle on the test data since it must have mastered some pattern with the blurry images in the dataset.</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/Cat-and-dog-3.jpg" alt="Image" width="600" height="400" loading="lazy"></p>
<p>In the picture above, you can see that we have some blurry images that cannot be labelled if they are cat or dog. In these instances, the model could also learn these patterns alongside relevant features. Removing these images can reduce overfitting.</p>
<h3 id="heading-insufficient-training-data">Insufficient training data</h3>
<p>There will be fewer patterns and noises to analyze if we don't have sufficient training data. This means that the machine can only learn a little about our data. </p>
<p>Using our previous example, if our training data contains fewer images of dogs but many more of cats, the model learns so much about cats that when we feed the system an image of a dog, it will likely give a wrong output.</p>
<h3 id="heading-overly-complex-model">Overly complex model</h3>
<p>In a complex model, there are many parameters capable of capturing patterns and relationships in training data. As a result, our model makes a more accurate prediction. </p>
<p>But this can pose a problem, since the model can start capturing noise, fluctuations, or outliers. Let's look at a decision tree model, how it works, and how overfitting can happen when it becomes too complex.</p>
<p>A decision tree model works by repeatedly breaking down data into significant features, making each point a node. This creates a tree like structure. </p>
<p>To make a prediction, it starts from the root node and follow the branches down, breaking and fitting every feature until it gets to the leaf node. The prediction is then made based on the value associated with the leaf node.</p>
<p>Let's look at a simple tree diagram of how a decision tree can predict if a customer is likely to default on loan base on certain features.</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/decision-tree-flowchart.png" alt="Image" width="600" height="400" loading="lazy">
<em>Tree diagram showing whether a customer is likely to default on a loan</em></p>
<p>This model starts by creating a parent node which is credit score. Depending on whether the credit score for the applicant is high or low, it goes down to the next node, which is either debt to income ratio or employment status. Then it makes the final prediction as to whether the customer is likely to default or not.</p>
<p>A decision tree can become overly complex when it creates too many nodes, making it too detailed or specific to the training data. </p>
<p>Let's see a sample machine learning program that predicts whether a customer will default a loan or not using decision tree model. For specificity, I wont be showing the cleaning process and visualization. I'll just lay emphasis on the required functions and how overfitting can happen with decision tree model. </p>
<p>The link to the complete repository containing cleaning and visualization can be found <a target="_blank" href="https://github.com/ogbemi-ejegi/Overfitting">here</a>, and you can get the dataset <a target="_blank" href="https://www.kaggle.com/datasets/rishikeshkonapure/home-loan-approval?select=loan_sanction_train.csv">here</a>.</p>
<pre><code class="lang-python"><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt
<span class="hljs-keyword">import</span> seaborn <span class="hljs-keyword">as</span> sns
<span class="hljs-keyword">from</span> sklearn.metrics <span class="hljs-keyword">import</span> accuracy_score
<span class="hljs-keyword">from</span> sklearn.model_selection <span class="hljs-keyword">import</span> train_test_split
<span class="hljs-keyword">from</span> sklearn.preprocessing <span class="hljs-keyword">import</span> LabelEncoder
<span class="hljs-keyword">from</span> sklearn <span class="hljs-keyword">import</span> tree
<span class="hljs-keyword">from</span> sklearn <span class="hljs-keyword">import</span> metrics
<span class="hljs-keyword">from</span> sklearn.tree <span class="hljs-keyword">import</span> DecisionTreeClassifier

%matplotlib inline

<span class="hljs-comment">#Importing our libraries</span>
train = pd.read_csv(<span class="hljs-string">'/content/train.csv'</span>)
test = pd.read_csv(<span class="hljs-string">'/content/test.csv'</span>)

<span class="hljs-comment">#Combine both training and test data</span>
df = pd.concat([train, test], axis=<span class="hljs-number">0</span>)
df.head()

<span class="hljs-comment">#View dataset</span>
train.head()

<span class="hljs-comment"># Copy require features to a variable df_</span>
df_ = train[[<span class="hljs-string">'Gender'</span>,
<span class="hljs-string">'Married'</span>,
<span class="hljs-string">'Education'</span>,
<span class="hljs-string">'Self_Employed'</span>,
<span class="hljs-string">'Dependents'</span>,
<span class="hljs-string">'ApplicantIncome'</span>,
<span class="hljs-string">'CoapplicantIncome'</span>,
<span class="hljs-string">'LoanAmount'</span>,
<span class="hljs-string">'Loan_Amount_Term'</span>,
<span class="hljs-string">'Property_Area'</span>,
<span class="hljs-string">'Credit_History'</span>]]

<span class="hljs-comment">### Duplicate a copy of df into X</span>
X = df_.copy()

<span class="hljs-comment">### label encode for Y</span>
y = train[<span class="hljs-string">'Loan_Status'</span>].map({<span class="hljs-string">'N'</span>:<span class="hljs-number">0</span>,<span class="hljs-string">'Y'</span>:<span class="hljs-number">1</span>}).astype(int)

<span class="hljs-comment">### train-test split</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span class="hljs-number">0.2</span>, random_state=<span class="hljs-number">42</span>)

<span class="hljs-comment"># train</span>
clf = DecisionTreeClassifier() <span class="hljs-comment">#change model here</span>
clf.fit(X_train, y_train)

<span class="hljs-comment"># predict</span>
predictions_clf = clf.predict(X_test)

<span class="hljs-comment">#Print Accuracy</span>
print(<span class="hljs-string">'Model Accuracy:'</span>, accuracy_score(predictions_clf, y_test))
</code></pre>
<p>To understand this better, I'll explain what each module does:</p>
<pre><code class="lang-python"><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt
<span class="hljs-keyword">import</span> seaborn <span class="hljs-keyword">as</span> sns
<span class="hljs-keyword">from</span> sklearn.metrics <span class="hljs-keyword">import</span> accuracy_score
<span class="hljs-keyword">from</span> sklearn.model_selection <span class="hljs-keyword">import</span> train_test_split
<span class="hljs-keyword">from</span> sklearn.preprocessing <span class="hljs-keyword">import</span> LabelEncoder
<span class="hljs-keyword">from</span> sklearn <span class="hljs-keyword">import</span> tree
<span class="hljs-keyword">from</span> sklearn <span class="hljs-keyword">import</span> metrics
<span class="hljs-keyword">from</span> sklearn.tree <span class="hljs-keyword">import</span> DecisionTreeClassifier
</code></pre>
<p>The first block is the import section. This is where we import all our dependencies. </p>
<ul>
<li>Numpy is a Python library used for scientific computing. </li>
<li>Pandas is a library for data analysis and manipulation. </li>
<li>Matplotlib and Seaborn are for statistical data visualization. </li>
<li><code>Accuracy_score</code> is a function to calculate the accuracy of our model.</li>
<li><code>train_test_split</code> is used to split our dataset into training and test data. </li>
<li>The <code>LabelEncoder</code> encodes categorical variables into numeric variables. </li>
<li><code>tree</code> is for building a decision tree classifier. </li>
<li><code>metrics</code> helps us evaluate our models.</li>
</ul>
<pre><code class="lang-python"><span class="hljs-comment">#Importing our dataset</span>
train = pd.read_csv(<span class="hljs-string">'/content/train.csv'</span>)
test = pd.read_csv(<span class="hljs-string">'/content/test.csv'</span>)
</code></pre>
<p>This module imports our datasets. Our train and test datasets have been downloaded from the public repository, so we import them separately.</p>
<pre><code class="lang-python"><span class="hljs-comment">#Combine both training and test data</span>
df = pd.concat([train, test], axis=<span class="hljs-number">0</span>)
df.head()
</code></pre>
<p>To work with both datasets, we need to combine them into one dataset. The concat function combines both datasets. We use <code>df.head()</code> to visualize the dataset which is shown below.</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/image-76.png" alt="Image" width="600" height="400" loading="lazy">
<em>Screenshot of our dataset</em></p>
<pre><code class="lang-python"><span class="hljs-comment"># Copy require features to a variable df_</span>
df_ = train[[<span class="hljs-string">'Gender'</span>,
<span class="hljs-string">'Married'</span>,
<span class="hljs-string">'Education'</span>,
<span class="hljs-string">'Self_Employed'</span>,
<span class="hljs-string">'Dependents'</span>,
<span class="hljs-string">'ApplicantIncome'</span>,
<span class="hljs-string">'CoapplicantIncome'</span>,
<span class="hljs-string">'LoanAmount'</span>,
<span class="hljs-string">'Loan_Amount_Term'</span>,
<span class="hljs-string">'Property_Area'</span>,
<span class="hljs-string">'Credit_History'</span>]]

<span class="hljs-comment">### Duplicate a copy of df into X</span>
X = df_.copy()
</code></pre>
<p>To start working with our features, we created a variable df_ to store all the features needed for prediction. We duplicated this into the variable X to create a copy to work with.</p>
<pre><code class="lang-python"><span class="hljs-comment">### label encode for Y</span>
y = train[<span class="hljs-string">'Loan_Status'</span>].map({<span class="hljs-string">'N'</span>:<span class="hljs-number">0</span>,<span class="hljs-string">'Y'</span>:<span class="hljs-number">1</span>}).astype(int)
</code></pre>
<p>To work with our outcome variable, we needed to convert it from a categorical value to an integer value. This also makes it easy for our model to understand. All values of N were converted to 0, while Y was converted to 1.</p>
<pre><code class="lang-python"><span class="hljs-comment">### train-test split</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span class="hljs-number">0.2</span>, random_state=<span class="hljs-number">42</span>)
</code></pre>
<p>We use our <code>train_test_split</code> to split our data into training and test data. The <code>test_size = 0.2</code> means we are using 20% of the data for testing and 80% for training.</p>
<pre><code class="lang-python"><span class="hljs-comment"># train</span>
clf = DecisionTreeClassifier()
clf.fit(X_train, y_train)
</code></pre>
<p>We assigned <code>DecisionTreeClassifier()</code> to the variable <code>clf</code>, which we'll use to train and fit our data. <code>DecisionTreeClassifier()</code> has an optional argument named <code>max_depth</code>. The number assigned to <code>max_depth</code> determines the depth of the tree. This is how we'll use it to cause overfitting in another section below.</p>
<pre><code class="lang-python"><span class="hljs-comment"># predict</span>
predictions_clf = clf.predict(X_test)
</code></pre>
<p>In the code snippet above, <code>clf.predict</code> is used to predict the data in <code>X_test</code>. </p>
<pre><code class="lang-python">print(<span class="hljs-string">'Model Accuracy:'</span>, accuracy_score(predictions_clf, y_test))
</code></pre>
<p>The model accuracy was printed using the accuracy_score function, which you can see in the screenshot below:</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/image-77.png" alt="Image" width="600" height="400" loading="lazy">
<em>Model accuracy - almost 70%</em></p>
<p>Now that we've seen how a decision tree works and even run a machine learning model to predict if a customer will default or not, let's see how to cause and diagnose overfitting by modifying the code using the <code>max_depth</code> argument.</p>
<h2 id="heading-how-to-diagnose-overfitting">How to Diagnose Overfitting</h2>
<h3 id="heading-visualizations">Visualizations</h3>
<p>Using visualizations can help us detect overfitting by providing insights into the behavior of our model. </p>
<p>Common visualization methods include plotting data points for the model's prediction, visualizing feature distributions, or creating plots of decision boundaries. </p>
<p>To visualize the overfitting for our loan application above, I had to tweak the code by creating an iteration using different <code>max_depth</code> values ranging from 1 to 24. Predictions are calculated based on training and test data and stored in a list.</p>
<pre><code class="lang-python"><span class="hljs-comment">#Creating a list to store accuracy values</span>
train_accuracies = []
test_accuracies = []

<span class="hljs-comment">#Loop</span>
<span class="hljs-keyword">for</span> depth <span class="hljs-keyword">in</span> range(<span class="hljs-number">1</span>, <span class="hljs-number">25</span>):
  tree_model = DecisionTreeClassifier(max_depth = depth)
  tree_model.fit(X_train, y_train)

  train_predictions = tree_model.predict(X_train)
  test_predictions = tree_model.predict(X_test)

  <span class="hljs-comment">#calculate training and test accuracy</span>
  train_accuracy = metrics.accuracy_score(y_train, train_predictions)

  test_accuracy = metrics.accuracy_score(y_test, test_predictions)

  <span class="hljs-comment">#Append accuracies</span>
  train_accuracies.append(train_accuracy)
  test_accuracies.append(test_accuracy)
</code></pre>
<p>The difference here is that we are creating two variables – <code>train_accuracies</code> and <code>test_accuracies</code> – to store the accuracy values. Using these variables, we can use the code below to generate a plot that shows the changes between these variables as the <code>max_depth</code> value changes.</p>
<pre><code class="lang-python"><span class="hljs-comment">#Creating our plot</span>
plt.figure(figsize = (<span class="hljs-number">10</span>, <span class="hljs-number">5</span>))
sns.set_style(<span class="hljs-string">"whitegrid"</span>)
plt.plot(train_accuracies, label= <span class="hljs-string">"train accuracy"</span>)
plt.plot(test_accuracies, label=<span class="hljs-string">"test accuracy"</span>)
plt.legend(loc = <span class="hljs-string">"upper left"</span>)
plt.xticks(range(<span class="hljs-number">0</span>, <span class="hljs-number">26</span>, <span class="hljs-number">5</span>))
plt.xlabel(<span class="hljs-string">"max_depth"</span>, size = <span class="hljs-number">20</span>)
plt.ylabel(<span class="hljs-string">"accuracy"</span>, size = <span class="hljs-number">20</span>)
plt.show()
</code></pre>
<p>This is how the plot looks:</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/overfitting-visualization-1.png" alt="Image" width="600" height="400" loading="lazy">
<em>Train accuracy vs test accuracy</em></p>
<p>You'll notice that as <code>max_depth</code> values on the x-axis begin to increase, the training data accuracy starts improving a lot to a perfect score. In spite of this, the test data accuracy decreased from 0.78 to 0.70. This is a classic example of overfitting as the model becomes too complex.</p>
<h3 id="heading-training-and-validation-accuracy-gap">Training and validation accuracy gap</h3>
<p>The accuracy gap is a good way to know if overfitting has occurred in your program. This means that there is a wide gap between training data and validation data when it comes to accuracy. </p>
<p>As a guide, a 5% gap is what you should look for. Cases where you have more than this are often an indicator of overfitting: for example, our visualization above shows that when our <code>max_depth</code> value was at 2o, our training accuracy was at 100% while our test accuracy was 70%.</p>
<h2 id="heading-how-to-prevent-overfitting">How to Prevent Overfitting</h2>
<h3 id="heading-collect-more-training-data">Collect more training data</h3>
<p>As discussed above, insufficient training data can cause overfitting as the model cannot capture the relevant patterns and intricacies represented in the data. </p>
<p>Machine learning generally requires thousands or millions of records in your dataset for training. With this, there will be enough patterns to capture. You can identify outliers or noise more easily if you've done proper cleaning on the dataset using relevant techniques.</p>
<h3 id="heading-use-regularization-techniques">Use regularization techniques</h3>
<p>Regularization techniques involve simplifying models by penalizing less influential features. These penalties are embedded in the model's loss function.</p>
<p>Regularization techniques for the decision tree model above include pruning, cost complexity pruning, and others.</p>
<p>Pruning is a technique that involves removing unnecessary branches from the decision tree. For example, we can set a minimum number of customers on a leaf, such as 20. This prevents the tree from making decisions based on a very small group of customers.</p>
<p>Cost complexity involves removing branches from the tree based on their complexity. This controls the trade-off between tree complexity and accuracy.</p>
<h3 id="heading-ensembling">Ensembling</h3>
<p>Ensembling entails combining several machine learning models to contribute their strengths and unique perspectives to make a prediction. </p>
<p>Ensembling leverages the wisdom of the crowd to make more accurate predictions on unseen data, which improves generalization and reduces the risk of overfitting. </p>
<p>Popular ensemble methods include bagging, boosting, and stacking, which have been successful in a wide range of machine-learning tasks.</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/10/ENSEMBLE-1.jpg" alt="Image" width="600" height="400" loading="lazy">
<em>Diagram showing how ensembling works</em></p>
<p>The diagram above shows how the ensembling method combines various machine learning models for making predictions. Each model is trained independently on its respective subset of data. The predictions for individual models are then combined or the mean is gotten to make a final prediction.</p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>Overfitting happens when a model fits training data too closely, resulting in great training performance but poor generalization. Overfitting can be problematic as it yields incorrect predictions.</p>
<p>This can be caused by a lack of training data, an overly complex model, or noisy data. Diagnosis involves assessing the training-validation accuracy gap, using visualizations to scrutinize model behavior, and so on.</p>
<p>Prevention strategies include collecting more training data, using regularization techniques, and employing ensemble methods. These approaches ensure models generalize well and make accurate predictions for informed decisions.</p>
<p>Thank you for reading! Please follow me on <a target="_blank" href="https://www.linkedin.com/in/ogbemi-ejegi/">LinkedIn</a> where I also post more data related content.</p>
<h4 id="heading-references">References:</h4>
<ul>
<li>Siddhardhan. "Overfitting in Machine Learning | Causes for Overfitting and its Prevention" [Video]. Retrieved from <a target="_blank" href="https://www.youtube.com/watch?v=gy8kXdd6K-o">https://www.youtube.com/watch?v=gy8kXdd6K-o</a></li>
<li>Udacity. "Ensemble Learners" [Video]. Retrieved from <a target="_blank" href="https://www.youtube.com/watch?v=Un9zObFjBH0">https://www.youtube.com/watch?v=Un9zObFjBH0</a></li>
<li>White Board Machine Learning. "Overfitting in Decision Trees" [Video]. Retrieved from <a target="_blank" href="https://www.youtube.com/watch?v=eU4X-dL8nYo">https://www.youtube.com/watch?v=eU4X-dL8nYo</a></li>
</ul>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ What Is Hypothesis Testing? Types and Python Code Example ]]>
                </title>
                <description>
                    <![CDATA[ Curiosity has always been a part of human nature. Since the beginning of time, this has been one of the most important tools for birthing civilizations. Still, our curiosity grows — it tests and expands our limits. Humanity has explored the plains of... ]]>
                </description>
                <link>https://www.freecodecamp.org/news/what-is-hypothesis-testing/</link>
                <guid isPermaLink="false">66c71f7855df43f1418b5b23</guid>
                
                    <category>
                        <![CDATA[ data analysis ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Data Science ]]>
                    </category>
                
                    <category>
                        <![CDATA[ statistics ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Mene-Ejegi Ogbemi ]]>
                </dc:creator>
                <pubDate>Fri, 22 Sep 2023 00:41:23 +0000</pubDate>
                <media:content url="https://www.freecodecamp.org/news/content/images/2023/09/Banner---Tech-writing---hypothesis.jpg" medium="image" />
                <content:encoded>
                    <![CDATA[ <p>Curiosity has always been a part of human nature. Since the beginning of time, this has been one of the most important tools for birthing civilizations. Still, our curiosity grows — it tests and expands our limits. Humanity has explored the plains of land, water, and air. We've built underwater habitats where we could live for weeks. Our civilization has explored various planets. We've explored land to an unlimited degree. </p>
<p>These things were possible because humans asked questions and searched until they found answers. However, for us to get these answers, a proven method must be used and followed through to validate our results. Historically, philosophers assumed the earth was flat and you would fall off when you reached the edge. While philosophers like Aristotle argued that the earth was spherical based on the formation of the stars, they could not prove it at the time. </p>
<p>This is because they didn't have adequate resources to explore space or mathematically prove Earth's shape. It was a Greek mathematician named Eratosthenes who calculated the earth's circumference with incredible precision. He used scientific methods to show that the Earth was not flat. Since then, other methods have been used to prove the Earth's spherical shape.</p>
<p>When there are questions or statements that are yet to be tested and confirmed based on some scientific method, they are called hypotheses. Basically, we have two types of hypotheses: null and alternate.</p>
<p>A <strong>null hypothesis</strong> is one's default belief or argument about a subject matter. In the case of the earth's shape, the null hypothesis was that the earth was flat.</p>
<p>An <strong>alternate hypothesis</strong> is a belief or argument a person might try to establish. Aristotle and Eratosthenes argued that the earth was spherical.</p>
<p>Other examples of a random alternate hypothesis include:</p>
<ul>
<li>The weather may have an impact on a person's mood.</li>
<li>More people wear suits on Mondays compared to other days of the week.</li>
<li>Children are more likely to be brilliant if both parents are in academia, and so on.</li>
</ul>
<h2 id="heading-what-is-hypothesis-testing">What is Hypothesis Testing?</h2>
<p>Hypothesis testing is the act of testing whether a hypothesis or inference is true. When an alternate hypothesis is introduced, we test it against the null hypothesis to know which is correct. Let's use a plant experiment by a 12-year-old student to see how this works.</p>
<p>The hypothesis is that a plant will grow taller when given a certain type of fertilizer. The student takes two samples of the same plant, fertilizes one, and leaves the other unfertilized. He measures the plants' height every few days and records the results in a table. </p>
<p>After a week or two, he compares the final height of both plants to see which grew taller. If the plant given fertilizer grew taller, the hypothesis is established as fact. If not, the hypothesis is not supported. This simple experiment shows how to form a hypothesis, test it experimentally, and analyze the results.</p>
<p>In hypothesis testing, there are two types of error: Type I and Type II.</p>
<p>When we reject the null hypothesis in a case where it is correct, we've committed a Type I error. Type II errors occur when we fail to reject the null hypothesis when it is incorrect.</p>
<p>In our plant experiment above, if the student finds out that both plants' heights are the same at the end of the test period yet opines that fertilizer helps with plant growth, he has committed a Type I error. </p>
<p>However, if the fertilized plant comes out taller and the student records that both plants are the same or that the one without fertilizer grew taller, he has committed a Type II error because he has failed to reject the null hypothesis.</p>
<h2 id="heading-what-are-the-steps-in-hypothesis-testing">What are the Steps in Hypothesis Testing?</h2>
<p>The following steps explain how we can test a hypothesis:</p>
<h3 id="heading-step-1-define-the-null-and-alternative-hypotheses">Step #1 - Define the Null and Alternative Hypotheses</h3>
<p>Before making any test, we must first define what we are testing and what the default assumption is about the subject. In this article, we'll be testing if the average weight of 10-year-old children is more than 32kg. </p>
<p>Our null hypothesis is that 10 year old children weigh 32 kg on average. Our alternate hypothesis is that the average weight is more than 32kg. <code>Ho</code> denotes a null hypothesis, while <code>H1</code> denotes an alternate hypothesis.</p>
<p>Ho = 32</p>
<p>H1 = 32</p>
<h3 id="heading-step-2-choose-a-significance-level">Step #2 - Choose a Significance Level</h3>
<p>The significance level is a threshold for determining if the test is valid. It gives credibility to our hypothesis test to ensure we are not just luck-dependent but have enough evidence to support our claims. We usually set our significance level before conducting our tests. The criterion for determining our significance value is known as p-value. </p>
<p>A lower p-value means that there is stronger evidence against the null hypothesis, and therefore, a greater degree of significance. A p-value of 0.05 is widely accepted to be significant in most fields of science. P-values do not denote the probability of the outcome of the result, they just serve as a benchmark for determining whether our test result is due to chance. For our test, our p-value will be 0.05.</p>
<h3 id="heading-step-3-collect-data-and-calculate-a-test-statistic">Step #3 - Collect Data and Calculate a Test Statistic</h3>
<p>You can obtain your data from online data stores or conduct your research directly. Data can be scraped or researched online. The methodology might depend on the research you are trying to conduct.</p>
<p>We can calculate our test using any of the appropriate hypothesis tests. This can be a T-test, Z-test, Chi-squared, and so on. There are several hypothesis tests, each suiting different purposes and research questions. In this article, we'll use the T-test to run our hypothesis, but I'll explain the Z-test, and chi-squared too.</p>
<p>T-test is used for comparison of two sets of data when we don't know the population standard deviation. It's a parametric test, meaning it makes assumptions about the distribution of the data. These assumptions include that the data is normally distributed and that the variances of the two groups are equal. In a more simple and practical sense, imagine that we have test scores in a class for males and females, but we don't know how different or similar these scores are. We can use a t-test to see if there's a real difference.</p>
<p>The Z-test is used for comparison between two sets of data when the population standard deviation is known. It is also a parametric test, but it makes fewer assumptions about the distribution of data. The z-test assumes that the data is normally distributed, but it does not assume that the variances of the two groups are equal. In our class test example, with the t-test, we can say that if we already know how spread out the scores are in both groups, we can now use the z-test to see if there's a difference in the average scores.</p>
<p>The Chi-squared test is used to compare two or more categorical variables. The chi-squared test is a non-parametric test, meaning it does not make any assumptions about the distribution of data. It can be used to test a variety of hypotheses, including whether two or more groups have equal proportions.</p>
<h3 id="heading-step-4-decide-on-the-null-hypothesis-based-on-the-test-statistic-and-significance-level">Step #4 - Decide on the Null Hypothesis Based on the Test Statistic and Significance Level</h3>
<p>After conducting our test and calculating the test statistic, we can compare its value to the predetermined significance level. If the test statistic falls beyond the significance level, we can decide to reject the null hypothesis, indicating that there is sufficient evidence to support our alternative hypothesis. </p>
<p>On the other contrary, if the test statistic does not exceed the significance level, we fail to reject the null hypothesis, signifying that we do not have enough statistical evidence to conclude in favor of the alternative hypothesis.</p>
<h3 id="heading-step-5-interpret-the-results">Step #5 - Interpret the Results</h3>
<p>Depending on the decision made in the previous step, we can interpret the result in the context of our study and the practical implications. For our case study, we can interpret whether we have significant evidence to support our claim that the average weight of 10 year old children is more than 32kg or not.</p>
<p>For our test, we are generating random dummy data for the weight of the children. We'll use a t-test to evaluate whether our hypothesis is correct or not.</p>
<pre><code class="lang-python"><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> scipy.stats <span class="hljs-keyword">as</span> stats

<span class="hljs-comment"># Create a dummy dataset of 10 year old children's weight</span>
data = np.random.randint(<span class="hljs-number">20</span>, <span class="hljs-number">40</span>, <span class="hljs-number">10</span>)

<span class="hljs-comment"># Define the null hypothesis</span>
H0 = <span class="hljs-string">"The average weight of 10 year old children is 32kg."</span>

<span class="hljs-comment"># Define the alternative hypothesis</span>
H1 = <span class="hljs-string">"The average weight of 10 year old children is more than 32kg."</span>

<span class="hljs-comment"># Calculate the test statistic</span>
t_stat, p_value = stats.ttest_1samp(data, <span class="hljs-number">32</span>)

<span class="hljs-comment"># Print the results</span>
print(<span class="hljs-string">"Test statistic:"</span>, t_stat)
print(<span class="hljs-string">"p-value:"</span>, p_value)

<span class="hljs-comment"># Conclusion</span>
<span class="hljs-keyword">if</span> p_value &lt; <span class="hljs-number">0.05</span>:
    print(<span class="hljs-string">"Reject the null hypothesis."</span>)
<span class="hljs-keyword">else</span>:
    print(<span class="hljs-string">"Fail to reject the null hypothesis."</span>)
</code></pre>
<p>For a better understanding, let's look at what each block of code does.</p>
<pre><code class="lang-python"><span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> scipy.stats <span class="hljs-keyword">as</span> stats
</code></pre>
<p>The first block is the import statement, where we import <code>numpy</code> and <code>scipy.stats</code>. Numpy is a Python library used for scientific computing. It has a large library of functions for working with arrays. Scipy is a library for mathematical functions. It has a stat module for performing statistical functions, and that's what we'll be using for our t-test.</p>
<pre><code class="lang-python"><span class="hljs-comment"># Create a dummy dataset of 10 year old children's weight</span>
data = np.random.randint(<span class="hljs-number">20</span>, <span class="hljs-number">40</span>, <span class="hljs-number">100</span>)
</code></pre>
<p>The weights of the children were generated at random since we aren't working with an actual dataset. The random module within the Numpy library provides a function for generating random numbers, which is <code>randint</code>. </p>
<p>The <code>randint</code> function takes three arguments. The first (20) is the lower bound of the random numbers to be generated. The second (40) is the upper bound, and the third (100) specifies the number of random integers to generate. That is, we are generating random weight values for 100 children. In real circumstances, these weight samples would have been obtained by taking the weight of the required number of children needed for the test.</p>
<pre><code class="lang-python"><span class="hljs-comment"># Define the null hypothesis</span>
H0 = <span class="hljs-string">"The average weight of 10 year old children is 32kg."</span>

<span class="hljs-comment"># Define the alternative hypothesis</span>
H1 = <span class="hljs-string">"The average weight of 10 year old children is more than 32kg."</span>
</code></pre>
<p>Using the code above, we declared our null and alternate hypotheses stating the average weight of a 10-year-old in both cases.</p>
<pre><code class="lang-python"><span class="hljs-comment"># Calculate the test statistic</span>
t_stat, p_value = stats.ttest_1samp(data, <span class="hljs-number">32</span>)
</code></pre>
<p><code>t_stat</code> and <code>p_value</code> are the variables in which we'll store the results of our functions. <code>stats.ttest_1samp</code> is the function that calculates our test. It takes in two variables, the first is the <code>data</code> variable that stores the array of weights for children, and the second (32) is the value against which we'll test the mean of our array of weights or dataset in cases where we are using a real-world dataset.</p>
<pre><code class="lang-python">
<span class="hljs-comment"># Print the results</span>
print(<span class="hljs-string">"Test statistic:"</span>, t_stat)
print(<span class="hljs-string">"p-value:"</span>, p_value)
</code></pre>
<p>The code above prints both values for <code>t_stats</code> and <code>p_value</code>.</p>
<pre><code class="lang-python"><span class="hljs-comment"># Conclusion</span>
<span class="hljs-keyword">if</span> p_value &lt; <span class="hljs-number">0.05</span>:
    print(<span class="hljs-string">"Reject the null hypothesis."</span>)
<span class="hljs-keyword">else</span>:
    print(<span class="hljs-string">"Fail to reject the null hypothesis."</span>)
</code></pre>
<p>Lastly, we evaluated our <code>p_value</code> against our significance value, which is 0.05. If our <code>p_value</code> is less than 0.05, we reject the null hypothesis. Otherwise, we fail to reject the null hypothesis. Below is the output of this program. Our null hypothesis was rejected.</p>
<pre><code class="lang-python">Test statistic: <span class="hljs-number">-5.114430435590074</span>
p-value: <span class="hljs-number">1.541000376540265e-06</span>
Reject the null hypothesis.
</code></pre>
<h2 id="heading-conclusion">Conclusion</h2>
<p>In this article, we discussed the importance of hypothesis testing. We highlighted how science has advanced human knowledge and civilization through formulating and testing hypotheses.</p>
<p>We discussed Type I and Type II errors in hypothesis testing and how they underscore the importance of careful consideration and analysis in scientific inquiry. It reinforces the idea that conclusions should be drawn based on thorough statistical analysis rather than assumptions or biases.</p>
<p>We also generated a sample dataset using the relevant Python libraries and used the needed functions to calculate and test our alternate hypothesis.</p>
<p>Thank you for reading! Please follow me on <a target="_blank" href="https://www.linkedin.com/in/ogbemi-ejegi/">LinkedIn</a> where I also post more data related content.</p>
 ]]>
                </content:encoded>
            </item>
        
            <item>
                <title>
                    <![CDATA[ Data Visualization with Matplotlib – a Step by Step Guide ]]>
                </title>
                <description>
                    <![CDATA[ SEE is a beautiful Apple TV series that depicts a dystopia where humans have lost their sight. Hundreds of years later, it was considered a myth that people could ever see.  Jason Momoa is one of the leads and plays Baba Voss, an elite warrior. Jason... ]]>
                </description>
                <link>https://www.freecodecamp.org/news/introduction-to-data-vizualization-using-matplotlib/</link>
                <guid isPermaLink="false">66c71f7455df43f1418b5b21</guid>
                
                    <category>
                        <![CDATA[ data visualization ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Matplotlib ]]>
                    </category>
                
                    <category>
                        <![CDATA[ Python ]]>
                    </category>
                
                <dc:creator>
                    <![CDATA[ Mene-Ejegi Ogbemi ]]>
                </dc:creator>
                <pubDate>Mon, 24 Apr 2023 18:32:32 +0000</pubDate>
                <media:content url="https://www.freecodecamp.org/news/content/images/2023/04/data-visualization-1.jpg" medium="image" />
                <content:encoded>
                    <![CDATA[ <p><a target="_blank" href="https://tv.apple.com/us/show/see/umc.cmc.3s4mgg2y7h95fks9gnc4pw13m">SEE</a> is a beautiful Apple TV series that depicts a dystopia where humans have lost their sight. Hundreds of years later, it was considered a myth that people could ever see. </p>
<p>Jason Momoa is one of the leads and plays Baba Voss, an elite warrior. Jason's wife gives birth to sighted twins, and years after, during battle, Baba Voss sometimes needs the aid of the sighted children. They helped him understand the terrain better, even with his battlefield mastery. We could say his children helped him visualize things.</p>
<p>In ancient times, before digital devices, data visualization was also a myth. Earlier humans understood the need for visualization, so they had resources like maps, hieroglyphs, rock art, and so on. Eyewitnesses typically draw their paths and other relevant information on stones, wood, or scrolls. </p>
<p>Like Baba Voss's kids, these resources make it easier for humans to have a visual perspective on things or environments. </p>
<p>So what does visualization actually mean in this context? We can define visualization as "any technique for creating images, diagrams, or animations to communicate a message." (<a target="_blank" href="https://en.wikipedia.org/wiki/Visualization_(graphics)">source</a>)</p>
<p>In this article, we'll explore what data visualization is and how you can use the data visualization tool Matplotlib to explore and analyze data. You'll learn how to use it to create charts that help business owners and stakeholders get more insight about data and make informed decisions.</p>
<h2 id="heading-what-is-data-visualization">What is Data Visualization?</h2>
<p>Data visualization refers to the integration of data and visual elements like images, charts, diagrams, and so on to communicate messages to different stakeholders. </p>
<p>These stakeholders can be users, team members, managers, or top executive members of an organization. </p>
<p>Data in this context refers to different input gathered from the organization database or gotten from external sources, like public databases or private organizations, that have given access through their APIs.</p>
<p>We'll work with an employee layoff dataset which contains details of employees that have been laid off in different industries from 2020 to 2022. The columns in the dataset include the names of companies, locations, industries, total laid off, percentage laid off, date, countries, and other relevant columns. </p>
<p>Below is a snapshot of the data frame:</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2024/08/layoff-table.png" alt="Image" width="600" height="400" loading="lazy"></p>
<h2 id="heading-what-is-matplotlib">What is Matplotlib?</h2>
<p>Matplotlib is a popular Python library for displaying data and creating static, animated, and interactive plots. This program lets you draw appealing and informative graphics like line plots, scatter plots, histograms, and bar charts. </p>
<p>Matplotlib is highly customizable and flexible, which makes it a preferred choice for data analysts and scientists working in fields such as finance, science, engineering, and social sciences. </p>
<p>In this article, I'll show you how to create a bar chart, a pie chart, and a line plot to explain how you can do data visualization using Matplotlib.</p>
<p>The first thing you need is to import the Matplotlib and other relevant libraries like Pandas, Numpy and their sub modules.</p>
<pre><code class="lang-python"><span class="hljs-comment">#Imports packages</span>
<span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd
<span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt
<span class="hljs-keyword">import</span> numpy <span class="hljs-keyword">as</span> np
<span class="hljs-keyword">import</span> matplotlib.dates <span class="hljs-keyword">as</span> mdates
<span class="hljs-keyword">from</span> matplotlib.ticker <span class="hljs-keyword">import</span> MaxNLocator
</code></pre>
<p>In the code above, we import the Pandas package, which analyzes and manipulates our data. We imported Matplotlib and we'll use the Pyplot module for data visualization. </p>
<p>We'll use the Numpy package imported in the third line for numerical computations. We'll also work with the date module for date manipulations when plotting our chart. The last module is the ticker module, which sets ticks on plot axes. With these modules, you can analyze, manipulate, compute, and visualize your data.</p>
<h2 id="heading-how-to-create-a-bar-chart">How to Create a Bar Chart</h2>
<p>Bar charts help you with categorical values. That is, if you want to compare different entities on quantity, a bar chart is an excellent way to visualize it. In the layoff dataset, we'll compare different companies that laid off employees according to the number of staff laid off.</p>
<pre><code class="lang-python">plt.figure(figsize= (<span class="hljs-number">8</span>, <span class="hljs-number">6</span>))
industry_val = df_layoffs.groupby(<span class="hljs-string">'company'</span>)[<span class="hljs-string">'total_laid_off'</span>].sum().sort_values(ascending = <span class="hljs-literal">False</span>).head(<span class="hljs-number">10</span>)
industry_val.plot(label=<span class="hljs-string">""</span>, kind=<span class="hljs-string">'bar'</span>)
plt.show()
</code></pre>
<p>The code above is one way to create a bar chart. It shows the top 10 companies with the highest number of layoffs. </p>
<p>We first set the size of the graph to 8 inches by 6 inches. Then, we group our data in the dataframe by the sum total of employees laid off by each company. We then sort in descending order and select the top 10 with the highest layoffs. Finally, we create our bar chart using the selected data. The last line (<code>plt.show()</code>) displays the graph which is shown below.</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/04/layoff-bar-chart.png" alt="Image" width="600" height="400" loading="lazy"></p>
<p>From the chart above, you will notice that Meta and Amazon had the highest number of laid off staff while Twitter had the fewest layoffs.</p>
<h3 id="heading-how-to-create-a-pie-chart">How to Create a Pie Chart</h3>
<p>A pie chart represents a whole sector, with each portion allocated according to its size to a sub-sector. The industry column will be a perfect fit for using pie chart. We'll see which industry had most and fewest layoffs.</p>
<pre><code class="lang-python"><span class="hljs-comment"># Group the data by industry and sum the total laid off employees</span>
industry_val = df_layoffs.groupby(<span class="hljs-string">'industry'</span>)[<span class="hljs-string">'total_laid_off'</span>].sum().sort_values(ascending=<span class="hljs-literal">False</span>).head()

<span class="hljs-comment"># create the pie chart and display the labels and values inside the pie</span>
plt.figure(figsize=(<span class="hljs-number">8</span>, <span class="hljs-number">6</span>))
plt.pie(industry_val, labels=industry_val.index, autopct=<span class="hljs-string">'%1.1f%%'</span>)
plt.title(<span class="hljs-string">'Laid Off Employees by Industry'</span>)
plt.show()
</code></pre>
<p>First, the code groups the data by industry and sums up the total number of laid-off employees for each industry. It then sorts the industries in descending order based on the total number of laid-off employees and selects the top values using the <code>head()</code> function.</p>
<p>Next, we create a pie chart to visualize the data. The size of each slice in the pie represents the proportion of laid-off employees in that industry. The pie chart labels show the names of the industries. The percentage values inside the slices show the proportion of laid-off employees in that industry. The chart is titled "Laid Off Employees by Industry."</p>
<p>Finally, the pie chart is displayed using the <code>plt.show()</code> function. Like we did in the bar chart, the <code>plt.figure(figsize=(8, 6))</code> function sets the chart size to be 8 inches wide and 6 inches tall.</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/04/layoff-industry.png" alt="Image" width="600" height="400" loading="lazy"></p>
<p>The chart above shows the proportion of layoffs across different industries. The transportation sector and consumer sector are the industry mostly affected followed by retail, finance and food industry.</p>
<h3 id="heading-how-to-create-a-line-chart">How to Create a Line chart</h3>
<p>Line charts show changes over time for an entity. With our dataset, a line chart could be used to show the trend of layoffs over the past year or two.  This depends on what you are trying to communicate, but we'll work with a one year analysis.</p>
<pre><code class="lang-python"><span class="hljs-comment"># convert date column to datetime object</span>
df_layoffs[<span class="hljs-string">'date'</span>] = pd.to_datetime(df_layoffs[<span class="hljs-string">'date'</span>])

<span class="hljs-comment"># select data for one-year duration starting from January 1st, 2022</span>
start_date = pd.Timestamp(<span class="hljs-string">'2022-01-01'</span>)
end_date = start_date + pd.DateOffset(years=<span class="hljs-number">1</span>)
df_one_year = df_layoffs.loc[(df_layoffs[<span class="hljs-string">'date'</span>] &gt;= start_date) &amp; (df_layoffs[<span class="hljs-string">'date'</span>] &lt; end_date)]

<span class="hljs-comment"># plot the selected data</span>
df_date = df_one_year.groupby(<span class="hljs-string">'date'</span>)[<span class="hljs-string">'total_laid_off'</span>].sum()
plt.figure(figsize=(<span class="hljs-number">10</span>, <span class="hljs-number">4</span>))
plt.plot(df_date.index, df_date.values)
plt.xlabel(<span class="hljs-string">'Date'</span>)
plt.ylabel(<span class="hljs-string">'Total Laid Off'</span>)
plt.title(<span class="hljs-string">'Laid Off Trend for 2022'</span>)
plt.xticks(rotation=<span class="hljs-number">45</span>)
<span class="hljs-comment"># set the format of the x-axis labels to show Month-Year</span>
date_fmt = mdates.DateFormatter(<span class="hljs-string">'%b-%Y'</span>)
plt.gca().xaxis.set_major_formatter(date_fmt)

<span class="hljs-comment"># Use MaxNLocator to reduce the number of xticks</span>
locator = MaxNLocator(nbins=<span class="hljs-number">10</span>)
plt.gca().xaxis.set_major_locator(locator)

plt.show()
</code></pre>
<p>In comparison to the bar charts and pie charts, this code is much more challenging. But here is an explanation:</p>
<p>The first line of the code converts the 'date' column of the DataFrame (df_layoffs) into a DateTime object so that the dates can be handled easily.</p>
<pre><code class="lang-python"><span class="hljs-comment"># convert date column to datetime object</span>
df_layoffs[<span class="hljs-string">'date'</span>] = pd.to_datetime(df_layoffs[<span class="hljs-string">'date'</span>])
</code></pre>
<p>Next, we select the data for a one-year duration starting on January 1st, 2022. The start date is defined as a Timestamp object, and the end date is set as one year from the start date using the pd.DateOffset function. The loc function is then used to filter the DataFrame rows, selecting only those that fall within this one-year duration. Remember we are working with a year's data.</p>
<pre><code class="lang-python"><span class="hljs-comment"># select data for one-year duration starting from January 1st, 2022</span>
start_date = pd.Timestamp(<span class="hljs-string">'2022-01-01'</span>)
end_date = start_date + pd.DateOffset(years=<span class="hljs-number">1</span>)
df_one_year = df_layoffs.loc[(df_layoffs[<span class="hljs-string">'date'</span>] &gt;= start_date) &amp; (df_layoffs[<span class="hljs-string">'date'</span>] &lt; end_date)]
</code></pre>
<p>After that, we group the selected data by date and calculate the total number of layoffs on each date using the groupby and sum functions. This is stored in a new DataFrame called <code>df_date</code>.</p>
<pre><code class="lang-python"><span class="hljs-comment"># plot the selected data</span>
df_date = df_one_year.groupby(<span class="hljs-string">'date'</span>)[<span class="hljs-string">'total_laid_off'</span>].sum()
</code></pre>
<p>Then, we create a plot of the laid off trend for 2022 using the matplotlib library. The plot size is set to (10, 4) using the figure function.</p>
<pre><code class="lang-python">plt.figure(figsize=(<span class="hljs-number">10</span>, <span class="hljs-number">4</span>))
</code></pre>
<p>The x-axis represents the date, and the y-axis represents the total number of layoffs. The xlabel function labels the x-axis as 'Date,' and the ylabel function labels the y-axis as 'Total Laid Off.'</p>
<pre><code class="lang-python">plt.plot(df_date.index, df_date.values)
plt.xlabel(<span class="hljs-string">'Date'</span>)
plt.ylabel(<span class="hljs-string">'Total Laid Off'</span>)
</code></pre>
<p>The plot title is set to 'Laid Off Trend for 2022' using the title function.</p>
<pre><code class="lang-python">plt.title(<span class="hljs-string">'Laid Off Trend for 2022'</span>)
</code></pre>
<p>The x-axis labels are rotated by 45 degrees using the xticks function to avoid overcrowding.</p>
<pre><code class="lang-python">plt.xticks(rotation=<span class="hljs-number">45</span>)
</code></pre>
<p>The format of the x-axis labels is set to show the Month-Year format using the DateFormatter function.</p>
<pre><code class="lang-python"><span class="hljs-comment"># set the format of the x-axis labels to show Month-Year</span>
date_fmt = mdates.DateFormatter(<span class="hljs-string">'%b-%Y'</span>)
plt.gca().xaxis.set_major_formatter(date_fmt)
</code></pre>
<p>Finally, the number of xticks on the plot is reduced using the MaxNLocator function, which reduces the number of xticks to 10.</p>
<pre><code class="lang-python"><span class="hljs-comment"># Use MaxNLocator to reduce the number of xticks</span>
locator = MaxNLocator(nbins=<span class="hljs-number">10</span>)
plt.gca().xaxis.set_major_locator(locator)
</code></pre>
<p>The plot is then displayed using the show function.</p>
<pre><code class="lang-python">plt.show()
</code></pre>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/04/layoff-line-chart.png" alt="Image" width="600" height="400" loading="lazy"></p>
<p>The chart above shows layoff trends and patterns for 2022.</p>
<p>You can also analyze how well an entity performed over different periods of time. The second chart shows an analysis of employee layoffs in 2020 versus 2022.</p>
<pre><code class="lang-python"><span class="hljs-keyword">import</span> pandas <span class="hljs-keyword">as</span> pd
<span class="hljs-keyword">import</span> matplotlib.pyplot <span class="hljs-keyword">as</span> plt
<span class="hljs-keyword">import</span> matplotlib.dates <span class="hljs-keyword">as</span> mdates
<span class="hljs-keyword">from</span> matplotlib.ticker <span class="hljs-keyword">import</span> MaxNLocator

<span class="hljs-comment"># convert date column to datetime object</span>
df_layoffs[<span class="hljs-string">'date'</span>] = pd.to_datetime(df_layoffs[<span class="hljs-string">'date'</span>])

<span class="hljs-comment"># filter data to only include 2020 and 2022</span>
df_filtered = df_layoffs[(df_layoffs[<span class="hljs-string">'date'</span>].dt.year == <span class="hljs-number">2020</span>) | (df_layoffs[<span class="hljs-string">'date'</span>].dt.year == <span class="hljs-number">2022</span>)]

<span class="hljs-comment"># group data by year and calculate total layoffs</span>
df_filtered[<span class="hljs-string">'year'</span>] = df_filtered[<span class="hljs-string">'date'</span>].dt.year
df_yearly = df_filtered.groupby([<span class="hljs-string">'year'</span>, <span class="hljs-string">'date'</span>])[<span class="hljs-string">'total_laid_off'</span>].sum().reset_index()

<span class="hljs-comment"># create subplots and plot the data for each year in separate charts</span>
fig, axs = plt.subplots(ncols=<span class="hljs-number">2</span>, figsize=(<span class="hljs-number">14</span>, <span class="hljs-number">8</span>))
<span class="hljs-keyword">for</span> i, year <span class="hljs-keyword">in</span> enumerate(df_yearly[<span class="hljs-string">'year'</span>].unique()):
    df_year = df_yearly.loc[df_yearly[<span class="hljs-string">'year'</span>] == year]
    axs[i].plot(df_year[<span class="hljs-string">'date'</span>], df_year[<span class="hljs-string">'total_laid_off'</span>])
    axs[i].set_xlabel(<span class="hljs-string">'Date'</span>)
    axs[i].set_ylabel(<span class="hljs-string">'Total Laid Off'</span>)
    axs[i].set_title(<span class="hljs-string">f'Laid Off Trend for <span class="hljs-subst">{year}</span>'</span>)
    axs[i].xaxis.set_major_formatter(mdates.DateFormatter(<span class="hljs-string">'%b-%Y'</span>))
    axs[i].tick_params(axis=<span class="hljs-string">'x'</span>, rotation=<span class="hljs-number">45</span>)
    locator = MaxNLocator(nbins=<span class="hljs-number">10</span>)
    axs[i].xaxis.set_major_locator(locator)

<span class="hljs-comment"># set y-axis limit to 0-14000 for each subplot</span>
<span class="hljs-keyword">for</span> ax <span class="hljs-keyword">in</span> axs:
    ax.set_ylim([<span class="hljs-number">0</span>, <span class="hljs-number">14000</span>])

plt.show()
</code></pre>
<p>Let's review the different components of the code above.</p>
<p>The 'date' column in the DataFrame is converted to a datetime object.</p>
<pre><code class="lang-python"><span class="hljs-comment"># convert date column to datetime object</span>
df_layoffs[<span class="hljs-string">'date'</span>] = pd.to_datetime(df_layoffs[<span class="hljs-string">'date'</span>])
</code></pre>
<p>Next, the code filters the data to only include layoffs from the years 2020 and 2022. It then groups the filtered data by year and date and calculates the total number of layoffs for each date.</p>
<pre><code class="lang-python"><span class="hljs-comment"># filter data to only include 2020 and 2022</span>
df_filtered = df_layoffs[(df_layoffs[<span class="hljs-string">'date'</span>].dt.year == <span class="hljs-number">2020</span>) | (df_layoffs[<span class="hljs-string">'date'</span>].dt.year == <span class="hljs-number">2022</span>)]

<span class="hljs-comment"># group data by year and calculate total layoffs</span>
df_filtered[<span class="hljs-string">'year'</span>] = df_filtered[<span class="hljs-string">'date'</span>].dt.year
df_yearly = df_filtered.groupby([<span class="hljs-string">'year'</span>, <span class="hljs-string">'date'</span>])[<span class="hljs-string">'total_laid_off'</span>].sum().reset_index()
</code></pre>
<p>We then create two subplots and plot the total number of layoffs for each year in separate charts. We set the x-axis labels to the date format of 'MMM-YYYY' (for example, Jan-2022) and rotate them by 45 degrees. We also set the y-axis label to 'Total Laid Off' and the chart title to 'Laid Off Trend for {year}' (for example, Laid Off Trend for 2020). Finally, we show the charts using the <code>plt.show()</code> command.</p>
<pre><code class="lang-python"><span class="hljs-comment"># create subplots and plot the data for each year in separate charts</span>
fig, axs = plt.subplots(ncols=<span class="hljs-number">2</span>, figsize=(<span class="hljs-number">14</span>, <span class="hljs-number">8</span>))
<span class="hljs-keyword">for</span> i, year <span class="hljs-keyword">in</span> enumerate(df_yearly[<span class="hljs-string">'year'</span>].unique()):
    df_year = df_yearly.loc[df_yearly[<span class="hljs-string">'year'</span>] == year]
    axs[i].plot(df_year[<span class="hljs-string">'date'</span>], df_year[<span class="hljs-string">'total_laid_off'</span>])
    axs[i].set_xlabel(<span class="hljs-string">'Date'</span>)
    axs[i].set_ylabel(<span class="hljs-string">'Total Laid Off'</span>)
    axs[i].set_title(<span class="hljs-string">f'Laid Off Trend for <span class="hljs-subst">{year}</span>'</span>)
    axs[i].xaxis.set_major_formatter(mdates.DateFormatter(<span class="hljs-string">'%b-%Y'</span>))
    axs[i].tick_params(axis=<span class="hljs-string">'x'</span>, rotation=<span class="hljs-number">45</span>)
    locator = MaxNLocator(nbins=<span class="hljs-number">10</span>)
    axs[i].xaxis.set_major_locator(locator)

plt.show()
</code></pre>
<p>Overall, the code is used to filter, group, and visualize data related to company layoffs specifically focusing on trends for 2020 and 2022. You can see the result in the chart below:</p>
<p><img src="https://www.freecodecamp.org/news/content/images/2023/04/line-chart-comparison-1.png" alt="Image" width="600" height="400" loading="lazy"></p>
<h2 id="heading-conclusion">Conclusion</h2>
<p>We started by discussing what visualization is and how data visualization is significant in transforming raw numbers into insight and business sense. </p>
<p>Then we used the popular Python library Matplotlib, which is a tool for data visualization, to create bar charts, pie charts, and line charts. There are also other use cases not covered in this article, like histograms, scatter plots, box plots, and so on. </p>
<p>By using these visualizations, we can make sense of our data and take actions that wouldn't be possible by looking at raw numbers. Data visualization can help us achieve better outcomes in other areas such as finance, science, engineering, etc. For further study, you can check the official matplotlib documentation <a target="_blank" href="https://matplotlib.org/stable/index.html">here</a>.</p>
<p>Thank you for reading! Please follow me on <a target="_blank" href="https://www.linkedin.com/in/ogbemi-ejegi/">LinkedIn</a> where I also post more data related content.</p>
 ]]>
                </content:encoded>
            </item>
        
    </channel>
</rss>
