Yet despite its impressive performance, GPT-3 revealed an important limitation: raw capability doesn't automatically create a useful assistant.

A language model can generate fluent text, answer questions, and solve complex tasks while still failing to follow what the user actually wants.

GPT-3 could produce responses that were inconsistent, overly confident, difficult to control, or misaligned with user instructions. It was a powerful prediction engine, but it wasn't designed to reliably act as a helpful assistant.

This challenge motivated one of the most influential papers in modern AI: Training Language Models to Follow Instructions with Human Feedback. Rather than making the model larger, the researchers focused on teaching it how to better follow human intent.

The result was InstructGPT, a system fine-tuned from GPT-3 that demonstrated how human feedback could transform a capable language model into a far more useful and aligned assistant.

This challenge became one of the most important problems in modern AI: alignment.

Researchers realized that building larger models was only part of the solution. While scaling improved capabilities, it didn't guarantee that models would reliably follow instructions or behave in ways that matched user expectations. The next stage of progress required teaching models how to respond in a more helpful, truthful, and safe manner.

This led to the development of instruction-following systems and Reinforcement Learning from Human Feedback (RLHF). Instead of optimizing models solely to predict the next word, researchers began training them to better align with human preferences and intentions.

This shift marked a major turning point in the evolution of large language models.

GPT-3 demonstrated the power of large-scale language modeling and introduced many people to prompting and few-shot learning.

InstructGPT built on that foundation by showing how human feedback could significantly improve instruction following and model behavior. ChatGPT then brought these ideas to a much broader audience by packaging aligned language models into an accessible conversational interface used by millions of people.

In many ways, language models became capable before they became aligned.

That's why the transition from GPT-3 to InstructGPT represents one of the most important milestones in the history of artificial intelligence. The focus was no longer only on making models more capable. It was also about making them more useful, reliable, and responsive to human intent.

The success of InstructGPT pioneered many of the alignment techniques that later became a core part of systems such as ChatGPT and GPT-4.

Paper Overview:

In this article, we’ll mainly focus on the paper Training Language Models to Follow Instructions with Human Feedback, published by OpenAI in 2022.

This paper introduced InstructGPT, one of the most important transitions in the history of large language models. While earlier GPT systems focused heavily on scaling model size and improving raw capabilities, this work shifted attention toward something equally important: alignment.

The paper explores how language models can be trained to better follow human instructions using reinforcement learning from human feedback (RLHF). Instead of optimizing only for next-token prediction, the model is further optimized to produce responses that humans actually prefer – responses that are more helpful, safer, and more aligned with user intent.

What makes this paper historically important is that it became the foundation for the modern ChatGPT alignment pipeline.

Many of the interaction patterns people now associate with ChatGPT (like instruction following, conversational behavior, refusal handling, and safer responses) can be traced directly back to the ideas introduced here.

Here’s the original paper again if you want to explore it directly: Training language models to follow instructions with human feedback

And here’s a quick infographic of what we’ll cover throughout this review:

Table of Contents:

• Executive Summary

• The Core Problem

• Why GPT-3 Was Not Enough

• InstructGPT: The Birth of Alignment-Centered LLMs

• RLHF Pipeline: How InstructGPT Learned to Behave Like an Assistant

  • Stage 1 — Supervised Fine-Tuning (SFT)

  • Stage 2 — Reward Model Training

  • Stage 3 — PPO Reinforcement Learning

• Helpful, Honest, Harmless

• Human Feedback as the New Scaling Factor

• Why ChatGPT Exploded Globally

• ChatGPT as an Interface Revolution

• Benchmarks and Results

• Truthfulness and Hallucinations

• Safety and Refusal Behavior

• Limitations

• Historical Importance

• Discussion: The Real Shift

• Connection to GPT-4

• GPT-3 vs InstructGPT vs ChatGPT vs GPT-4: Key Differences

• From GPT-1 to GPT-4: A Timeline of Modern AI Systems and Alignment Evolution

• Final Insight

• Resources

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with a few foundational ideas.

Reading the previous reviews in this series will be especially helpful:

• AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)

• AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)

• AI Paper Review: Language Models are Few-Shot Learners (GPT-3)

Even though GPT-4 was released after InstructGPT, reading the GPT-4 review can still be helpful. It provides a broader view of how alignment techniques evolved and how they were combined with stronger reasoning and multimodal capabilities in later generations of GPT models.

• AI Paper Review: GPT-4 Technical Report (GPT-4)

It also helps to have:

• A general understanding of natural language processing (NLP) and large language models

• A high-level idea of Transformer-based autoregressive models

• Familiarity with prompting, few-shot learning, and in-context learning

• A basic understanding of reinforcement learning and human feedback systems

• General machine learning concepts like training data, fine-tuning, scaling, and inference

• Some familiarity with alignment, safety, and AI behavior control concepts

You don't need to be an AI researcher to follow this article, though.

I’ll keep the explanations practical and intuitive, focusing more on understanding how InstructGPT changed modern AI systems rather than getting lost in dense mathematical details or academic terminology.

Executive Summary

The paper Training Language Models to Follow Instructions with Human Feedback marks one of the biggest turning points in the history of modern AI systems. Instead of asking only how to make language models larger or smarter, OpenAI focused on a different question: how do we make these models actually helpful for real people?

The paper introduces InstructGPT, a version of GPT-3 fine-tuned to follow human instructions more accurately using a method called Reinforcement Learning from Human Feedback (RLHF).

The core insight of the paper is simple but extremely important:

Bigger language models don't automatically become better assistants.

Even highly capable models like GPT-3 could still:

• ignore instructions

• hallucinate facts

• generate toxic or biased outputs

• produce responses that were technically fluent but not actually useful to users

To solve this problem, OpenAI built a multi-stage alignment pipeline: humans first demonstrate ideal answers, humans then rank model outputs, and finally the model learns from those preferences using reinforcement learning.

This changed the direction of modern AI development.

The paper shows that alignment and usability can matter more than raw model size itself. One of the most surprising findings was that the 1.3B InstructGPT model was often preferred by human evaluators over the original 175B GPT-3 model, despite being dramatically smaller.

The paper also demonstrates improvements in instruction following, truthfulness, toxicity reduction, conversational behavior, and general user preference.

Historically, this paper became the foundation behind modern conversational AI systems.

GPT-3 proved that language models could learn from prompts.

GPT-4 later proved that scaling and multimodal reasoning could unlock even stronger capabilities.

But InstructGPT showed something equally important: AI systems must be aligned with human intent to become truly usable products.

In many ways, this paper represents the transition from raw language modeling to aligned assistants, capability scaling to behavior shaping, and research demos to real-world conversational AI systems.

And that transition eventually led directly to ChatGPT.

The Core Problem

One of the most important ideas in this paper is that raw language modeling is not the same thing as building a useful assistant.

Before InstructGPT, models like GPT-3 were trained mainly with a simple objective: predict the next token in a sequence.

That objective made language models extremely powerful at generating fluent text, but it also created a major limitation. The model learned how to continue internet text, not necessarily how to help humans.

This became one of the defining realizations behind modern AI alignment research.

Despite its impressive capabilities, GPT-3 often struggled to behave like a reliable assistant. The model could produce fluent text, but it was not explicitly trained to follow user intent.

Here are some examples that highlight the differences between GPT-3 and InstructGPT in how they respond to user prompts:

Source: Aligning language models to follow instructions

Source: Aligning language models to follow instructions

These examples reveal the central weakness of early GPT systems. GPT-3 often continued the pattern of the prompt rather than completing the requested task. InstructGPT, by contrast, responded directly to the user's instruction. The difference wasn't a matter of raw intelligence. It was a difference in training objectives.

GPT models were trained on massive internet-scale datasets where the goal was simply to predict what text comes next. As a result, the model optimized for plausibility, continuation, and pattern completion. Not necessarily for truthfulness, safety, helpfulness, or alignment with human goals.

This created a major gap between: language capability and useful assistant behavior.

For example, if a user asked a harmful, misleading, or nonsensical question, the model might still attempt to continue the pattern naturally instead of recognizing the issue. In many cases, the model behaved more like an internet text simulator than a reliable assistant.

The paper repeatedly emphasizes that scaling alone couldn't solve this problem.

Researchers increasingly recognized that better behavior would require more than scaling alone.

Models also needed stronger instruction following, better alignment with human intent, improved safety behavior, greater truthfulness, and optimization around real user needs.

Why GPT-3 Was Not Enough

When GPT-3 was released, it felt like a massive leap forward in AI capabilities.

The model could perform few-shot learning, answer questions, summarize text, generate code, translate languages, and even solve certain reasoning tasks: all without traditional fine-tuning. For many researchers, it was the first time a language model started to feel genuinely general-purpose.

Yet using GPT-3 in practice was often less reliable than its benchmark performance suggested.

In practice, using GPT-3 often required careful prompt engineering. Small wording changes could completely change the quality of the response. Sometimes the model followed instructions well, and other times it ignored them entirely.

Users often found themselves rewriting prompts repeatedly to obtain the response they actually wanted.

This became the core motivation behind InstructGPT.

OpenAI responded by exploring ways to make model behavior more consistent, predictable, and useful for users.

InstructGPT: The Birth of Alignment-Centered LLMs

The release of InstructGPT marked one of the biggest shifts in the history of large language models.

Before InstructGPT, most advances in language models came from scaling data, compute, and model size.

The focus shifted toward alignment: building systems that could follow instructions more reliably and behave in ways users actually preferred.

This is where InstructGPT introduced one of the most important ideas in modern AI systems: Reinforcement Learning from Human Feedback (RLHF).

Instead of optimizing models only to predict internet text, OpenAI started optimizing models based on what humans actually preferred. Human labelers ranked model outputs, and those preferences became part of the training process itself.

This fundamentally changed the objective of language models.

Rather than optimizing solely for next-token prediction, the system was increasingly optimized to produce responses that humans judged to be helpful, safe, and aligned with their intentions.

That distinction may sound subtle, but it completely changed the direction of AI development.

InstructGPT combined instruction-following training with human preference optimization, creating a model whose behavior could be shaped directly through feedback rather than solely through pretraining.

The model was no longer trained only to imitate the internet. It was trained to behave more like an assistant.

RLHF Pipeline: How InstructGPT Learned to Behave Like an Assistant

At the center of the InstructGPT paper is a training pipeline that completely changed how modern AI assistants are built.

RLHF was designed to build on traditional language-model pretraining rather than replace it.

The InstructGPT paper introduced a different idea: instead of training models only on internet text, why not train them using human preferences directly?

This led to the development of the RLHF pipeline: Reinforcement Learning from Human Feedback. This approach would later become a standard component of modern conversational AI systems.

The paper’s Figure 2 is especially important because it visualizes the entire alignment pipeline introduced by OpenAI. Rather than relying on a single training stage, the system uses multiple stages where human feedback gradually shapes model behavior.

Source: Training Language Models to Follow Instructions with Human Feedback (OpenAI, 2022).

As you can see in the image above, the process happens in three major stages.

Stage 1 — Supervised Fine-Tuning (SFT)

The first stage starts with human-written demonstrations.

Labelers are given prompts and asked to write ideal responses – the kinds of answers a helpful assistant should produce. These examples become the initial training dataset for the model.

At this stage, the model learns the basic patterns of assistant-style responses.

This is still traditional supervised learning, but the goal is different from standard language modeling. Instead of learning only from web text, the model now learns from examples of preferred assistant behavior.

This stage creates what the paper calls the Supervised Fine-Tuned model (SFT model).

And while this already improves behavior significantly, OpenAI realized something important: human preferences are more complex than simple “correct answers.”

There are often many possible responses to a prompt, but humans may strongly prefer some answers over others.

That leads to the next stage.

Stage 2 — Reward Model Training

In the second stage, humans no longer write responses directly.

Instead, the model generates multiple answers for the same prompt, and human labelers rank them from best to worst.

For a given prompt, one response may be clearer, another more accurate, and another safer or more appropriate. Human labelers rank these alternatives according to their preferences

The rankings are then used to train a separate neural network called the Reward Model (RM).

This model learns something extremely important: which outputs humans prefer.

In other words, the system converts human preferences into a trainable reward signal.

This becomes one of the biggest conceptual breakthroughs in the paper. Instead of manually programming behavior rules, OpenAI trains the model to approximate human judgment itself.

The reward model captures patterns in human preferences and turns them into a training signal.

That reward signal becomes the foundation for the final training stage.

Stage 3 — PPO Reinforcement Learning

The final stage uses reinforcement learning to optimize the language model against the reward model.

More specifically, the paper uses PPO (Proximal Policy Optimization), a reinforcement learning algorithm commonly used in policy optimization tasks.

At this stage, the model generates responses, receives scores from the reward model, and gradually updates its behavior to maximize those scores.

The model gradually shifts toward responses that receive higher scores from the reward model.

The key innovation is that optimization now occurs against a learned representation of human preferences rather than only a language-modeling objective.

According to the paper, this RLHF pipeline significantly improved instruction following and user preference ratings while also reducing toxic and unsafe behavior.

And in many ways, this pipeline became the blueprint for the modern era of conversational AI systems.

Helpful, Honest, Harmless

The authors argue that evaluating language models requires more than measuring capability alone. They should also be evaluated by how they behave around humans.

At the time, this represented a significant shift in how researchers evaluated language models.

That is why the paper repeatedly emphasizes a new alignment philosophy centered around three goals:

• Helpful

• Honest

• Harmless

These ideas became the conceptual foundation behind modern alignment research and conversational AI systems.

Helpful

The first goal is straightforward: the model should genuinely help the user accomplish what they want.

In practice, helpfulness means following instructions clearly, answering questions directly, providing relevant information, and adapting to the user's intent.

This may seem simple, but it fundamentally changes the training objective.

The model is no longer optimized only for linguistic fluency. It's optimized for usefulness.

Honest

The second goal is honesty.

One of the biggest problems with large language models is that they often produce convincing answers even when those answers are wrong. The models can hallucinate facts, invent references, or respond confidently despite uncertainty.

The paper recognizes that a useful assistant shouldn't merely sound intelligent. It should also behave truthfully and acknowledge uncertainty when necessary.

This is especially important because language models are optimized to generate plausible text, not verified truth.

As a result, earlier models sometimes prioritized sounding coherent over being accurate.

The alignment process introduced in InstructGPT attempts to reduce this behavior through human feedback and preference optimization. Human evaluators consistently prefer responses that are more accurate, transparent, and reliable, and those preferences gradually shape the model during RLHF training.

The paper doesn't claim that hallucinations disappear completely. Far from it. But it marks one of the first large-scale attempts to explicitly optimize language models for truthfulness and reliability rather than pure text generation quality.

Harmless

The third goal is harmlessness.

Large language models trained on internet data inevitably absorb toxic, biased, unsafe, or harmful patterns from that data. Without alignment, models may generate dangerous instructions, offensive content, or manipulative behavior.

The paper directly addresses this concern and treats safety as a central part of model development.

Through RLHF and human preference ranking, the model learns to refuse certain harmful requests, avoid toxic generations, produce safer responses, and behave more responsibly during interaction.

This became one of the defining characteristics of modern conversational AI systems.

Instead of maximizing unrestricted generation, the system begins balancing usefulness, safety, and alignment with human values.

But the paper is also honest about limitations.

The authors acknowledge that harmful outputs, biases, and unsafe behavior can still appear. Alignment is imperfect, and human values themselves are complex and difficult to define universally.

But historically, this paper marks the moment when safety and alignment became core engineering goals rather than secondary concerns.

Taken together, these three principles (helpful, honest, and harmless) became much more than training objectives. They became the philosophical foundation behind ChatGPT-era AI systems.

Earlier GPT papers mainly explored how to scale intelligence. But InstructGPT explored something deeper: how to make intelligence usable for humans.

Human Feedback as the New Scaling Factor

One of the most fascinating ideas behind the InstructGPT paper is that it quietly changed what “scaling” meant in modern AI.

For years, progress in language models was largely measured through scaling.

GPT-1 showed that pretraining works. GPT-2 showed that larger models develop stronger zero-shot behavior. GPT-3 pushed this idea even further by scaling to 175 billion parameters and demonstrating impressive few-shot learning abilities.

And to some extent, that was true. Larger models became better at reasoning, code generation, language understanding, translation, and generalization.

That is where human feedback became central.

Instead of relying purely on internet-scale text, OpenAI introduced a training pipeline where human preferences directly shaped model behavior. Human labelers ranked responses, evaluated quality, and guided the system toward outputs people actually preferred.

In many ways, this created a completely new scaling dimension for AI systems:

• scaling human feedback

• scaling preference learning

• scaling alignment pipelines

Historically, this shifted attention from model scale alone toward the quality of model behavior

InstructGPT focused on scaling usability. And the results were surprisingly powerful.

According to the paper, a much smaller aligned model was often preferred over the original 175B GPT-3 model by human evaluators.

That finding changed how the industry thought about progress.

The result suggested that improving behavior could sometimes matter as much as increasing scale.

This is why RLHF became one of the defining ideas of the ChatGPT era.

After InstructGPT, modern AI systems were no longer evaluated only by benchmark scores, parameter counts, or scaling curves.

They were increasingly evaluated by usefulness, conversational quality, safety, reliability, and how well they interact with humans.

And that shift fundamentally changed the future direction of large language models.

Why ChatGPT Exploded Globally

When ChatGPT launched publicly, the reaction was immediate and unlike anything the AI industry had seen before.

Millions of people started using it within days. Developers, students, writers, researchers, businesses, and everyday users suddenly felt like they were interacting with AI in a completely different way.

What made this moment so important was that advanced AI capabilities finally became accessible to ordinary users. After all, the underlying language models were already extremely capable before ChatGPT existed. GPT-3 could generate essays, answer questions, write code, summarize text, and perform impressive few-shot learning tasks. GPT-4 later pushed reasoning and multimodal abilities even further.

The challenge was no longer whether language models could perform useful tasks, but whether people could interact with them naturally.

ChatGPT combined powerful language-model capabilities with RLHF-based alignment, conversational interaction, safer behavior, and a user-friendly chat interface.

Earlier systems often required significant prompt experimentation to achieve consistent results. Users had to carefully engineer prompts, retry questions, or work around strange outputs. The models could be brilliant one moment and confusing the next.

ChatGPT changed that experience dramatically.

Thanks to the alignment techniques introduced in the InstructGPT paper, the system became far better at following instructions, maintaining conversational flow, understanding intent, and responding in a way that felt cooperative rather than purely generative.

The conversational interface itself also mattered enormously.

Before ChatGPT, interacting with advanced AI systems often required APIs, coding knowledge, prompt experimentation, or technical understanding.

ChatGPT simplified everything into a familiar chat format: you simply typed naturally, and the system responded naturally.

That design decision may sound small, but historically it was transformative. It turned large language models from research tools into consumer products.

Although imperfect, the system felt substantially more reliable than earlier language-model interfaces.

The system was designed to communicate in ways that felt more natural and cooperative.

The breakthrough was not simply that the AI became smarter. The breakthrough was that the AI became usable.

And that usability is what transformed large language models from impressive research demonstrations into globally adopted AI assistants.

ChatGPT as an Interface Revolution

One of the most important things about ChatGPT is that it changed how humans interact with computers.

Before ChatGPT, powerful AI systems mostly lived behind APIs, research demos, developer tools, and complex prompting workflows.

Using advanced language models often required technical knowledge. Developers experimented with prompt engineering, API parameters, temperature settings, and carefully structured inputs just to get reliable outputs from the model.

Even GPT-3, despite being extremely powerful, still felt like a research system for many users. You had to learn how to “talk to the model.”

And in many cases, the interaction felt fragile. Slight changes in wording could completely change the quality of the response.

ChatGPT changed that dynamic almost overnight.

Instead of making users adapt to the AI, the AI became much better at adapting to humans.

Natural conversation became the interface.

For decades, human-computer interaction depended on commands, menus, search boxes, forms, programming languages, and specialized software interfaces.

ChatGPT introduced something different: you could simply explain what you wanted in plain language. And the system would usually understand.

This made AI feel accessible to people who had never written code, used APIs, or interacted with machine learning systems before.

In many ways, ChatGPT transformed prompting into a universal interface for computing. And that single shift affected nearly every digital field.

In education, students started using conversational AI to explain difficult concepts, summarize lessons, practice languages, and receive tutoring-style help.

In coding, developers began using AI systems for debugging, code generation, documentation, and learning new frameworks.

This eventually led to the rise of AI coding assistants integrated directly into development environments.

In writing and content creation, conversational AI became a brainstorming partner capable of drafting ideas, rewriting text, organizing articles, and helping people communicate more effectively.

Search behavior also started changing. Instead of searching through lists of links, users increasingly expected direct conversational answers. This fundamentally challenged traditional search-engine interaction models.

And across productivity tools, AI systems began acting less like software features and more like collaborative assistants.

This shift was enabled by advances in conversational AI and interaction design that made dialogue feel natural and useful.

The alignment techniques introduced by InstructGPT were an important part of making these conversational experiences practical.

Historically, this may become one of the most important consequences of the GPT era: earlier software required humans to learn interfaces. ChatGPT pushed computing toward interfaces that learn humans instead.

Benchmarks and Results

We've already discussed how one of the biggest improvement didn't come from making the model larger. Instead, it came from making the model better aligned with humans.

This is one of the central findings of the entire paper, and it changed how many researchers thought about progress in large language models.

Before this work, the dominant belief was that scaling was the main path forward, with bigger models, more parameters, more compute, and more data. And GPT-3 seemed to confirm that idea. Larger models consistently showed stronger few-shot learning, reasoning, and generalization abilities.

But the InstructGPT paper introduced a different perspective. The researchers found that a relatively small 1.3B parameter InstructGPT model was often preferred by human evaluators over the original 175B GPT-3 model.

That result was extremely important. It suggested that alignment sometimes outperformed scale.

This became one of the defining insights of the ChatGPT era.

According to the paper, human evaluators consistently preferred InstructGPT responses because they were more helpful, more accurate, safer, and better aligned with what users were actually asking for.

The improvements appeared across several important areas.

One major improvement was instruction following. Earlier GPT models often ignored instructions, drifted off-topic, or generated responses that sounded fluent but failed to solve the user’s actual task. InstructGPT behaved much more like a cooperative assistant and followed prompts more reliably.

The paper also reports improvements in truthfulness. Large language models are known for hallucinating information and confidently generating false statements. Through RLHF and preference optimization, InstructGPT reduced some of these behaviors and produced answers humans judged to be more truthful and reliable.

Another important improvement involved toxicity and harmful outputs. The researchers evaluated the system on toxicity benchmarks and found that aligned models generated fewer toxic or unsafe responses compared to earlier GPT systems.

What makes these findings historically important is that they changed the industry’s understanding of what “better AI” actually meant.

Before InstructGPT, improvement was mostly measured through benchmark scores, scaling curves, and parameter counts.

After InstructGPT, researchers increasingly focused on usability, safety, alignment, conversational quality, and human preference satisfaction.

This was a major shift in AI development philosophy.

Truthfulness and Hallucinations

A major challenge for language models is that fluent responses are not always truthful.

This behavior is now commonly called hallucination.

Hallucinations can take many forms, including invented facts, fabricated references, incorrect explanations, or confident answers that lack factual support.

And because the responses are fluent and natural, the mistakes can sometimes look believable to users. The InstructGPT paper treats this as a serious issue rather than a minor flaw.

The authors note that language models are optimized for plausibility rather than verified truth. This is an important distinction: a language model can generate text that looks correct while still being inaccurate.

This is why the paper places particular emphasis on truthfulness and factual reliability.

Through RLHF and human preference optimization, InstructGPT was trained to produce answers humans judged to be more accurate and trustworthy. Human evaluators generally preferred responses that were more transparent about uncertainty and less likely to contain misleading information.

The paper also evaluates the model on truthfulness benchmarks such as TruthfulQA, where aligned models demonstrated improvements compared to earlier GPT systems.

But the paper is also careful not to overstate the results. Hallucinations didn't disappear. The aligned models could still make reasoning mistakes, generate false information, misunderstand prompts, or produce overconfident answers.

This nuance is extremely important: the paper doesn't claim that RLHF solved factuality or reasoning completely. Instead, alignment improved behavior, not perfection.

That distinction became increasingly important as ChatGPT and later GPT-4 systems reached millions of users worldwide.

The models became more useful, more truthful, and more aligned, but they still remained probabilistic language models rather than guaranteed fact engines.

In many ways, the InstructGPT paper marks the beginning of large-scale efforts to make AI systems not only intelligent, but also trustworthy enough for real-world human interaction.

Safety and Refusal Behavior

As language models became more powerful, researchers realized that safety was becoming a deployment problem.

A model that can generate human-like language at scale can also generate harmful instructions, produce toxic content, spread misinformation, or be manipulated into unsafe behavior.

The InstructGPT paper treats these risks very seriously and frames alignment as a necessary part of deploying large language models responsibly.

One of the biggest changes introduced through RLHF was safer refusal behavior.

Earlier GPT systems often attempted to answer almost anything. As a result, they often responded to unsafe prompts rather than recognizing when a refusal was appropriate.

InstructGPT begins changing that behavior.

Through human feedback and preference optimization, the model learns that some requests shouldn't be answered directly. Human labelers consistently prefer safer responses, refusals for harmful instructions, and outputs that avoid dangerous or toxic behavior.

This leads to systems that are better at refusing unsafe requests, avoiding toxic generations, and behaving more cautiously during interaction.

The paper also evaluates toxicity reduction using safety-related benchmarks and finds that aligned models generally produce fewer harmful outputs than earlier GPT systems.

Another important issue is harmful content filtering. Large language models absorb patterns from massive internet datasets, which inevitably contain biased language, misinformation, unsafe instructions, and toxic behavior.

Without alignment, models may reproduce these patterns surprisingly easily.

RLHF acts as a corrective layer on top of pretraining. Instead of only imitating internet text, the model is further optimized toward responses humans judge to be safer and more appropriate.

Of course, the paper is also realistic about limitations.

The authors acknowledge that alignment is incomplete and that unsafe outputs can still occur. Models may still be vulnerable to adversarial prompting or attempts to bypass safety behavior (what later became widely known as jailbreaks).

This is an important nuance: alignment reduces risk, but it doesn't eliminate it.

And historically, this realization became incredibly important for the future of large-scale AI deployment.

In many ways, the InstructGPT paper marks the beginning of modern AI safety engineering inside flagship language models.

InstructGPT introduced large-scale behavior alignment. Then GPT-4 expanded this even further with red teaming, adversarial testing, deployment monitoring, and much larger safety evaluation pipelines.

So this paper becomes a direct bridge between early generative language models and the much more safety-focused AI systems that followed in the GPT-4 era.

Limitations

One of the strongest aspects of the InstructGPT paper is that it doesn't present alignment as a solved problem.

Even though the results are impressive, the authors are careful and surprisingly honest about the system’s remaining weaknesses and risks.

This balance is important because the paper isn't arguing that RLHF creates perfect AI systems. The authors consistently frame alignment as a work in progress rather than a finished solution.

One major limitation is that the models still hallucinate.

The paper acknowledges that hallucinations remain a significant challenge despite alignment improvements.

RLHF improves truthfulness and instruction adherence, but it doesn't fundamentally solve the probabilistic nature of language models. The system still predicts likely text patterns rather than verifying objective truth.

Another important issue is reward hacking.

Because the model is optimized against a learned reward signal, it can sometimes discover shortcuts that maximize reward without genuinely improving reasoning or understanding. In other words, the model may learn behaviors that look aligned to evaluators while still hiding deeper problems underneath.

This is a common challenge in reinforcement learning systems more broadly.

The paper also hints at a problem that later became widely discussed in ChatGPT-era systems: over-refusal and sycophancy.

Sometimes aligned models become too cautious and refuse harmless requests unnecessarily. In other cases, models may become overly agreeable, telling users what they appear to want to hear instead of providing more balanced or truthful responses.

This creates a difficult tension between safety, helpfulness, and honesty.

Another major limitation is bias.

Since these systems are trained on massive internet datasets and further shaped through human labeling, they inevitably inherit biases from both sources. The paper explicitly acknowledges that alignment doesn't remove all harmful or biased behavior.

And perhaps most importantly, the paper emphasizes that RLHF aligns models to labeler preferences not universal human values. This is a very important nuance.

The system learns from the judgments of specific human annotators operating within specific cultural and organizational contexts. That means alignment itself is subjective and imperfect.

There is no single universally agreed definition of helpfulness, fairness, safety, or acceptable behavior.

The paper discusses these concerns carefully and recognizes that human feedback introduces its own limitations and assumptions.

The alignment itself is also fragile. Even aligned systems can sometimes be manipulated through adversarial prompting or jailbreak-style attacks that bypass safety behavior. This later became one of the defining challenges of ChatGPT and GPT-4 deployment.

And finally, there's the practical issue of scale.

RLHF requires large amounts of human labeling, ranking, evaluation, and monitoring. Building these alignment pipelines is expensive, time-consuming, and operationally complex. Unlike raw pretraining data scraped automatically from the internet, human feedback doesn't scale nearly as easily.

In many ways, the paper reveals an important truth about modern AI systems: making models intelligent is difficult. But making them reliably aligned with humans may be even harder.

Historical Importance

Looking back now, it's difficult to overstate how important the InstructGPT paper became for the entire AI industry.

Earlier GPT papers focused mostly on one central question: How do we make language models more capable?

That era was largely driven by larger datasets, larger parameter counts, scaling laws, and benchmark performance.

The models became increasingly impressive at generating text, solving tasks, and demonstrating emergent abilities. But they still behaved primarily like prediction engines trained to continue internet text.

InstructGPT changed the focus completely. For the first time, large-scale AI development began shifting from model-centric AI to interaction-centric AI.

This was a major philosophical transition: the industry realized that users didn't only care about raw intelligence, benchmark scores, or parameter counts.

They cared about usability, conversational quality, safety, trust, and whether the system could actually help them effectively.

This is why ChatGPT felt so different to the public. The underlying language model capabilities were important, but the real breakthrough came from how those capabilities were shaped into a usable human experience.

The interface became conversational. The system became more cooperative. The AI became more aligned with user intent.

That shift fundamentally changed public perception of artificial intelligence.

Before ChatGPT, most people saw AI as research software, technical demos, or specialized tools for experts.

After ChatGPT, millions of people started interacting with AI systems conversationally on a daily basis.

And that changed everything.

Earlier GPT papers focused mainly on discovering what scaling could achieve. InstructGPT introduced a different challenge: How do we safely deploy these systems in the real world?

That shift helped create entirely new areas of research and engineering, including RLHF pipelines, safety tuning, refusal behavior, red teaming, adversarial testing, policy frameworks, and large-scale human-feedback infrastructure.

In many ways, the ChatGPT era began the moment researchers realized that building powerful models was only part of the problem.

The harder challenge was making those systems reliable enough for human interaction at global scale.

It also helps explain why later systems placed much greater emphasis on safety, alignment, deployment practices, and real-world reliability.

The industry was no longer building language models only for research papers. It was building AI systems intended to operate in the real world. And the InstructGPT paper became one of the clearest turning points in that transformation.

Discussion: The Real Shift

The transition from GPT-3 to ChatGPT represents something much deeper than a simple improvement in model performance.

It changed the central question driving the entire AI industry.

During the GPT-3 era, the big question was, “Can language models learn tasks directly from prompts?”

That was the breakthrough introduced by GPT-3.

Research attention shifted toward scaling and emergent capabilities.

But the ChatGPT era introduced a completely different challenge: the question was no longer simply “Can the model perform the task?” Instead, it became, “Can humans actually trust and use these systems every day?”

That shift changed everything.

Once millions of people began interacting with AI systems directly, raw intelligence alone was no longer sufficient. Users needed systems that were understandable, reliable, safe, conversational, and aligned with human expectations.

This is exactly why the InstructGPT paper became so historically important. It introduced the idea that large language models should not only optimize for capability, but also for human interaction quality.

In many ways, the industry moved from “How smart is the model?” to “How usable is the model?”

And that transition fundamentally changed AI development.

After ChatGPT, success was no longer measured only by benchmark scores, parameter counts, or scaling curves.

It was increasingly measured by alignment, conversational quality, safety, and real-world usability.

This also explains why alignment research suddenly became central to modern AI systems.

GPT-3 showed that models could learn from prompts. ChatGPT showed that humans needed models that could cooperate.

That was the real shift.

And it may ultimately become one of the most important turning points in the history of artificial intelligence.

Connection to GPT-4

One of the most important things to understand about GPT-4 is that it didn't appear out of nowhere.

It was built on top of the alignment ideas introduced by InstructGPT and refined through the large-scale deployment experience of ChatGPT.

GPT-4 is often discussed in terms of its reasoning, multimodal abilities, and benchmark performance.

But beneath all of those improvements is something equally important: the alignment pipeline.

Without the work introduced in the InstructGPT paper, GPT-4 would likely feel far less usable as a real-world assistant.

That distinction matters enormously.

Many of GPT-4's alignment techniques can be traced back to ideas introduced by InstructGPT, including RLHF, instruction tuning, conversational alignment, safer refusal behavior, and human preference optimization.

ChatGPT then became the large-scale real-world testing ground for these ideas.

Millions of user interactions exposed weaknesses ranging from hallucinations and jailbreak attempts to broader safety and usability issues.

Those deployment lessons became incredibly valuable.

By the time GPT-4 arrived, OpenAI was no longer simply training a larger language model. It was building a large-scale aligned conversational system shaped by RLHF pipelines, human feedback, safety engineering, adversarial testing, and real-world user interaction.

This is why GPT-4 feels fundamentally different from earlier GPT models.

In many ways, GPT-4 represents the convergence of two major ideas: scaling capability and scaling alignment.

• GPT-3 proved that language models could learn tasks from prompts.

• InstructGPT proved that models could be shaped through human feedback.

• ChatGPT proved that aligned conversational AI could work at global scale.

• GPT-4 combined all of those ideas into a much more capable multimodal system.

That historical progression is important because it shows that modern AI systems aren't built through scaling alone. They're built through the combination of intelligence, alignment, interaction design, and deployment experience.

And the InstructGPT paper became one of the key foundations that made GPT-4 possible.

GPT-3 vs InstructGPT vs ChatGPT vs GPT-4: Key Differences

By this point, we've discussed GPT-3, InstructGPT, ChatGPT, and GPT-4 individually. But it can be helpful to see them side by side.

Although these systems are closely related, each one introduced a different shift in the evolution of modern AI.

GPT-3 focused on capability through scale, InstructGPT focused on alignment through human feedback, ChatGPT focused on conversational usability, and GPT-4 combined these ideas with stronger reasoning and multimodal capabilities.

The table below summarizes the main differences between them and shows how each system built on the progress of the previous generation.

From GPT-1 to GPT-4: A Timeline of Modern AI Systems and Alignment Evolution

Before we wrap up, it's worth stepping back and looking at the bigger picture.

The InstructGPT paper didn't emerge in isolation. It was part of a much larger evolution that transformed GPT models from research-focused language models into the conversational AI systems we use today.

Each generation introduced a new idea that pushed the field forward.

GPT-1 introduced large-scale pretraining, GPT-2 demonstrated zero-shot capabilities, GPT-3 popularized prompting and in-context learning, and InstructGPT introduced alignment through human feedback. ChatGPT then brought these ideas to millions of users through a conversational interface, while GPT-4 combined alignment with stronger reasoning and multimodal capabilities.

The timeline below summarizes the key transitions that shaped the modern AI era.

Final Insight

When people look back at the history of modern AI, they often focus on the moments when models became larger, more powerful, or more capable. But the story of the GPT series is not just a story about scale. It is also a story about learning how to make that intelligence useful.

GPT-1 showed that language models could learn surprisingly rich representations from large amounts of text before being adapted to specific tasks.

GPT-2 expanded that idea and revealed that scale itself could unlock new behaviors.

GPT-3 pushed the field into entirely new territory, demonstrating that a single model could perform a wide variety of tasks simply by responding to prompts and examples.

For a moment, it seemed as though scaling might be the answer to everything.

Then InstructGPT arrived and exposed a different challenge.

The problem was no longer whether a model could generate text, answer questions, or complete tasks. Models were already becoming remarkably capable.

The real question was whether people could actually rely on them. Could they follow instructions consistently? Could they respond in ways users found helpful? Could they become something more than sophisticated prediction engines?

That was the breakthrough at the heart of InstructGPT.

Rather than focusing solely on making models smarter, the paper focused on making them behave better.

Human feedback became part of the training process itself.

Alignment moved from a research concern to a core design principle. For the first time, improving the relationship between humans and AI became just as important as improving the model's raw capabilities.

The impact of that shift extended far beyond a single paper.

It laid the groundwork for ChatGPT, which introduced millions of people to conversational AI. Suddenly, interacting with advanced language models no longer required APIs, research expertise, or carefully engineered prompts. People could simply ask questions, seek advice, explore ideas, or learn something new through natural conversation.

That change transformed AI from a research breakthrough into a widely used product.

GPT-4 would later build on this foundation, combining stronger reasoning and broader capabilities with the alignment techniques that began with InstructGPT. But by then, the industry had already learned an important lesson: capability alone was not enough. Intelligence had to be usable.

In hindsight, the lasting significance of the InstructGPT paper is not that it introduced a new training pipeline. It is that it helped redefine the goal of modern AI.

The challenge was no longer just building systems that could generate language.

It was building systems that people could work with, learn from, and trust.

And that may ultimately be the transition that defined this era of artificial intelligence.

Resources:

• Pytorch Projects for GPT series

• Training Language Models to Follow Instructions with Human Feedback

• Language Models are Few-Shot Learners

• Learning to Summarize from Human Feedback

• Fine-Tuning Language Models from Human Preferences

• Deep Reinforcement Learning from Human Preferences

• Learning to Summarize with Human Feedback

• Aligning AI With Shared Human Values

• Asking for Help on Recursive Decomposition

• WebGPT: Browser-assisted Question-Answering with Human Feedback

• Constitutional AI: Harmlessness from AI Feedback

• TruthfulQA: Measuring How Models Mimic Human Falsehoods

• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

• The Power of Scale for Parameter-Efficient Prompt Tuning

• Finetuned Language Models Are Zero-Shot Learners

• Multitask Prompted Training Enables Zero-Shot Task Generalization

Contact Me

• Github

• X

• Linkedin

The pipeline included:

• proper subject-grouped train/validation splits

• robust preprocessing

• carefully decoded segmentation masks

• sensible loss functions

• consistent evaluation

And the model still struggled to learn.

The interesting part isn't that the model underperformed. What mattered was the diagnosis: a series of simple checks that traced the problem back to the dataset, not the model.

Those checks are useful far beyond medical imaging. They apply to almost any machine learning project.

If you're new to ML, this is a lesson worth carrying into every project: understand your data before you tune your model.

I set out to build a medical image segmentation tutorial. I ended up learning a more valuable lesson: no amount of careful engineering can rescue a model from a dataset that can't support the task.

By the end of this article, you'll understand:

• How to evaluate whether a dataset can actually support your task

• Why "the model isn't learning" is often a data problem

• How to rule out engineering bugs before blaming the data

• Practical diagnostics you can run in minutes

• Why synthetic training data often struggles in real-world deployment

• When to stop tuning and walk away from a dataset

This is not a beginner introduction to deep learning – it assumes familiarity with concepts like UNet architectures and training loops. But the data-quality lessons apply broadly to many ML projects.

What We'll Cover:

• The Dataset

• Step 1: Rule Out the Pipeline Before Blaming the Data

  • Subject-grouped splits

  • Decoding masks correctly

  • Loss design and class weighting

• Step 2: The Model Still Struggled

• Step 3: Interrogating the Dataset

  • Diagnostic 1: What Does the Dataset Actually Contain?

  • Diagnostic 2: Do Synthetic and Real Images Look Similar?

  • Diagnostic 3: Can the gap be fixed by adding real data?

• Step 4: Knowing When to Stop

• A Practical Dataset Evaluation Checklist

• What I Would Try Next

• The Bigger Lesson

The Dataset

I used the US Simulation & Segmentation dataset, a public collection of abdominal ultrasound images with organ segmentation labels from Kaggle.

It contains:

• 926 synthetic ultrasound images — generated by a ray-casting simulator from CT scans, with full organ annotations

• 617 real ultrasound images — from an actual ultrasound scanner

• Labels for 8 organs — liver, kidney, gallbladder, pancreas, spleen, bones, vessels, and adrenals

At first glance, the dataset looked ideal:

• thousands of images

• multiple organ classes

• both synthetic and real ultrasound data

Whether it actually supported the task was a different question.

Step 1: Rule Out the Pipeline Before Blaming the Data

Ground rule: you should always rule out the pipeline before blaming the data. A model failing on buggy code looks exactly like a model failing on bad data. The engineering needs to be trustworthy.

Subject-Grouped Splits

A common mistake in medical imaging is randomly splitting images into train and test sets.

That approach is problematic because many frames come from the same patient. Those frames share anatomy, scanner settings, and noise patterns.

If frames from the same patient appear in both the train and test sets, the model can partially memorize patient-specific patterns. Test scores look artificially good, even though the model may fail on truly unseen patients.

This is called subject leakage.

The fix is to split by patient instead of by image:

from sklearn.model_selection import GroupShuffleSplit

def assign_splits(manifest, val_fraction=0.15, seed=42):
    train_data = manifest[manifest["orig_split"] == "train"]
    groups = train_data["subject_id"].values

    gss = GroupShuffleSplit(n_splits=1, test_size=val_fraction, random_state=seed)
    train_idx, val_idx = next(gss.split(X=train_data, y=None, groups=groups))

    train_subjects = set(train_data.iloc[train_idx]["subject_id"].unique())
    val_subjects = set(train_data.iloc[val_idx]["subject_id"].unique())

    # Crash loudly if leakage ever sneaks in
    assert train_subjects.isdisjoint(val_subjects), "Subject leak detected!"
    return train_subjects, val_subjects

That assertion matters. If the split logic ever breaks, the pipeline fails loudly instead of silently producing misleading metrics.

Decoding Masks Correctly

The dataset stores labels as color-coded masks. Each organ corresponds to a different RGB color.

Training requires converting those colors into integer class labels.

A naïve implementation uses exact color matching, but resizing operations can slightly alter colors at mask boundaries.

A more robust approach maps each pixel to its nearest palette color:

import numpy as np

PALETTE = np.array([
    [0, 0, 0],
    [100, 0, 100],
    [255, 255, 255],
    [0, 255, 0],
    [255, 255, 0],
    [0, 0, 255],
    [255, 0, 0],
    [255, 0, 255],
    [0, 255, 255],
], dtype=np.int32)

def decode_mask(mask_rgb):
    h, w = mask_rgb.shape[:2]
    flat = mask_rgb.reshape(-1, 3).astype(np.int32)
    d2 = (
        (flat[:, None, :] - PALETTE[None, :, :]) ** 2
    ).sum(-1)
    classes = d2.argmin(axis=1).astype(np.uint8)
    return classes.reshape(h, w)

Before training, it’s worth visually checking a few decoded masks against the original images. This catches issues like incorrect palettes, RGB/BGR channel swaps, or resizing artifacts that silently corrupt labels.

These bugs rarely throw errors. Instead, the model simply learns poorly. And “trained on wrong labels” looks exactly like “the model can’t learn the data.”

Verifying masks early removes that uncertainty.

Loss Design and Class Weighting

For training, I usd standard MONAI segmentation losses. The goal wasn’t to aggressively maximize performance, but to establish a stable and trustworthy baseline.

The training curves below show that the model optimized normally: the loss decreased consistently, and the validation dice stabilized rather than diverging. This helped rule out optimization instability as the primary cause of poor final performance.

Three choices were deliberate:

• Dice + Cross-Entropy combined: Cross-entropy keeps learning stable early on – Dice directly rewards good region overlap. Together they balance each other.

• include_background=False for binary segmentation: In a single-organ task, background can be 85–90% of the pixels. Counting it in the loss drowns out the signal for the organ you actually care about, so it's better left out.

• Class weighting for multi-class segmentation: With organs of very different sizes, an unweighted loss lets the model ignore the small, rare ones and still score well. Weighting rare-class mistakes more heavily pushes back against that.

Step 2: The Model Still Struggled

The first experiment focused on liver segmentation — the simplest single-organ task in the dataset.

Dice scores range from 0 (no overlap) to 1 (perfect overlap).

Qualitatively, the predictions often captured rough liver regions but failed at boundaries and consistency across real scans.

Especially important:

• the model struggled even on synthetic in-domain data

• performance dropped further on real ultrasound images

At this point, two explanations were possible:

1. the model or pipeline was flawed

2. the dataset itself was limiting performance

Because the engineering had been carefully validated, the second possibility became worth investigating seriously.

That's where the real lesson began.

Step 3: Interrogating the Dataset

Rather than endlessly tuning the model, the productive move is to turn the diagnostic lens on the dataset.

Three simple checks revealed the real problem. None required retraining or expensive experiments.

Diagnostic 1: What Does the Dataset Actually Contain?

The first step was simply plotting the dataset composition.

• 926 labeled synthetic images (the bulk of training data)

• Only 60 labeled real images — less than 4% of the dataset

• 557 unlabeled real images — real data exists, but without labels it can't be used for supervised training

This immediately changed the interpretation of the dataset.

Although the dataset contains many real ultrasound scans, almost all labeled training data is synthetic.

The model is effectively trained on synthetic ultrasound and expected to generalize to real ultrasound.

That's a difficult transfer problem from the start.

The limitation is simple: the real images mostly don't have labels, so supervised training has very little real-world data to learn from.

Lesson: Before training anything, chart the dataset composition. A headline image count can be misleading. "1,500 images" sounds large until you discover that only a tiny fraction are labeled examples from the target domain.

Diagnostic 2: Do Synthetic and Real Images Look Similar?

The next question was whether the synthetic and real ultrasound images actually followed similar visual distributions.

Plotting intensity histograms showed a clear mismatch.

• synthetic images clustered heavily near darker intensities

• real ultrasound images had broader mid-range intensity distributions

The synthetic simulator captured anatomical geometry reasonably well, but it didn't reproduce the texture and noise characteristics of real ultrasound:

• speckle patterns

• intensity falloff

• scanner-specific artifacts

This is the classic synthetic-to-real domain gap.

The model learned features tuned to synthetic images and then encountered a substantially different distribution during evaluation. Poor transfer performance became expected rather than surprising.

Lesson: Whenever training and deployment happen on different domains — synthetic → real, scanner A → scanner B, hospital A → hospital B — measure the distribution shift directly. Simple histogram comparisons can reveal major problems in minutes.

Diagnostic 3: Can the gap be fixed by adding real data?

The obvious next idea was: why not include some real labeled data during training?

But before implementing that approach, it's worth checking how many distinct patients actually had labels.

Labeled real images: 60
Distinct subjects (labeled real): 4

Frames per subject:
  subject h: 26
  subject a: 16
  subject g: 10
  subject b: 8

Only four patients.

That result fundamentally changed the situation.

Proper medical imaging evaluation requires subject-grouped train/test splits. But with only four patients, any evaluation becomes statistically unstable.

Training on two or three patients and testing on one or two patients would produce highly unreliable metrics that depend heavily on which patient happened to be held out.

At that point, the dataset simply couldn't support trustworthy real-world evaluation.

Lesson: In medical imaging, count subjects, not images. The true size of a dataset is bounded by the number of independent patients, not the number of files.

Step 4: Knowing When to Stop

At this point, additional tuning no longer made sense.

The bottleneck was not the architecture, optimizer, or learning rate. The bottleneck was the dataset itself.

The pipeline was still valuable and reusable. But this particular dataset couldn't reliably support the intended segmentation task.

That distinction matters: sometimes a problem is difficult but solvable, and sometimes the data simply can't support the conclusion you want to draw.

Learning to recognize the difference is an important ML skill.

A Practical Dataset Evaluation Checklist

Before committing weeks to model development, these checks are worth running on any dataset:

1. Chart the dataset composition — labeled vs unlabeled, class distribution, domain distribution

2. Count subjects, not images — independent patients matter more than frame count

3. Check class balance — rare classes are often ignored without weighting or sampling strategies

4. Compare train and deployment distributions — especially for cross-domain problems

5. Verify labels visually — catch preprocessing or annotation errors early

6. Look for published baselines — low published performance may indicate dataset limitations

These checks take minutes and can save weeks of unnecessary tuning.

What I Would Try Next

Improving results would likely require better data rather than a larger model. The next steps I'd prioritize:

• collecting more labeled real ultrasound scans, from more distinct patients

• improving annotation consistency

• semi-supervised learning to make use of the unlabeled real images

• domain adaptation between synthetic and real ultrasound

All of these target the actual bottleneck: data quality and data diversity.

The Bigger Lesson

In machine learning, it's easy to focus most of our attention on architectures, hyperparameters, optimization tricks, and newer models.

But the dataset quietly defines the ceiling.

A sophisticated model on weak data often disappoints, while a simpler model on strong data performs surprisingly well.

That was the real lesson from this project.

The most valuable skill wasn't building the pipeline. It was diagnosing why the model couldn't succeed and being willing to trust what the data was saying.

The workflow — checking dataset composition, counting subjects, comparing distributions, ruling out engineering bugs, and deciding when to stop — transfers to almost any ML project.

In many projects, better judgment about the data matters more than a better model.

The pipeline code and diagnostic notebooks are available at the MONAI Ultrasound Working Group repository. Questions, corrections, and improvements are always welcome.

That idea eventually led to prompt engineering, AI assistants, and the first wave of large language model applications.

But GPT-4 felt different.

GPT-3 still felt like a research breakthrough: powerful, experimental, and sometimes unpredictable. GPT-4, on the other hand, felt like the beginning of a real AI platform. The focus was no longer just on scaling language models to achieve better benchmarks. Instead, the conversation shifted toward reliability, multimodal understanding, alignment, safety, and real-world deployment.

This change is visible throughout the GPT-4 Technical Report released by OpenAI.

Unlike the earlier GPT papers, OpenAI didn't publish a traditional research paper with detailed architecture diagrams, parameter counts, datasets, or training configurations. Instead, they released a more limited technical report focused primarily on capabilities, evaluations, safety work, and deployment considerations.

That decision itself reflects how much the field had changed.

By the time GPT-4 arrived, large language models were no longer just research projects used inside labs. They had become globally deployed systems used by millions of people through products like ChatGPT. Questions about misuse, hallucinations, bias, cybersecurity risks, and alignment were now just as important as raw model performance.

GPT-4 also introduced another major shift: multimodality.

Previous GPT models worked only with text. GPT-4 expanded this idea by accepting both images and text as input, allowing the model to analyze screenshots, diagrams, documents, visual jokes, and other mixed forms of information. This pushed large language models closer to more general-purpose AI systems rather than narrow text generators.

Historically, the progression becomes surprisingly clear:

• GPT-1 introduced pretraining and transfer learning

• GPT-2 introduced zero-shot multitask learning

• GPT-3 introduced few-shot prompting and in-context learning

• GPT-4 introduced the era of aligned, multimodal AI systems

In many ways, GPT-4 marks the moment when large language models stopped being viewed primarily as research experiments and started becoming foundational computing interfaces for real-world applications.

Paper Overview

In this article, we’ll review the GPT-4 Technical Report published by Open AI in 2023.

Many important technical details were intentionally omitted from this report, including:

• parameter count

• exact architecture

• training compute

• dataset composition

• hardware configuration

According to OpenAI, these limitations were introduced partly because of the competitive landscape and the growing safety implications surrounding large-scale AI systems.

That difference is historically important.

The GPT-1, GPT-2, and GPT-3 papers openly discussed architecture scaling, datasets, and training methodology in significant detail. GPT-4 marks a noticeable shift toward more restricted disclosure as language models became commercially valuable and widely deployed.

You can read the original report here:

GPT-4 Technical Report

And here’s a quick infographic of what we’ll cover throughout this review:

Table of Content:

• Executive Summary

• Goals of the Report

• Core Idea

• Predictable Scaling

• Model Architecture

• Multimodal Learning

• Fine-Tuning vs Zero-Shot vs Few-Shot vs Aligned Multimodal Learning

• RLHF and Alignment

• Benchmarks and Experiments

• Coding and Reasoning Ability

• Multilingual Capabilities

• Emergent Behavior

• Limitations

• Safety and Risks

• Discussion

• Conclusion

• Final Insight

• GPT-1 vs GPT-2 vs GPT-3 vs GPT-4: Key Differences

• PyTorch Implementations of the GPT Architecture Evolution

• Resources:

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with some of the core ideas behind modern language models.

Reading the earlier reviews in this series will be especially useful:

• AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)

• AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)

• AI Paper Review: Language Models are Few-Shot Learners (GPT-3)

GPT-4 builds directly on many of the concepts introduced in those papers, especially large-scale pretraining, zero-shot and few-shot learning, and in-context prompting.

It also helps to have a general understanding of:

• Transformer architectures and self-attention

• The evolution from GPT-1 → GPT-3

• Few-shot learning and prompting

• Basic prompt engineering concepts

• Reinforcement Learning from Human Feedback (RLHF)

• Scaling laws and why larger models often develop new capabilities

You don't need deep mathematical knowledge to follow this article, though.

As with the previous reviews, I’ll focus more on explaining the ideas intuitively and practically rather than diving too deeply into heavy equations or dense academic terminology.

Executive Summary

GPT-4 is not simply a larger version of GPT-3.

That may sound obvious today, but at the time, many people initially assumed GPT-4 was just another scaling step in the same direction. But the technical report shows something more important: GPT-4 represents a shift from experimental language models toward deployable general-purpose AI systems.

According to the report, GPT-4 introduces several major advances at once.

First, as mentioned above, the model becomes multimodal. Unlike previous GPT systems that only worked with text, GPT-4 can process both images and text as input while still generating text outputs. This allows the model to analyze screenshots, diagrams, documents, photographs, visual jokes, and mixed media prompts.

Second, GPT-4 demonstrates significantly stronger reasoning and benchmark performance across a wide range of professional and academic evaluations. The report shows GPT-4 achieving near human-level results on exams including the Uniform Bar Exam, LSAT, GRE, SAT, AP tests, coding benchmarks, and advanced reasoning tasks.

The report also places heavy emphasis on alignment and factuality improvements.

Earlier GPT systems often produced unsafe, misleading, or overly confident outputs. GPT-4 still has these problems, but OpenAI invested heavily in reinforcement learning from human feedback (RLHF), adversarial testing, refusal behavior, and safety evaluation pipelines to reduce harmful behavior and improve adherence to user intent.

Another major theme throughout the report is predictable scaling.

According to the authors, OpenAI developed infrastructure and optimization methods that allowed them to accurately predict GPT-4’s final performance using much smaller training runs.

That detail matters more than it might seem.

GPT-3 demonstrated that scaling works. GPT-4 demonstrates that scaling large language models was becoming an engineering discipline with increasingly predictable behavior.

The broader implication is what makes this report historically important.

GPT-4 transforms large language models from research demonstrations into deployable AI assistants capable of reasoning across many domains, interacting through natural language, following instructions more reliably, and operating at global scale through systems like ChatGPT.

In many ways, this report marks the beginning of the modern AI deployment era.

Goals of the Report

The GPT-4 Technical Report is not only about showing a more capable language model. In many ways, the report is about demonstrating that large AI systems can be developed more reliably, more safely, and more predictably than before.

One of the main goals behind GPT-4 was improving reasoning and reliability across a broad range of tasks, which we discussed above.

Another major objective was improving alignment with user intent – investing in RLHF, safety fine-tuning, refusal training, and adversarial testing to make the model more helpful and better aligned with intended behavior.

The report also marks a significant shift beyond text-only AI systems, as GPT-4 introduces multimodal capabilities. This expands the system from being purely a language generator into something closer to a general-purpose reasoning interface capable of interpreting visual and textual information together.

Safety is another central theme throughout the report.

OpenAI repeatedly emphasizes efforts to reduce harmful outputs, improve refusal behavior, mitigate misuse risks, and build safer deployment systems around the model. The report discusses red teaming, domain expert testing, policy enforcement, and model-assisted safety pipelines designed to reduce dangerous behavior during real-world usage.

But one of the most historically important goals may actually be predictability.

According to the authors, GPT-4 was developed using infrastructure and optimization methods designed to scale in highly predictable ways. OpenAI claims they could estimate aspects of GPT-4’s final performance using models trained with thousands of times less compute.

That idea may sound technical, but it represents a major shift in how frontier AI systems were being built.

Earlier generations of language models often involved substantial uncertainty during scaling. GPT-4 suggests that large-scale AI development was becoming more systematic and engineering-driven rather than purely experimental.

In practice, the report reflects a broader transition happening across the AI industry, from research prototypes to deployable infrastructure systems designed for real-world use at massive scale.

Core Idea

One of the most surprising things about GPT-4 is that, underneath all the hype and new capabilities, the core learning objective is still fundamentally very simple.

Like GPT-1, GPT-2, and GPT-3, GPT-4 is still trained primarily as a next-token prediction model. In other words, the system learns by repeatedly predicting the next piece of text in a sequence.

The architecture also remains Transformer-based and autoregressive.

That means GPT-4 generates outputs one token at a time while using self-attention to understand relationships between words, sentences, images, and context inside the input sequence.

At a high level, the underlying principle hasn't changed very much since GPT-2:

• train on massive amounts of data

• predict the next token

• scale the model aggressively

But GPT-4 pushes this approach much further.

According to the report, the model is substantially larger, more optimized, and trained using infrastructure designed specifically for predictable large-scale behavior.

The biggest conceptual change is that GPT-4 is no longer limited to text-only input.

Another major difference is the importance of post-training alignment.

GPT-3 already demonstrated strong few-shot learning abilities, but GPT-4 places much heavier emphasis on reinforcement learning from human feedback (RLHF), safety tuning, refusal behavior, and instruction following. According to the report, these post-training processes significantly improve factuality, adherence to desired behavior, and response safety.

This leads to one of the most important ideas behind modern AI systems:

Capability doesn't emerge from scale alone.

GPT-4 suggests that powerful AI behavior comes from the combination of:

• large-scale pretraining

• scaling laws

• optimization improvements

• alignment training

• RLHF

• post-training refinement

In practice, GPT-4 feels less like a raw predictive model and more like an interactive assistant because of this additional alignment layer.

That distinction matters historically.

GPT-3 showed that scaling language models could unlock powerful emergent behavior. GPT-4 shows that scaling alone is not enough — the model also needs alignment, safety training, and deployment-focused refinement to become broadly usable in the real world.

Predictable Scaling

One of the most important ideas in the GPT-4 Technical Report is something that many people overlooked when the paper first came out: predictable scaling.

Earlier generations of large language models involved a huge amount of uncertainty.

Researchers could train larger systems and hope performance would improve, but nobody fully knew how far scaling would go or whether massive training runs would behave the way they expected.

GPT-4 changed that. According to the report, OpenAI developed infrastructure and optimization methods that allowed them to accurately predict GPT-4’s final training loss, and even some capabilities, using models trained with thousands of times less compute.

This is far more important than it first sounds. GPT-3 proved that scaling language models works.

GPT-4 suggested that scaling was starting to become predictable engineering rather than trial-and-error experimentation.

That shift introduced several major advantages:

• Better capability forecasting before training massive models

• Reduced risk of wasting millions of dollars on failed training runs

• Safer deployment planning through earlier evaluation of model behavior

• More reliable scaling from small experiments to frontier-scale systems

The report also shows that model loss followed remarkably stable power-law behavior across scales, allowing OpenAI to estimate GPT-4’s final performance long before training finished.

But the paper also makes an important point: not every capability scales smoothly. Some behaviors, especially reasoning-related tasks, can emerge unpredictably or even temporarily worsen before improving again.

Some important limitations of predictable scaling include:

• Some capabilities still emerge unpredictably at larger scales

• Benchmark performance can behave nonlinearly instead of improving smoothly

• Scaling laws may not hold forever as models continue growing

• Even with predictable training curves, reasoning failures and hallucinations can still appear unexpectedly

That tension between predictable scaling and unexpected emergence became one of the defining themes of modern frontier AI research.

Model Architecture

One of the most unusual aspects of the GPT-4 Technical Report is how little OpenAI reveals about the actual model architecture.

As discussed above, in the GPT-1, GPT-2, and GPT-3 papers, OpenAI openly discussed details like parameter counts, dataset sizes, scaling configurations, and training methodology.

As you now know, GPT-4 is very different. The report leaves out several major technical details like the exact parameter count, the precise architecture configuration, the dataset size and composition, the training compute used, and the hardware infrastructure and setup.

The report explicitly states that these omissions were motivated by both the competitive landscape and safety considerations surrounding large-scale AI systems.

That decision became one of the most discussed aspects of the release.

Historically, GPT-4 marks a transition where frontier AI research started becoming more closed and product-oriented. Earlier GPT papers felt like traditional research publications. GPT-4 feels more like a controlled systems report from a company deploying AI at global scale.

Even though many implementation details remain hidden, the report still confirms several important things:

1. GPT-4 is still fundamentally a Transformer-based model trained using autoregressive next-token prediction.

2. Like previous GPT systems, it generates outputs sequentially while using self-attention mechanisms to process context.

3. GPT-4 is multimodal, meaning it can accept both image and text inputs while producing text outputs.

This is one of the biggest architectural shifts in the GPT series because it extends the model beyond pure language understanding into combined visual and textual reasoning.

Another important component is post-training alignment, which we've already discussed a bit. In practice, it means that GPT-4 isn't just a raw pretrained language model anymore. It's a heavily refined system built through multiple stages:

• large-scale pretraining

• optimization and scaling improvements

• multimodal integration

• RLHF alignment

• safety fine-tuning

• deployment-oriented post-training

The secrecy surrounding GPT-4’s architecture is historically important because it reflects a broader change happening in AI.

As language models became commercially valuable and socially impactful, frontier AI research started moving away from full openness toward controlled disclosure, safety-focused deployment, and competitive protection.

Multimodal Learning

One of the most important breakthroughs in GPT-4 is that the model is no longer limited to text alone. GPT-4 can accept both images and text as input while generating text outputs.

That may sound simple today, but at the time, this represented a major shift in how people thought about large language models.

Earlier GPT systems worked purely with language. GPT-4 expands the idea into something much broader: a model capable of reasoning across multiple forms of information at the same time.

In practice, GPT-4 can analyze:

• screenshots

• diagrams

• photographs

• documents

• charts

• visual jokes and memes

• mixed image-and-text prompts

The report demonstrates this capability through several examples, but one became especially memorable: the famous VGA cable meme example.

In the image, a smartphone appears connected to a massive VGA monitor cable adapter – something clearly absurd in real life. GPT-4 correctly explains that the humor comes from the mismatch between outdated VGA hardware and a modern phone charging port.

What made this example important was not just object recognition. The model was interpreting contextual humor from a visual scene.

That distinction matters.

Traditional computer vision systems could often identify objects inside images, but GPT-4 demonstrated something closer to multimodal reasoning: understanding relationships, context, intent, and even jokes across combined visual and textual information.

The report also notes that many prompting techniques developed for language models (including few-shot prompting and chain-of-thought reasoning) continue working effectively in multimodal settings.

This suggests that GPT-4 is not simply attaching an image classifier onto a chatbot. Instead, the model appears to integrate visual and language understanding into a more unified reasoning system.

Historically, this was a major moment for the GPT series.

• GPT-1 focused on language pretraining

• GPT-2 expanded zero-shot capabilities

• GPT-3 introduced in-context learning

• GPT-4 publicly demonstrated practical multimodal AI

And unlike many earlier research demos, GPT-4’s multimodal abilities were not just experimental prototypes hidden inside papers. They became part of real-world products used by millions of people.

That shift made multimodal AI feel practical and deployable rather than purely theoretical.

Fine-Tuning vs Zero-Shot vs Few-Shot vs Aligned Multimodal Learning

One of the clearest ways to understand how GPT models evolved is by comparing how they learn and adapt to tasks.

Earlier NLP systems relied heavily on fine-tuning with labeled datasets, while later GPT models increasingly shifted toward zero-shot prompting, few-shot learning, and eventually aligned multimodal interaction.

The table below summarizes how these approaches differ in flexibility, training requirements, scalability, and real-world usability.

RLHF and Alignment

One of the biggest differences between GPT-4 and earlier GPT models is how much emphasis the report places on alignment and safety.

GPT-3 demonstrated impressive few-shot learning abilities, but it also exposed serious weaknesses. The model could hallucinate facts, generate harmful instructions, confidently produce false information, or fail to follow user intent reliably.

GPT-4 was designed with these problems in mind.

A major part of this improvement comes from Reinforcement Learning from Human Feedback (RLHF).

At a high level, RLHF works by collecting human feedback about model responses and then using that feedback to train the model toward preferred behavior. Instead of learning only from internet text, the system also learns from human judgments about what kinds of answers are helpful, safe, accurate, or appropriate.

According to the report, GPT-4 undergoes extensive post-training alignment designed to improve:

• factuality

• instruction following

• refusal behavior

• harmlessness

• adherence to user intent

This alignment layer is a major reason GPT-4 feels different from raw pretrained language models.

The report repeatedly emphasizes refusal behavior as an important safety capability.

Earlier versions of GPT-4 could sometimes generate dangerous instructions, including harmful chemical synthesis advice or weapon-related content during internal testing. OpenAI used adversarial testing, domain experts, RLHF training, and additional safety pipelines to reduce these behaviors significantly.

The examples shown in the report are especially revealing.

In one case, an earlier GPT-4 version provided detailed responses about creating dangerous materials. Later aligned versions instead refuse the request and redirect the conversation safely.

What makes this important is that GPT-4 is not simply being made “more restrictive.”

The report also discusses the opposite problem: models becoming too cautious. OpenAI specifically worked on reducing unnecessary refusals for harmless requests while still blocking dangerous ones.

In practice, alignment becomes a balancing act between:

• usefulness

• safety

• honesty

• flexibility

• and reliability

The paper also introduces rule-based reward models and model-assisted safety pipelines that help guide GPT-4 toward safer behavior during training.

Historically, this section of the report marks another major transition in AI development.

Earlier GPT papers focused primarily on capabilities and scaling. GPT-4 treats alignment and deployment safety as core engineering problems rather than secondary concerns.

That shift reflects a deeper realization across the industry: once AI systems become powerful enough for real-world deployment at global scale, improving intelligence alone is no longer enough. The systems also need to behave safely, follow human intent reliably, and resist harmful misuse.

Benchmarks and Experiments

One of the most striking parts of the GPT-4 Technical Report is the sheer scale of the evaluation process.

According to the report, OpenAI tested GPT-4 across a wide range of academic exams, professional certifications, reasoning tasks, coding benchmarks, and traditional NLP evaluations.

The goal was not simply to show that GPT-4 could generate fluent text. The evaluations were designed to measure whether the model could reason, solve problems, follow instructions, answer questions, and generalize across many different domains.

The human exam results attracted enormous attention when the report was released.

GPT-4 achieved particularly strong scores on several well-known exams:

• Uniform Bar Exam → around the top 10% of test takers

• LSAT → roughly 88th percentile

• SAT Reading & Writing → around 93rd percentile

• GRE Verbal → around the 99th percentile

• Strong performance across many AP exams

GPT Performance on Academic and Professional Exams

The table below summarizes GPT-4’s performance across a wide range of academic and professional exams, showing how the model compared with GPT-3.5 on tests such as the Uniform Bar Exam, LSAT, GRE, SAT, AP exams, and coding challenges.

Source: GPT-4 Technical Report (OpenAI, 2023), Table 1.

The comparison with GPT-3.5 was especially dramatic in some cases. For example, the report notes that GPT-3.5 scored near the bottom 10% on the simulated bar exam, while GPT-4 reached the top 10%.

These results helped change public perception of large language models.

Earlier systems were often viewed mainly as autocomplete engines or text generators. GPT-4 demonstrated that scaling and alignment could produce systems capable of performing competitively on many tasks originally designed for humans.

The figure below visualizes GPT-4’s percentile rankings across multiple exams, highlighting the significant improvement over GPT-3.5 in areas such as reasoning, language understanding, mathematics, and professional testing.

Source: GPT-4 Technical Report (OpenAI, 2023), Figure 4.

The report also evaluates GPT-4 on a wide collection of standard NLP benchmarks.

Some of the most important include:

• MMLU → broad academic and professional reasoning benchmark

• HellaSwag → commonsense reasoning

• HumanEval → coding and Python synthesis tasks

• GSM8K → grade-school mathematics reasoning

• ARC → science reasoning questions

• WinoGrande → pronoun and commonsense reasoning

Across most of these evaluations, GPT-4 substantially outperforms GPT-3.5 and often surpasses previous state-of-the-art language models. In several cases, it even exceeds systems that relied on benchmark-specific fine-tuning or specialized engineering pipelines.

One especially important benchmark is MMLU (Massive Multitask Language Understanding), which tests knowledge and reasoning across 57 different subjects. GPT-4 achieves remarkably strong performance on this benchmark, including multilingual variants translated into many languages.

The coding evaluations are also historically significant. On HumanEval and LeetCode-style tasks, GPT-4 demonstrates major improvements in code generation and problem solving compared to earlier GPT systems.

This capability eventually became one of the foundations behind modern AI coding assistants.

The table below compares GPT-4 with previous language models and state-of-the-art systems on major AI benchmarks such as MMLU, HellaSwag, ARC, HumanEval, and GSM-8K, demonstrating the model’s strong performance across reasoning, coding, and language understanding tasks.

Source: GPT-4 Technical Report (OpenAI, 2023), Table 2.

What makes these experiments especially important is that GPT-4 performs well across many different categories simultaneously:

• reasoning

• coding

• mathematics

• language understanding

• professional exams

• multilingual tasks

• commonsense reasoning

That breadth is part of what made GPT-4 feel qualitatively different from earlier systems.

Instead of excelling in one narrow benchmark, GPT-4 demonstrated increasingly general behavior across a wide variety of intellectual tasks.

Coding and Reasoning Ability

One of the areas where GPT-4 shows some of its most noticeable improvements over earlier models is coding and structured reasoning.

While GPT-3 was already capable of generating code, GPT-4 pushes these abilities much further. According to the report, the model demonstrates substantial gains on programming benchmarks, mathematical reasoning tasks, and multi-step problem solving.

A key benchmark highlighted in the report is HumanEval, which measures the model’s ability to generate working Python functions from natural language descriptions.

GPT-4 achieves significantly higher performance than GPT-3.5 on this benchmark, showing much stronger code synthesis and problem-solving ability.

The report also includes LeetCode-style evaluations across easy, medium, and hard programming problems.

Although GPT-4 still struggles with many difficult competitive programming tasks, it performs substantially better than GPT-3.5, especially on easier and medium-level coding challenges.

These improvements became extremely important in practice.

Around the release of GPT-4, AI coding assistants started becoming genuinely useful for real software development workflows. Systems built on GPT-4 could help developers:

• generate functions

• explain code

• debug errors

• refactor implementations

• write documentation

• solve algorithmic problems

This was one of the first moments where large language models began functioning as practical engineering tools rather than experimental demos.

The report also highlights the importance of chain-of-thought prompting for reasoning tasks.

Instead of forcing the model to produce an immediate answer, chain-of-thought prompting encourages GPT-4 to reason step by step before reaching a conclusion.

For example, on benchmarks like GSM8K (a dataset of grade-school mathematics problems), GPT-4 performs much better when allowed to generate intermediate reasoning steps.

This became another major shift in how people interacted with large language models. Earlier systems were often treated like direct answer generators. GPT-4 demonstrated that prompting the model to “think through” a problem could significantly improve performance on reasoning-heavy tasks.

Compared to GPT-3.5, GPT-4 consistently shows stronger reasoning across many domains:

• coding

• mathematics

• structured problem solving

• commonsense reasoning

• academic evaluations

Of course, the model is still far from perfect.

The report repeatedly notes that GPT-4 can still hallucinate, make logical mistakes, fail at complex reasoning chains, or confidently produce incorrect solutions.

But historically, this section of the report matters because it helped establish a new category of AI applications: large language models as interactive reasoning and coding assistants.

That idea quickly became one of the defining use cases of modern AI systems.

Multilingual Capabilities

One of the more underrated aspects of the GPT-4 Technical Report is how strongly the model performs across multiple languages.

Earlier language models were often heavily English-centric. Even when multilingual support existed, performance in lower-resource languages usually dropped significantly compared to English benchmarks.

GPT-4 shows noticeable progress in this area.

To evaluate multilingual reasoning ability, OpenAI translated the MMLU benchmark – a broad academic and professional reasoning benchmark covering 57 subjects – into many different languages using machine translation systems.

According to the report, GPT-4 performs extremely well across most tested languages and even surpasses the English-language performance of earlier models in many cases.

What makes this especially important is that the improvements are not limited to high-resource languages like French, German, or Spanish.

The report specifically highlights strong performance gains in lower-resource languages such as:

• Latvian

• Welsh

• Swahili

• Bengali

• Nepali

• Marathi

• Telugu

This suggests something important about large-scale language modeling: as models scale and training data becomes more diverse, the learned capabilities start generalizing beyond English in a much more robust way.

In other words, the scaling effects observed in GPT-3 were not purely English-language phenomena.

GPT-4 demonstrates that many reasoning and language understanding capabilities can transfer across languages, even when available training data is far more limited.

This is historically significant because it moves large language models closer to becoming globally useful systems rather than tools optimized mainly for English-speaking users.

The multilingual results also reinforce another major theme throughout the report: GPT-4 is not narrowly specialized for a single domain or benchmark. Instead, it behaves increasingly like a general-purpose reasoning system capable of adapting across:

• languages

• tasks

• modalities

• domains

• and interaction styles

Of course, multilingual performance is still uneven.

The report doesn't claim perfect fluency or equal reasoning quality across all languages. Lower-resource languages still present major challenges, and evaluation itself remains difficult in many multilingual settings.

But compared to earlier GPT systems, GPT-4 demonstrates a substantial step forward in multilingual generalization. And that became an important milestone for globally deployed AI systems.

Emergent Behavior

One of the most fascinating ideas surrounding GPT-4 is the concept of emergent behavior.

In the context of large language models, emergence refers to abilities that appear unexpectedly as models become larger and more capable. Instead of improving smoothly in every area, some skills seem to “switch on” once the model reaches a certain scale.

GPT-3 already hinted at this phenomenon through few-shot learning and in-context adaptation. GPT-4 continues that trend much more strongly.

According to the report, many capabilities improve nonlinearly as scale increases.

In simpler terms, doubling the size or compute of a model doesn't just make it slightly better at the same tasks. Sometimes, entirely new behaviors emerge that were weak or mostly absent in smaller systems.

This becomes especially visible in reasoning tasks.

GPT-4 demonstrates major improvements over GPT-3.5 in coding, mathematical reasoning, academic evaluations, instruction following, and structured problem solving.

The report also highlights how prompting strategies become more effective at larger scales.

Few-shot prompting (where the model learns from examples inside the prompt) works far more reliably in GPT-4 than in earlier systems. Similarly, chain-of-thought prompting becomes significantly more useful for reasoning-heavy tasks.

Instead of immediately generating an answer, GPT-4 can often improve performance by reasoning step by step through a problem.

What makes this important is that these abilities weren't explicitly programmed into the system. The model was still trained primarily through next-token prediction. Yet at sufficient scale, behaviors like:

• multi-step reasoning

• code synthesis

• contextual adaptation

• multilingual generalization

• instruction following

• and visual-text reasoning

began appearing much more robustly.

The report’s discussion of predictable scaling also connects directly to this idea. OpenAI explains that GPT-4’s capabilities could often be estimated from smaller training runs using scaling laws.

At the same time, some behaviors remain difficult to predict cleanly. The paper even notes cases where certain tasks improve unexpectedly or reverse earlier scaling trends as models become larger.

Historically, GPT-4 reinforces one of the biggest lessons from the GPT series: large language models don't simply become more fluent as they scale. They begin exhibiting qualitatively different behaviors.

That realization fundamentally changed AI research. Instead of treating language models as narrow NLP systems, researchers increasingly started viewing them as general-purpose learning systems whose capabilities could continue emerging with scale, alignment, and better training methods.

Limitations

Despite the impressive benchmark results and multimodal capabilities, the GPT-4 Technical Report is surprisingly direct about the model’s weaknesses.

The paper repeatedly emphasizes that GPT-4 is still not fully reliable.

One of the biggest problems is still hallucination.

Like earlier GPT systems, GPT-4 can confidently generate information that's incorrect, fabricated, or misleading. The model may produce answers that sound highly convincing even when the underlying facts are wrong.

This becomes especially dangerous because GPT-4 is often more fluent and persuasive than previous models. In practice, stronger language generation can sometimes make mistakes harder for users to notice.

The report also discusses reasoning failures.

Although GPT-4 performs much better than GPT-3.5 across many benchmarks, it can still fail at relatively simple logical tasks, make arithmetic mistakes, or break down during longer reasoning chains.

Another important limitation is overconfidence.

GPT-4 doesn't naturally “know when it does not know.” The model can present uncertain or incorrect answers with a high degree of confidence, which creates risks in high-stakes situations like medicine, law, education, or cybersecurity.

The report also notes that GPT-4 has a knowledge cutoff. Most of the model’s training data ends around September 2021, meaning the system lacks reliable awareness of many events that happened afterward.

One particularly interesting section discusses calibration.

According to the report, the pretrained GPT-4 model was actually fairly well calibrated – meaning its confidence often matched the probability of correctness. But post-training alignment and RLHF reduced calibration quality in some cases.

This reveals an important tradeoff: making models more helpful and aligned doesn't automatically make them more truthful or better calibrated.

The paper is also honest about bias and unsafe behavior.

Because GPT-4 learns from large internet-scale datasets, it can still reflect social biases, stereotypes, and problematic patterns present in training data.

OpenAI discusses extensive efforts to reduce harmful outputs, but the report explicitly acknowledges that unsafe behavior is still possible.

One example is jailbreaking: attempts to bypass safety mechanisms using adversarial prompts or clever conversational manipulation. According to the report, GPT-4’s safety systems reduce harmful behavior significantly, but determined users can still sometimes elicit dangerous or policy-violating outputs.

The paper also emphasizes that GPT-4 should not be blindly trusted in high-risk environments without additional safeguards, human oversight, or verification systems.

That honesty is one reason the report remains important: instead of presenting GPT-4 as a solved form of intelligence, OpenAI frames it as a powerful but imperfect system whose growing capabilities also create growing risks.

Historically, this reflects a major shift in AI research culture.

Earlier papers focused mostly on increasing performance. GPT-4 places equal emphasis on capability and failure modes, because once models become widely deployed, understanding limitations becomes just as important as demonstrating strengths.

Safety and Risks

One of the clearest signs that the AI field had changed by the time GPT-4 was released is how much of the report is dedicated to safety, risk analysis, and deployment concerns.

Earlier GPT papers focused primarily on capability improvements, scaling behavior, and benchmark performance. The GPT-4 Technical Report still discusses those topics, but safety becomes a central engineering theme rather than a secondary discussion.

According to the report, OpenAI conducted extensive red teaming and adversarial testing before deployment.

Red teaming involves intentionally trying to break the system, bypass safeguards, trigger unsafe outputs, or expose dangerous behaviors. OpenAI worked with external domain experts to evaluate risks across areas like cybersecurity, misinformation, chemistry, and biological threats.

This type of testing reflects a major shift in mindset.

The goal was no longer simply: “Can the model do impressive things?” But also: “What happens if capable systems are misused at global scale?”

The report repeatedly discusses concerns around dangerous instruction generation.

During internal evaluations, earlier GPT-4 versions were sometimes capable of generating unsafe or harmful information related to dangerous materials, offensive content, or exploitative behavior. OpenAI used RLHF, safety fine-tuning, rule-based reward models, and policy systems to reduce these risks significantly before public deployment.

Cybersecurity concerns also receive substantial attention. The report discusses risks involving:

• phishing assistance

• malware-related guidance

• social engineering

• exploit generation

• automation of cyber abuse workflows

Although GPT-4 isn't presented as an autonomous hacking system, OpenAI clearly recognizes that increasingly capable language models could amplify existing cybersecurity threats if deployed irresponsibly.

Another especially important topic is biosecurity.

The report explains that domain experts evaluated whether GPT-4 could meaningfully assist users with harmful biological or chemical knowledge. OpenAI specifically investigated whether the model could help lower the barrier for dangerous misuse.

This was one of the first times a major AI paper openly treated advanced language models as potential dual-use technologies with real-world security implications.

The report also emphasizes deployment monitoring and iterative safety improvement.

Rather than treating safety as something solved before release, OpenAI frames deployment itself as part of the learning process. Monitoring user interactions, identifying failure modes, updating safeguards, and improving refusal systems became ongoing operational responsibilities rather than one-time research tasks.

Historically, this section may be one of the most important parts of the entire report.

GPT-4 marks the moment when AI safety stopped being a niche research discussion and became a core component of flagship frontier model development.

That shift reflects a deeper realization across the industry: once AI systems become powerful enough for large-scale deployment, increasing capability and managing risk become inseparable engineering problems.

Discussion

Looking back at the GPT series, GPT-4 feels less like the release of a single research model and more like the beginning of a new computing platform.

GPT-1 introduced the idea of large-scale language pretraining. GPT-2 demonstrated zero-shot multitask behavior. GPT-3 showed that models could adapt through prompting and in-context learning.

But GPT-4 changes the conversation again.

According to the technical report, the focus is no longer only about making models larger or improving benchmark scores. The report repeatedly emphasizes reliability, deployment, alignment, infrastructure, multimodal interaction, and safety engineering.

That shift is historically important.

Earlier GPT papers felt like research milestones published mainly for the machine learning community. GPT-4 feels like infrastructure designed for real-world deployment at global scale.

This becomes especially clear through systems like ChatGPT.

GPT-4 was not simply released as a downloadable research artifact or benchmark model. Instead, it became part of an entire AI product ecosystem:

• conversational assistants

• coding copilots

• enterprise APIs

• productivity tools

• educational systems

• multimodal interfaces

In practice, GPT-4 helped transform large language models from isolated research demos into continuously deployed software platforms.

Another major change is the increasing secrecy surrounding frontier AI systems.

Unlike GPT-2 and GPT-3, the GPT-4 report intentionally omits many technical details, including parameter counts, architecture specifics, training compute, and dataset composition.

OpenAI explains this partly through safety concerns and the competitive landscape, but the broader implication is significant: frontier AI models were becoming strategically valuable technologies rather than purely academic research projects.

This marks the beginning of a much more closed era in large-scale AI development.

The report also shows why alignment became such a central concern.

As language models became more capable, the risks associated with hallucinations, harmful outputs, cybersecurity misuse, misinformation, and unsafe reasoning also increased. GPT-4 treats alignment not as an optional improvement layer, but as a core engineering requirement.

This is another major transition in the history of AI systems.

Earlier models were evaluated mostly on capability:

• accuracy

• perplexity

• benchmark scores

• scaling behavior

GPT-4 expands the discussion toward:

• safety

• deployment monitoring

• refusal behavior

• policy enforcement

• human oversight

• operational reliability

The model is no longer judged only by what it can do, but also by how safely and consistently it behaves in real-world environments.

In many ways, GPT-4 also represents the rise of the modern foundation model ecosystem.

Instead of training separate systems for every individual task, one large aligned model can serve as a shared base for many applications:

• coding

• tutoring

• search

• writing

• research assistance

• customer support

• multimodal interaction

• enterprise workflows

That idea fundamentally changed the software industry.

Historically, GPT-4 may ultimately be remembered less for a single benchmark result and more for what it represented: the moment large language models became practical, continuously deployed, general-purpose AI infrastructure.

Conclusion

The GPT-4 Technical Report marks one of the most important turning points in the history of modern AI systems.

According to the report, GPT-4 is not simply a larger language model. It's a multimodal, aligned foundation model designed for real-world deployment at global scale.

The model combines several major ideas that evolved throughout the GPT series:

• large-scale Transformer pretraining

• autoregressive next-token prediction

• scaling laws

• few-shot prompting

• multimodal reasoning

• reinforcement learning from human feedback

• safety-focused post-training

Together, these components produce a system that feels qualitatively different from earlier GPT models.

GPT-4 demonstrates that scaling alone is no longer the entire story.

GPT-3 showed that larger models could develop powerful emergent abilities through scale. GPT-4 shows that alignment, safety engineering, post-training refinement, and deployment infrastructure became equally important parts of building useful AI systems.

This combination of scale and alignment ultimately became the dominant paradigm behind modern frontier AI development.

The report also reflects a broader transition happening across the industry.

Large language models were no longer being treated as isolated research experiments or benchmark systems. GPT-4 pushed AI toward real-world deployment through products, APIs, multimodal assistants, coding systems, enterprise tools, and globally accessible conversational interfaces like ChatGPT.

Historically, GPT-4 represents the moment when foundation models became practical infrastructure for everyday computing.

And that shift continues shaping the direction of modern AI today.

Final Insight

Looking across the entire GPT series, the progression becomes remarkably clear.

GPT-1 introduced the idea that large-scale pretraining could produce transferable language representations. Instead of training separate NLP systems from scratch for every task, models could first learn general language patterns and then adapt through fine-tuning.

GPT-2 pushed this idea further by showing that sufficiently large language models could perform tasks in a zero-shot setting without explicit supervised training. The model was no longer just memorizing tasks – it was beginning to generalize from language itself.

GPT-3 changed the paradigm again. Few-shot prompting and in-context learning showed that models could adapt dynamically during inference simply from examples written inside the prompt. This transformed prompting into a new interface for interacting with AI systems.

Then GPT-4 expanded the idea into something much larger. The focus was no longer only about scaling models or improving benchmarks. GPT-4 introduced the era of aligned multimodal foundation models: systems designed not just to generate language, but to operate safely, follow instructions, reason across modalities, and function as deployable infrastructure for real-world applications.

Historically, that may be the most important shift of all.

GPT-4 was not simply a larger language model.

It marked the transition from experimental large language models to globally deployed AI assistants integrated into everyday computing, software development, education, productivity tools, and multimodal human-computer interaction.

And in many ways, we're still only at the beginning of that transition.

GPT-1 vs GPT-2 vs GPT-3 vs GPT-4: Key Differences

A simple way to see how the GPT series evolved is by looking at what each generation introduced.

GPT-1 introduced modern pretraining, GPT-2 showed that large language models could perform tasks through zero-shot prompting, GPT-3 pushed few-shot prompting and in-context learning into the mainstream, and GPT-4 expanded the idea further through alignment, multimodal reasoning, and real-world deployment.

The comparison below shows how the focus gradually shifted from task-specific NLP models to general-purpose AI systems capable of conversation, coding, reasoning, and multimodal understanding.

PyTorch Implementations of the GPT Architecture Evolution

GPT-1: Pre-training + Fine-Tuning Architecture

class GPT1(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(512, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(d_model)
            for _ in range(n_layers)
        ])

        self.ln_f = nn.LayerNorm(d_model)

        # Language modeling head
        self.lm_head = nn.Linear(d_model, vocab_size)

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.ln_f(x)

        logits = self.lm_head(x)

        return logits

GPT1 inherits from nn.Module, which is the base class used to build neural networks in PyTorch. The constructor (init) defines all trainable layers used by the model.

nn.Embedding(vocab_size, d_model) creates a learnable lookup table that converts token IDs into dense vectors. Each token in the vocabulary is mapped to a vector of size d_model.

The positional embedding layer adds information about token order. Since Transformers process tokens in parallel, they need explicit positional information to understand sequence structure.

nn.ModuleList([...]) stores multiple Transformer blocks while ensuring PyTorch properly tracks their parameters during training. Each TransformerBlock typically contains masked self-attention and a feed-forward network.

nn.LayerNorm(d_model) applies layer normalization before the output projection. This helps stabilize training and improves gradient flow in deep Transformer architectures.

The language modeling head (nn.Linear) projects the hidden representations back into vocabulary space. The output size equals vocab_size, producing prediction scores for every possible next token.

Inside the forward() method, input_ids.size(1) retrieves the sequence length, and torch.arange(...) generates positional indices for each token position.

The token embeddings and positional embeddings are added together to produce the initial Transformer input representation.

The model then passes the representation through each Transformer block sequentially:

for block in self.transformer_blocks:
    x = block(x)

This iterative stacking is what allows GPT models to learn increasingly abstract contextual representations.

After normalization, the final hidden states are passed into lm_head, producing logits. These logits are unnormalized prediction scores used to compute probabilities for next-token generation.

The model finally returns the logits tensor, which is typically passed through softmax during inference or used directly with CrossEntropyLoss during training.

GPT-2: Zero-Shot Multitask Architecture

class GPT2(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(1024, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                pre_layer_norm=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Like GPT-1, the model begins with token embeddings and positional embeddings. nn.Embedding converts token IDs into dense vectors, while positional embeddings provide information about token order in the sequence.

One noticeable difference is the larger positional embedding size (1024 instead of 512), allowing GPT-2 to process longer contexts.

The Transformer layers are stored using nn.ModuleList, but each TransformerBlock now uses:

pre_layer_norm=True

This means layer normalization is applied before attention and feed-forward operations rather than after them. This “Pre-LN” design significantly improves gradient flow and training stability in deeper Transformer models.

The forward pass follows the same overall pipeline:

1. Generate positional indices with torch.arange()

2. Add token and positional embeddings

3. Pass representations through stacked Transformer blocks

4. Apply final normalization

5. Project outputs into vocabulary space

The sequential block processing happens here:

for block in self.transformer_blocks:
    x = block(x)

GPT-2 also introduces a small optimization in the output layer:

self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

The bias term is removed because it provides little benefit in large language modeling setups and slightly reduces parameter count.

Finally, the model returns logits, which contain prediction scores for every token in the vocabulary at each sequence position.

GPT-3: Few-Shot / In-Context Learning Architecture

class GPT3(nn.Module):
    def __init__(
        self,
        vocab_size=50257,
        d_model=12288,
        n_layers=96,
        n_heads=96,
        context_length=2048
    ):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(context_length, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                n_heads=n_heads,
                pre_layer_norm=True,
                sparse_attention=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        self.lm_head = nn.Linear(
            d_model,
            vocab_size,
            bias=False
        )

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Compared to earlier GPT versions, this model dramatically increases scale. The embedding size (d_model=12288) and the number of Transformer layers (96) allow the network to learn highly complex language patterns and long-range dependencies.

The model also uses 96 attention heads:

n_heads=96

Multi-head attention allows the model to focus on different relationships between tokens simultaneously, improving contextual understanding.

The positional embedding length is expanded to 2048, enabling the model to process much longer sequences than GPT-2.

Each Transformer block is configured with:

pre_layer_norm=True,
sparse_attention=True

Pre-layer normalization improves training stability in very deep networks, while sparse attention reduces the computational cost of attention by limiting how many tokens attend to each other. This becomes important at GPT-3 scale, where full attention over long sequences is extremely expensive.

The forward pass follows the standard GPT pipeline:

1. Convert token IDs into embeddings

2. Add positional information

3. Pass representations through stacked Transformer blocks

4. Apply final layer normalization

5. Generate vocabulary logits

The core iterative processing happens here:

for block in self.transformer_blocks:
    x = block(x)

Finally, the output layer projects the hidden states into vocabulary space, producing logits used for next-token prediction during training and text generation.

GPT-4: Aligned Multimodal Foundation Model Architecture

class GPT4(nn.Module):
    def __init__(
        self,
        vocab_size=50257,
        d_model=12288,
        n_layers=120,
        n_heads=96,
        context_length=8192
    ):
        super().__init__()

        # Text embeddings
        self.token_embedding = nn.Embedding(
            vocab_size,
            d_model
        )

        self.position_embedding = nn.Embedding(
            context_length,
            d_model
        )

        # Vision encoder for image inputs
        self.vision_encoder = VisionTransformer(
            embed_dim=d_model
        )

        # Multimodal projection layer
        self.image_projection = nn.Linear(
            d_model,
            d_model
        )

        # Decoder-only Transformer blocks
        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                n_heads=n_heads,
                pre_layer_norm=True,
                flash_attention=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        # Language modeling head
        self.lm_head = nn.Linear(
            d_model,
            vocab_size,
            bias=False
        )

        # RLHF alignment head
        self.reward_head = RewardModel(
            hidden_size=d_model
        )

    def forward(
        self,
        input_ids,
        image_inputs=None
    ):

        positions = torch.arange(
            input_ids.size(1)
        )

        text_embeddings = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        # Encode image if provided
        if image_inputs is not None:

            image_features = self.vision_encoder(
                image_inputs
            )

            image_embeddings = self.image_projection(
                image_features
            )

            x = torch.cat(
                [image_embeddings, text_embeddings],
                dim=1
            )

        else:
            x = text_embeddings

        # Transformer decoding
        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Like previous GPT models, the architecture starts with token embeddings and positional embeddings. nn.Embedding converts token IDs into dense vector representations, while positional embeddings preserve sequence order information.

One major difference is the addition of a vision encoder:

self.vision_encoder = VisionTransformer(
    embed_dim=d_model
)

This module processes image inputs and converts them into visual feature representations that can be understood by the Transformer.

The image features are then passed through a projection layer:

self.image_projection = nn.Linear(
    d_model,
    d_model
)

This aligns image representations with the same embedding space used for text tokens, making multimodal processing possible.

The Transformer stack remains decoder-only, but now uses:

flash_attention=True

Flash Attention is an optimized attention implementation that reduces memory usage and improves training and inference speed, especially for very long context windows like 8192 tokens.

Inside the forward() method, text embeddings are created first. If an image is provided, the image is encoded and projected into embeddings:

image_features = self.vision_encoder(
    image_inputs
)

The image and text embeddings are then combined using:

x = torch.cat(
    [image_embeddings, text_embeddings],
    dim=1
)

torch.cat() concatenates tensors along the sequence dimension, allowing the Transformer to process image and text tokens together as a single sequence.

The combined representations pass through all Transformer blocks sequentially:

for block in self.transformer_blocks:
    x = block(x)

After normalization, the final hidden states are projected into vocabulary space to produce logits for next-token prediction.

The architecture also introduces a reward model head:

self.reward_head = RewardModel(
    hidden_size=d_model
)

This component represents reinforcement learning from human feedback (RLHF), which is used to align model outputs with human preferences and improve response quality and safety.

Resources:

• Pytorch Projects for GPT series

• Attention Is All You Need

• Improving Language Understanding by Generative Pre-Training (GPT-1)

• Language Models are Unsupervised Multitask Learners (GPT-2)

• Language Models are Few-Shot Learners (GPT-3)

• GPT-4 Technical Report

• Scaling Laws for Neural Language Models

• Training Compute-Optimal Large Language Models

• PaLM: Scaling Language Modeling with Pathways

• Training Language Models to Follow Instructions with Human Feedback

• Constitutional AI: Harmlessness from AI Feedback

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

• Self-Consistency Improves Chain of Thought Reasoning in Language Models

• TruthfulQA: Measuring How Models Mimic Human Falsehoods

• HumanEval: Evaluating Large Language Models Trained on Code

• Measuring Massive Multitask Language Understanding (MMLU)

Contact Me

• Github

• X

• Linkedin

Now, not everyone enjoys reading research papers or has the time to comb through and digest all that (sometimes very dense) info. So I decided to do the hard work for you and share the key insights in a series of AI paper reviews.

The goal isn’t to turn this into a heavy academic discussion, but to explain the main ideas in a clear and practical way. You'll learn what problem the paper was trying to solve, what approach it introduced, and why it mattered.

In each article, you’ll get a simple breakdown of the paper, how it works, and what you should take away from it. By the end, you should understand the key idea without needing to go through the full research paper yourself.

Paper Overview

The first paper I'll be reviewing is "Improving Language Understanding by Generative Pre-Training", by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.

Here's the actual paper if you want to read it yourself: Read the paper.

And here's a little infographic of what we'll cover here:

Table of Contents

• Executive Summary

• Goals of the Paper

• Methodology

• Transformer vs. BERT vs. GPT

• Model Architecture

• Key Techniques

• Key Findings

• Conclusions

• Limitations

• Related Work & Context

• Final Insight

• Resources

Prerequisites

To get the most out of this breakdown, it helps to be familiar with a few basic ideas:

• A general understanding of natural language processing (NLP) and how machines work with text

• A high-level idea of what a Transformer model is (you don’t need deep details, just the concept)

• The difference between supervised and unsupervised learning

• Basic machine learning concepts like training data and models

If you’re not fully comfortable with all of these, that’s okay, you can still follow along. The goal here is to keep things clear and intuitive.

Executive Summary

Before models like GPT became what we know today, there was a key limitation: AI systems were good at specific tasks, but struggled with general understanding.

In this paper, the authors introduce a simple but powerful idea. Instead of training a model separately for each task, they first train it on a large amount of unlabeled text to learn the structure of language. Then, they adapt it to specific tasks using smaller labeled datasets.

According to the authors, this two-step approach (pre-training followed by fine-tuning) allows a single model to handle many different tasks with minimal changes.

In practice, this marked a major shift: rather than building a new model for every problem, we can train one general model that learns language itself and then reuse it across tasks.

Goals of the Paper

To understand the motivation behind this work, it helps to look at the main limitations in NLP at the time.

Most models depended heavily on large labeled datasets, which weren’t always available. Many tasks simply didn’t have enough labeled data to train effective systems. On top of that, existing models were usually designed for a single task, making them hard to reuse or adapt.

Because of this, the authors aimed to reduce the reliance on labeled data and move toward a more general approach. Their goal was to build a language model that could learn from large amounts of raw text and then be applied across different tasks.

According to the paper, they also wanted to enable transfer learning: the ability to take knowledge learned from one task and apply it to others. They also wanted to improve performance without needing to redesign a new model each time.

Methodology

To understand how the authors approached this problem, let’s look at the core idea behind their method.

Pre-Training

At the heart of the paper is a simple but powerful approach built in two stages. The first stage is pre-training, where the model learns directly from raw text.

According to the authors, the model is trained on a large corpus of unlabeled text using a language modeling objective (predicting the next word in a sequence) – specifically, predicting the next word based on the previous ones to solve the intractable problem of high dimension probabilities. Through this process, the model gradually learns important aspects of language, such as grammar, context, structure, and general patterns.

The paper highlights that datasets like BooksCorpus are used in this stage because they contain long, continuous text. This is important, since it helps the model understand relationships across sentences rather than just short fragments.

Fine-Tuning (Adapting to Tasks)

Once the model has learned general language patterns, the next step is fine-tuning, where it is adapted to specific tasks using labeled data.

According to the authors, this includes tasks like question answering, text classification, natural language inference, and semantic similarity. Instead of building a new model for each task, the same pre-trained model is reused with only small adjustments.

In practice, this is what makes the approach powerful: the model already understands language at a general level, so it can quickly adapt to different tasks without needing to be redesigned from scratch.

Transformer vs. BERT vs. GPT

Before diving into GPT-1, it helps to understand how modern language models are structured. Most of them are based on the Transformer architecture, but they use it in different ways: encoder-only models (like BERT), decoder-only models (like GPT), or full encoder–decoder models.

The original encoder–decoder Transformer was mainly used for tasks like machine translation. Encoder-only models are typically used for understanding tasks such as text classification and sentiment analysis, while decoder-only models are designed for generation tasks like text creation, powering systems such as ChatGPT, Gemini, and Claude.

Illustration comparing Transformer, GPT, and BERT architectures, adapted from Comparing Large Language Models: GPT vs. BERT vs. T5 showing encoder-decoder, decoder-only, and encoder-only designs

Transformer vs BERT vs GPT: Key Differences

Model Architecture

To support this pre-training and fine-tuning approach, the GPT-1 model is built on a Transformer (decoder) architecture.

According to the authors, this choice is important for a few reasons. Unlike older models such as LSTMs, Transformers handle long-range dependencies more effectively, meaning they can better understand relationships between words that are far apart in a sentence.

They also rely on self-attention, a mechanism that allows the model to focus on the most relevant parts of the text when processing each word. This helps the model capture context more accurately.

Another key advantage is that Transformers make transfer learning more effective, since the same learned representations can be reused across different tasks with minimal changes.

The paper highlights that, in these transfer learning scenarios, Transformers outperform LSTM-based models.

Figure 1 from “Improving Language Understanding by Generative Pre-Training” (Radford et al., 2018), showing the Transformer architecture and task-specific input transformations.

Key Techniques

Along with the main approach, the authors introduce a few practical techniques that make the model more flexible across tasks.

According to the paper, different tasks are handled by converting them into text-based formats, so they can all be processed in a similar way. This makes it easier to use the same model across multiple problems without redesigning it each time.

Another important point is that the model requires only minimal architectural changes when switching between tasks. Most of the knowledge learned during pre-training is reused as-is.

The authors also include an auxiliary language modeling objective during fine-tuning, which helps the model retain its general understanding of language while adapting to specific tasks.

Key Findings

After training and evaluation, the results weren't just strong – they were surprisingly competitive.

According to the authors, the model outperformed state-of-the-art systems in 9 out of 12 tasks. It also showed clear improvements, including +8.9% in commonsense reasoning and +5.7% in question answering.

Another important observation is that the model performed well across datasets of different sizes, although performance was weaker on some smaller datasets.

This suggests that the pre-training step helped it generalize better, even when labeled data was limited.

In practice, what makes these results significant is that a single model was able to compete with specialized systems that were specifically designed for each individual task.

Figure 2 from “Improving Language Understanding by Generative Pre-Training” (Radford et al., 2018), illustrating performance gains from layer transfer and zero-shot learning behavior.

Conclusions

To wrap things up, this paper introduced a major shift in how AI systems are built.

According to the authors, instead of training a new model from scratch for every task, we can first teach a model the structure of language through pre-training, and then adapt it to specific tasks through fine-tuning. This simple idea turns out to be highly effective.

The key takeaway is that language models can develop a general understanding of text, especially when combined with Transformer architectures and large-scale data. This makes transfer learning practical across many different tasks.

In my view, this is what makes the paper so impactful. It doesn’t just improve performance on a few benchmarks. It changes the overall approach to building AI systems.

This idea later became the foundation for models like GPT-2, GPT-3, and ChatGPT, and continues to shape modern large language models today.

Limitations

Like any approach, this method comes with its own limitations.

According to the paper, one of the main challenges is the need for large amounts of unlabeled data during the pre-training stage, which may not always be easy to get. The model’s performance also depends heavily on how well the fine-tuning step is done.

The authors also note that multi-task learning was not fully explored in this work, leaving some open questions about how well the model can handle multiple tasks at the same time.

In practice, another limitation is that performance can be weaker when working with very small datasets, especially if the fine-tuning process is not carefully handled.

Related Work & Context

To better understand where this paper fits, it helps to look at the ideas it builds on.

According to the authors, earlier approaches such as word embeddings (like Word2Vec and GloVe), LSTM-based language models, and semi-supervised learning had already made progress in understanding language. But these methods were often limited to learning representations at the word level or required more task-specific design.

What this paper does differently is move beyond that. Instead of focusing only on individual words, it learns broader language representations that capture context and meaning across entire sequences. This shift is what enables the model to generalize better across different tasks.

Final Insight

If there’s one idea to take away from this paper, it’s this: you don’t need to teach an AI system every task separately.

According to the authors, once a model learns the structure of language, it can adapt to a wide range of tasks with minimal changes. That shift – from task-specific models to general language understanding – is what makes this work so important.

In my view, this is the moment where things really changed. What started here with GPT-1 became the foundation for the systems we use today, including ChatGPT and other modern language models.

Resources:

• Pytorch Projects for GPT series

• Word2Vec (Mikolov et al., 2013)

• GloVe (Pennington et al., 2014)

• Attention Is All You Need (Vaswani et al., 2017)

• Semi-supervised Sequence Learning (Dai and Le, 2015)

• Universal Language Model Fine-tuning for Text Classification (Howard and Ruder, 2018)

• Deep Contextualized Word Representations (Peters et al., 2018)

• Semi-supervised Multitask Learning for Sequence Labeling (Rei, 2017)

• Skip-Thought Vectors (Kiros et al., 2015)

• Supervised Learning of Universal Sentence Representations (Conneau et al., 2017)

Contact Me

• Github

• X

• Linkedin

A straight line equation y = ax+b answers it in the simplest way possible. y can increase, decrease, or stay the same when x changes.

On the other hand, a deep neural network tries to answer it in a flexible way. It’s only possible because of multiple layers of straight line calculations stacked one over another along with non linear adjustments to help the network adapt and produce the desired result.

Since a straight line is the essence of neural networks, I think it’s time we try to understand the subtle details of y = ax+b, which I refer to as the magical equation. We’ll also go through the basics of linear regression and classification, which should help you understand the progression of a simple straight line to a complex deep neural network.

Table of Contents

• Prerequisites

• y=ax+b

• Linear Regression

• Linear Classification

• Comparison

• Key Additions to Help Build Deep Neural Networks

• Modelling a Deep Neural Network

• Final Thoughts

Prerequisites

• A basic understanding of linear algebra, particularly y=ax+b.

• General idea about linear regression and classification.

• Familiarity with the concept of deep neural networks.

y=ax+b

A straight line simply means that output changes steadily as input changes. There are no surprises (that is, no non linearity). Let’s analyze it properly.

y => Output variable
x => Input variable
a => Amount by which y changes when x changes (slope)
b => Value of y when x is 0 (y intercept)

We can take an example and model it in the same form to understand it better.

Ms. Poly is a math teacher who wants to formulate a study plan for her students to excel in an upcoming final exam. For simplicity, she creates a rule of thumb using only one factor: the number of hours studied per week. It has a direct impact on the marks scored by a student.

Before beginning, she makes certain assumptions:

• Every student is capable of scoring at least 30 without studying.

• For every hour a student studies, an additional 3 marks can be scored.

She then comes up with the following equation based on her ideas: y = 3x+30

y => Marks scored.
x => Number of hours studied.
a=3 => Increase in marks for every hour studied
b=30 => Minimum marks

In the above graph, she plots the points based on the results of the equation. As expected, it is a straight line. If she needs the marks scored for 9 hours of study, she can get it by just substituting x=9 in y=3x+30. Note that the data (x and y) are just based on her hunch and aren’t real.

But Ms. Poly wants to guide her students on how to prepare for the final exam based on actual data. So she conducts a pop quiz and grades it. In order to formulate a study plan, she interviews her students and collects information on how many hours they study math per week. She creates a table with two columns: number of hours studied (x) per week and marks scored (y). She tries her old formula y=3x+30, but it doesn’t seem to work. Thus, she doesn’t have any sensible equation describing the relation between x and y.

Let’s assume that a new student who hasn’t attended any exam (no y available) joins the class the next day, and Ms. Poly only knows the number of hours dedicated per week (x). How can she answer the question below?

If the new student studies for a certain number of hours (x), what can be the marks scored (y) in the exam?

It’s impossible unless there’s an equation defining the sample data. So, her task is to find one that fits the given points. This process is called curve fitting or regression.

Linear Regression

The core idea of linear regression to find a straight line that captures the trend of the existing data to facilitate predictions for new input data. Now, let’s dive straight into the example to understand the concept better.

Ms. Poly is determined to arrive at a solution. She plots the collected data on a graph to get a better picture.

She has absolutely no idea how x and y are related. So, she must figure out a formula, by trial and error, that roughly fits the points. She has to start with an intuitive guess, try to improve it in the subsequent steps and then arrive at the best possible solution.

Trial 1: Ms. Poly begins with her previous straight line equation.

y = 3x+30

She substitutes different values of x and plots it alongside the collected input data. This way she can get a clear picture of the differences in her assumption and reality.

Trial 2: She observes that the line needs a little more slope. This simply means that, in reality, more marks are being scored for every additional hour of study. By changing it from 3 to 4, the equation becomes:

y = 4x+30

The following graph depicts the new line alongside the sample data:

Trial 3: It looks better but she feels there is a need to shift the whole line upwards. This means that higher marks are being scored even if a student doesn’t dedicate any time for math in a week. She decides to retain the previous slope but changes the starting marks by 10, thus arriving at:

y = 4x+40

This particular line covers most of the points and can be considered the best possible solution.

Now, if she wishes to ascertain the marks scored by the new student who studied for 3.5 hours, she pins the value inside the formula and calculates the answer: y = 4*(3.5)+40=54

We saw how Ms. Poly arrived at a straight line equation to predict the output for an unknown input. Now she can chalk out a study plan for her class based on the equation.

Here, an expression is formulated to ascertain the change in output when the input changes. It looks like Ms. Poly is thinking like a data scientist. She has in fact modelled a very simple neural network for regression. The equation y=4x+40 can be considered as the only neuron (processing unit) within it. She’s adjusted the parameters a (weight) and b (bias) to arrive at the final formula which covers most of the points (thus minimizing the loss).

Here’s a breakdown of the y = 4x+40 equation:

y => Marks scored.
x => Number of hours studied.
a=4 => Increase in marks for every hour studied
b=40 => Minimum marks

At present, it is a rudimentary neural network which has no layering and non-linearity.

Now let’s shift our attention to a completely different scenario. Ms. Poly, being a teacher, wants to ensure that all her students pass the exam. Assuming, as an end result, she’s not interested in predicting the marks scored. She just wants to know:

If a student studies for a certain number of hours (x), will the student pass/fail(y) the exam?

This leads her to the process of classification.

Linear Classification

The linear classification process uses a simple straight line to divide the data into categories or classes. The line acts as a boundary so that the classes fall on either side of it. First, Ms. Poly defines the boundary condition for pass and fail.

If marks scored>=50, pass

If marks scored<50, fail

According to the data table, x=3 corresponds to y=52 (boundary condition). Therefore she considers x=3 as the classification line***.***

x=3 seems to segregate the points into the categories properly. She tries to confirm it by substituting another value. Thus, if a student studied for 9 hours, the score would lie towards the right side of x=3. So, they’d pass as per the classification equation.

Again, she’s arrived at an expression to ascertain the change in output when the input changes. But here, she has modelled a basic neural network for classification. The equation x=3 is the only neuron within it. It can be considered to be having two parts as explained below.

1. Pre-Activation Part: This portion of the neuron computes an intermediate value which is helpful in further processing. She’s figured out the parameters a (weight) and b (bias) to arrive at the following formula: z = x-3

    z => Intermediate Value.
    x => Number of hours studied.
    a=1 => Influence of the number of hours studied on the marks scored
    b=-3 => Minimum number of hours to study to pass the exam = 3
   

2. Activation Part: This portion triggers the neuron to make decisions based on a threshold value. The following equation segregates the points into two classes.

    y = 1 (Pass) if z>=0
    y = 0 (Fail) if z<0
   

This is a very plain neural network which has no layering and non-linearity but has pre-activation and activation parts inside a neuron.

Comparison

We looked at the examples of both linear regression and classification used by Ms. Poly. Regression helps in predicting a value while Classification helps in decision making. Let’s draw a small table to summarize the differences.

Upon careful observation we notice that both answer the question of how input change affects output.

But at a slightly higher level of complexity than a straight line. Because in the case of both regression and classification, we try to figure out the equation parameters by trial and error.

Here, since the requirements are simple, Ms. Poly just uses a straight line to solve both. A simple linear equation can handle only one steady trend. But in real life, problems that need solving are far more challenging and unpredictable. Some examples are:

Image Classification: An output label is produced based on the input images.

Text Translation: An English sentence can be given as an input to be translated to say, Spanish.

Chatbots: A text prompt is typed in by a user and a meaningful and relevant output is generated.

She probably should have to use a deep neural network if both data and task were complex. That presents another question: How does one build a deep neural network?

We will explore it further by extending the same example to a more realistic version.

Key Additions to Help Build Deep Neural Networks

In the above sections, we noted that Ms. Poly was interested in predicting the exam results of a student using just one factor - number of hours studied. However, in practice, is that one factor sufficient in determining the marks scored or whether the student passes the exam?

No. It’s not enough. She needs to take into account a lot of aspects like:

• Number of hours studied

• Number of hours of sleep/rest

• Burnout due to over-studying

• Difficulty level of topics in math

• Pattern of the exam, and so on.

All the above neither act independently nor do they have a simple linear relation with the marks scored. So, she has to solve this problem by stacking the contributing factors one above the other in layers and also adding the element of non linearity. Let’s take a look at each in detail.

Layering

Burnout leads to lower score whereas good sleep increases score. But burnout can be reduced if the student is well rested. So, the impact on the final score when these two factors interact should be taken into account. This is possible only when the system solves it in layers. The first layer can deal with how they independently influence the score, the next layer can explore the interaction between them.

Non-Linearity

If the number of hours studied increases, the score might increase but when burnout overpowers the effect of study hours, the score reduces. The combined effect results in a non-linear graph. There is a rise and then dip in the score based on number of hours studied. It’s evident that the relationship is not straightforward as in a straight line. That’s where it becomes necessary to add non-linearity in the calculations. It helps the system to respond differently according to the conditions, allowing for flexibility in dealing with real world data and conditions.

Thus, Ms. Poly would have to extend the idea of linear regression/classification by including layering and non-linearity to build a fully functional neural network to help build a practical study plan.

Modelling a Deep Neural Network

Ms. Poly should start the work on modelling a deep neural network by following the steps mentioned below:

Step #1 - Define the Problem Clearly

The following factors should be considered before she begins the process of modelling:

• What are the input features?

• What are the output features?

• What type of problem is it (regression/classification)?

Step #2 - Define the Input Layer

The input features form the first layer. There is no computation in this stage. They are represented as:

x1: Number of hours studied
x2: Number of hours of sleep/rest
x3: Burnout due to over-studying
x4: Difficulty level of topics in Maths
x5: Pattern of the exam

Step #3 - Define the First Hidden Layer

This step consists of two parts:

Apply Linear Transformation: The actual learning begins here. A straight line equation is used to understand the combined effect of the inputs. The general formula is z=Wx+b.

z: Intermediate value or Pre-activation
W: Weight matrix which consists of values corresponding to the impact of
each input feature
x: Matrix consisting of input features, [x1, x2, x3, x4, x5]
b: Bias which represents the initial assumptions of the teacher(when x=0)

It looks similar to a linear regression/classification equation. At first W and b are initialized to random values. Then in the subsequent steps, they are adjusted like it was done in earlier examples. We can consider the following combinations assuming we have two neurons in this layer:

Neuron 1: It can focus on study hours, burnout, and rest, with other features contributing less significantly.

Neuron 2: It can emphasize more on the difficulty level of the topic and the exam type compared to other inputs.

It’s important to note that this layer doesn’t calculate the interactions between the features but only on the way different linear combinations work together but independently. To make it clearer, how they contribute independently are added together. We don’t know how one input feature influences the other. For example, we know sleep increases score and burnout reduces score, but what we don’t know at this stage is if sleep reduces burnout, which in turn can influence the final score.

Add Non-Linearity: This step, also called activation, helps in capturing the complexities in different combinations of the features. Less study results in low marks, and too much burnout also results in low marks. It means there is a curve in the score graph which can’t be represented by a linear equation. The activation function is applied to the intermediate value and can be expressed as:

a = g(z)

a: Activation output
g: Activation function
z: Intermediate value or Pre-activation

For example: ReLU is an activation function which outputs z only if z is positive, else 0.

y = ReLU(z)=max(0,z)

We can see that it has no steady slope and is a non-linear activation function. It can suit this scenario as it lets the value pass through to the next layer only if the combined effect of features is greater than 0. Neuron 1 will let it’s output go to the next layer only if the intermediate value (z) that results from study hours, burnout and rest, is large enough to be influencing the final decision, else it’s ignored. There are multiple options for non-linear activation functions that one can choose from.

Step #4 - Stack Layers One Above the Other

This step helps in learning the mutual interactions between the inferences learned from the first hidden layer. The network attempts to understand the intricate details of the influencing factors and build a stable system. It is here that details of whether sleep reduces burnout are figured out. Every layer consists of linear and non linear transformations applied on the input, which are values obtained from the previous layer. Likewise multiple layers can be stacked one over the other based on the requirements. In this example, for representation, we have taken two hidden layers with two neurons each. The number of layers and neurons can vary based on requirements.

Step #5 - Define the Output Feature(s)

This appears to be the final stage in a deep neural network. Ms. Poly can decide what she wants for output: predict the marks scored by a student or predict if the student passes/fails the exam. If she wants the final marks scored, she just has to apply linear transformation in the neuron in the final layer to produce the output. If she wants pass/fail status, she has to apply both linear and non-linear transformations to achieve the desired results.

The diagram below shows an abstract representation of the deep neural network.

The next steps are:

Training the model: The network is trained in the following way:

• Random weights and biases are assigned to the linear transformation portions of the network.

• Then the network makes a prediction which is compared with the expected result.

• If there are gaps between the actual result and the predicted result, corrections are made in weights and biases (this step is similar to what was done in linear regression and classification).

• The steps above are repeated until the results improve.

Using the model: After the model has been trained, it is capable of yielding results for new input values.

Final Thoughts

In this article, we began with the basics of a straight line equation. Then we gradually navigated through slightly more elaborate concepts like linear regression and classification. They laid the groundwork for delving into the seemingly mysterious deep neural networks. But they are in fact built by stacking layers of linear transformations and non-linear activations, which help understand sophisticated real world patterns.

Despite all the complexities and layers, we can see that the straight line remains the foundation upon which neural networks are built. As we saw earlier, the equation that a deep neural network begins with is our magical equation: y = ax+b.

If you’re new to Machine Learning and are just getting started, then a free Kaggle or Colab might be enough for you. But that won’t be the case when you want to go deeper. You’ll need a GPU, which can get costly if you’re continuously using it on the cloud.

But there’s some good news: you can utilize your computer’s Nvidia GPU (GTX/RTX) quite easily and perform machine learning-related tasks right on your local machine. The cool thing is, it won’t cost you anything other than the electricity it uses!

When you’re running Machine Learning models on your local machines, the most suitable operating system is a Linux-based one, like Ubuntu. But Windows has improved a lot for this purpose. If you’re using the latest Windows 11, you can leverage Windows Subsystem for Linux (WSL) and use your GPU directly for Machine Learning-related workflows.

This process can be quite tricky, though, as can making two popular Machine Learning frameworks, TensorFlow and PyTorch, compatible with your system GPU in Windows 11. That’s why I have written this comprehensive guide to ease your pain.

In it, I’ll help you set up CUDA on Windows Subsystem for Linux 2 (WSL2) so you can leverage your Nvidia GPU for machine learning tasks.

By following these steps, you’ll be able to run ML frameworks like TensorFlow and PyTorch with GPU acceleration on Windows 11.

Keep in mind that this guide assumes you have a compatible Nvidia GPU. Make sure to check Nvidia's official compatibility list before proceeding.

I have also prepared a video for you that’ll help you follow proper guidelines throughout this article.

Also, if this tutorial helps you, then don’t forget to add a star to the GitHub repository CUDA-WSL2-Ubuntu-v2. If you face any issues or have any suggestions/improvements, then please raise an issue in the GitHub repository. Currently, the live website is available at ml-win11-v2.fahimbinamin.com.

Table of Contents

1. Prerequisites

2. Windows Terminal

3. Windows PowerShell (Latest & Greatest)

4. Configure Windows Terminal

5. Configuration of my computer

6. CPU Virtualization

7. Install WSL2

8. Install Latest LTS Ubuntu via WSL2

9. Update & Upgrade Ubuntu Packages

10. Install and Configure Miniconda

11. Install Jupyter & Ipykernel

12. Nvidia Driver

13. Install CUDA dependencies

14. CUDA Toolkit

15. Add Path to Shell Profile for CUDA

16. nvcc Version

17. cuDNN SDK

18. TensorFlow GPU

    • Check TensorFlow GPU

19. PyTorch GPU

    • Check PyTorch GPU

    • Check PyTorch & TensorFlow GPU inside Jupyter Notebook

20. Conclusion

Prerequisites

Before you begin, make sure you have the following requirements met:

• Windows 11 operating system

• Nvidia GPU (GTX/RTX series)

• Administrator access to your PC

• At least 30 GB of free disk space

• Internet connection for downloads

• Latest Nvidia drivers installed

Windows Terminal

First, you’ll need to ensure that you have Windows Terminal installed properly in your operating system. It is the newest terminal application for users of command-line tools and shells like Command Prompt, PowerShell, and WSL. You can download it from the Microsoft Store.

After ensuring that it’s installed properly, you can proceed to the next steps.

Windows PowerShell (Latest & Greatest)

Windows PowerShell is a modern and updated command-line shell from Microsoft. You can use some Linux specific commands directly on it. It comes with built-in command suggestions. You can download it from the official GitHub page.

Download the latest x64 installer and install it. After ensuring that it is installed properly, you can proceed to the next steps.

Configure Windows Terminal

Now you’ll need to configure your Windows Terminal to use PowerShell as the default shell. It’s optional and you might skip this step. But I recommend doing it for a better experience.

Open Windows Terminal. Click on the down arrow icon in the title bar and select "Settings".

In the Settings tab, under "Startup", find the "Default profile" dropdown menu. Select "PowerShell" from the list.

Now for the "Default terminal application", select "Windows Terminal".

By default, Windows PowerShell always shows the version number in the title bar. If you want to disable it, select the "PowerShell" profile from the left sidebar. Click on the "Command Line" field and add an --nologo argument at the end of the command. After this, the line becomes "C:\Program Files\PowerShell\7\pwsh.exe" --nologo.

If you don’t use other shells frequently and want to hide them in the dropdown, then you’ll need to select those profiles one by one from the left sidebar. Scroll down to the bottom and find the "Hide profile from dropdown" toggle and enable it. It will hide that specific shell from the dropdown menu.

For example, I am hiding the Azure Cloud Shell profile as I don't use it frequently:

Now click on the "Save" button at the bottom right corner to apply the changes. Close the Windows Terminal for now.

Configuration of My Computer

I figured it’d be helpful to share my current computer’s configuration so you can have a clear idea of which setup I’m using in this guide. Here are the details:

Now that you have an idea about my computer’s configuration, we can proceed to the next steps.

CPU Virtualization

As we are going to use WSL2, we’ll need to make sure that the CPU virtualization is enabled. To check whether virtualization is enabled or not from Windows, simply open the Windows Task Manager. Go to the Performance tab and select CPU from the left sidebar. In the bottom right corner, you will see the Virtualization status. If it shows "Enabled", then you are good to go. If it shows "Disabled", then you need to enable it from the BIOS.

⚠️ You have to ensure that CPU Virtualization is enabled in your BIOS settings. Different manufacturers have different ways to access the BIOS. Usually, you can access the BIOS by pressing the Delete or F2 key during the boot process. Once in BIOS, look for settings related to "Virtualization Technology" or "Intel VT-x"/"AMD-V" and make sure it is enabled. Save the changes and exit the BIOS.

Install WSL2

Open the Windows Terminal or Windows PowerShell as an administrator. Run the following command to install WSL2 along with the latest Ubuntu LTS distribution:

wsl.exe --install

It will install Windows Subsystem for Linux 2 (WSL2). After the installation is complete, you will be prompted to restart your computer. Do so to finalize the installation.

⚠️ If you encounter any issues during installation, refer to the official Microsoft documentation for troubleshooting WSL installation problems.

Install Latest LTS Ubuntu via WSL2

Open the Windows Terminal or Windows PowerShell again with the administrator privileges. If you want to check the available Linux distributions to install via WSL, run the following command:

wsl.exe --list --online

For installing any specific distribution, run the following command:

wsl.exe --install <DistroName>

We are going to install the latest LTS Ubuntu distribution. As of now, the latest LTS version is Ubuntu 24.04. But I prefer to install the Ubuntu directly as it always points to the latest LTS version. So, run the following command:

wsl.exe --install Ubuntu

You need to give it a default user account name. For me, I am going with fahim.

It also comes with a nice GUI management tool for WSL.

You can configure a lot of stuff in it including restricting core, RAM, disk space and a lot of specifications from the settings GUI window.

Update & Upgrade Ubuntu Packages

Open your Ubuntu terminal from Windows Terminal. First, we need to update and upgrade the existing packages to their latest versions.

To update the Ubuntu system, simply use the following command:

sudo apt update -y

To upgrade all the packages at once, simply use the following command:

sudo apt upgrade -y

⚠️ Make sure that you have a stable internet connection during the update and upgrade process to avoid any interruptions.

Install and Configure Miniconda

In Machine Learning, we need to manage multiple environments with different package versions. Conda is a popular package and environment management system that makes it easy to create and manage isolated environments for different projects. We will install Miniconda, a minimal installer for Conda, to manage our Python environments. But if you prefer Anaconda, you can install it instead.

Go to the official website of Miniconda. Currently the Miniconda installer is inside Anaconda here. If the official website gets updated, you can always search for "Miniconda installer" on Google to find the latest version. Also, you can create an issue in the official GitHub repository of this project to notify me about it.

As we are installing it inside WSL, we have to select the macOS/Linux Installation. Then select Linux Terminal Installer and choose Linux x86 for downloading the installer.

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

It will download the installer to your WSL directory. Then use the following command to install it properly:

bash ~/Miniconda3-latest-Linux-x86_64.sh

⚠️ Make sure that you are in the correct directory where the installer is downloaded. If you downloaded it to a different location, adjust the path accordingly. Also, replace bash with zsh or sh if you are using a different shell.

Make sure to choose the initialization option properly. I prefer to keep the conda env active whenever I open a new shell. Therefore, I chose "Yes".

Make sure that the installation succeeds without any errors.

For the changes to take effect, you can close and reopen the current shell. But you can also do that without closing and reopening the shell by applying the command below.

source ~/.bashrc

⚠️ If you’re using a different shell like zsh or fish, make sure to source the appropriate configuration file (e.g., ~/.zshrc for zsh).

Install Jupyter & Ipykernel

I prefer to use Jupyter Notebook for running my machine learning experiments. It provides an interactive environment for coding and data analysis. We’ll install Jupyter Notebook and Ipykernel to run Jupyter notebooks in our conda environment. We will do that in all conda environments starting with the base environment. It also helps us to keep the conda environment kernel inside Jupyter Notebook.

First, make sure that you are in the base conda environment. You will see (base) on the left side of the terminal.

Now install Jupyter and Ipykernel both by applying the following command:

conda install jupyter ipykernel -y

Make sure that you accept the terms of service of Conda.

Now, I will create a separate conda environment for both TensorFlow and the PyTorch GPU. You can directly install them in the base environment or in any other environment as per your preference. I am not specifying any specific Python version while creating the environment. It will automatically install the latest stable version of Python.

conda create -name ml -y

To activate any specific conda environment, you have to use the following command:

conda activate <conda-env-name>

For example, if I want to activate my newly created ml environment, I will use this command:

conda activate ml

If you’re not sure which conda environments are installed in your system, you can check all available and installed conda environments in your system by running the following command:

conda env list

Nvidia Driver

Ensure that you have the latest Nvidia drivers installed on Windows. WSL2 uses the Windows driver, so no separate driver installation is needed in Ubuntu. You can download the latest drivers from the official Nvidia website.

If you are just installing the latest GPU driver, then after installing the drivers, restart your computer to ensure the changes take effect. You can either use the GeForce Game Ready Driver or the NVIDIA Studio Driver. But I recommend using the Studio Driver for better stability with creative and ML applications.

Install CUDA Dependencies

You might face some issues if you do not have the CUDA dependencies installed properly. I recommend that you install the required dependencies before proceeding further:

sudo apt install gcc g++ build-essential

After installing the dependencies, you can then verify the CUDA installation if you had any issues earlier.

CUDA Toolkit

TensorFlow GPU is very picky about the CUDA version. So we need to install a specific version of CUDA Toolkit that is compatible with the TensorFlow version we are going to install.

To understand exactly which CUDA version is compatible with which TensorFlow version, you can check the official TensorFlow GPU support matrix here.

At the time I’m writing this article, the TensorFlow GPU documentation says that we should have CUDA Toolkit 12.3. So I will ensure that I install exactly that version. You can simply click on that version link in the official docs and it will redirect you to the official Nvidia CUDA Toolkit download page. But if the link gets updated in the future, you can always search for "Nvidia CUDA Toolkit" on Google to find the latest version.

As TensorFlow GPU is asking for exact Version 12.3, I will select version 12.3.0 exactly.

In the CUDA Toolkit download page, make sure to choose the operating system as Linux, Architecture as x86_64, Distribution as WSL-Ubuntu, Version as 2.0 and the Installer type as runfile(local).

⚠️ As we are using Ubuntu in our WSL2, you can also choose Ubuntu as your operating system. But I prefer to choose WSL-Ubuntu for better compatibility.

After selecting those, it will give you the download commands. You have to apply them sequentially. Make sure that you don't keep the checkmark in "Kernel Objects" during installing CUDA.

⚠️ Make sure to copy and paste the commands one by one in your WSL Ubuntu terminal to download and install the CUDA Toolkit properly. If you face any issues related to CUDA dependency, then quickly go through the Install CUDA dependencies section, where I have explained how to install the CUDA dependencies properly.

Add Path to Shell Profile for CUDA

After installing CUDA Toolkit, we need to add the CUDA binaries to our shell profile for easy access. This will allow us to run CUDA commands from any directory in the terminal.

Note that, depending on the shell you are using (bash, zsh, and so on), you need to add the CUDA path to the appropriate configuration file. Make sure to replace .bashrc with .zshrc or other configuration files if you are using a different shell.

To add the CUDA binary path, follow the command below:

echo 'export PATH=/usr/local/cuda-12.3/bin:$PATH' >> ~/.bashrc

You have to use the updated path where you installed it. Your terminal will show it after installing the CUDA:

Now, you need to add the path inside the Library path. Just use the exact path where you installed CUDA. Your terminal will list the path properly.

echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.3/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc

After adding those paths, you need to source the shell profile for the changes to take effect. You can do that by running the following command:

source ~/.bashrc

nvcc Version

NVCC stands for Nvidia CUDA Compiler. It is basically a compiler driver for the CUDA platform that allows developers to write parallel programs to run on Nvidia GPUs. As we have already installed the CUDA toolkit, we need to see whether the compiler is also properly activated. To check that, we need to verify the version.

Verify that CUDA is properly installed by checking the version:

nvcc --version

If the output shows the correct CUDA version, then you have successfully installed CUDA Toolkit in your WSL2 Ubuntu environment.

cuDNN SDK

The cuDNN (CUDA Deep Neural Network) SDK is a GPU accelerated library of primitives for deep neural networks, developed by Nvidia. It provides highly optimized building blocks for common deep learning operations, significantly speeding up the training and inference processes of AI models on Nvidia GPUs.

Note: Even though TensorFlow GPU suggests a specific cuDNN version, it’s often compatible with multiple versions. Because of this, I recommend downloading the latest cuDNN version that is compatible with your installed CUDA version. You can find the cuDNN download page here.

Select the Operating System as Linux, Architecture as x86_64, Distribution as Ubuntu, Version as 24.04, Installer Type as deb (local), Configuration as FULL. After selecting those, it will give you the download commands. You have to apply them sequentially.

⚠️ Make sure to copy and paste the commands one by one in your WSL Ubuntu terminal to download and install the cuDNN SDK properly. If you face any issues related to CUDA dependency, then quickly go through the Install CUDA dependencies section, where I have explained how to install the CUDA dependencies properly.

TensorFlow GPU

Now, we are going to install TensorFlow GPU in our conda environment. Make sure that you have activated the conda environment where you want to install it. I’m going to install it in my previously created ml environment. To activate it, I’ll use the following command:

conda activate ml

⚠️ Make sure that you have activated the correct conda environment before installing TensorFlow GPU. You will see the environment name in the terminal prompt.

I will install ipykernel and jupyter in this new environment.

conda install jupyter ipykernel -y

Now, to install TensorFlow GPU, I will simply use the following command:

pip install tensorflow[and-cuda]

It might take a couple of minutes depending on the internet speed you have. Just have patience and wait for it to finish the installation.

Check TensorFlow GPU

After installing TensorFlow GPU, we need to verify that it is working properly with GPU support. Open a Python shell in your Ubuntu terminal and run the following commands:

python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

If the output shows a list of available GPU devices, then TensorFlow GPU is successfully installed and working properly.

PyTorch GPU

Now, we’re going to install PyTorch GPU in our conda environment. Make sure that you have activated the conda environment where you want to install it. I’m going to install it in my previously created ml environment. To activate it, I will use the following command:

conda activate ml

Installing PyTorch GPU is very straightforward. You can use the official PyTorch installation command generator here.

Make sure to select PyTorch Build as the latest Stable one, Your OS as Linux, Package as Pip, Language as Python. For the Compute Platform, select the CUDA version that matches your installed CUDA Toolkit. For me, it is CUDA 12.3. But, if you can not find the exact one then choose the closest. As CUDA 12.3 is not available for me now, I am choosing CUDA 12.6.

After selecting those, it will give you the installation command. You have to apply it in your WSL Ubuntu terminal.

It might take a couple of minutes depending on the internet speed you have. Just have patience and wait for it to finish the installation.

Check PyTorch GPU

After installing PyTorch GPU, verify that it is working properly with GPU support. Open a Python shell in your Ubuntu terminal and run the following commands:

python3 - << 'EOF'
import torch
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.current_device())
print(torch.cuda.device(0))
print(torch.cuda.get_device_name(0))
EOF

The output should look similar to the screenshot, showing:

• True: GPU is available for PyTorch

• 1: Number of detected CUDA devices

• 0: Index of the current active CUDA device

• A device object representation

• NVIDIA GeForce RTX 3060 (or your GPU name)

Check PyTorch & TensorFlow GPU inside Jupyter Notebook

Now that the environment is fully configured, we will verify GPU support directly inside Jupyter Notebook. This ensures both PyTorch and TensorFlow can successfully detect and use your GPU.

1. Test PyTorch GPU

Create a new Jupyter Notebook and run the following commands one by one:

import torch

print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.current_device())
print(torch.cuda.device(0))
print(torch.cuda.get_device_name(0))

If everything is configured correctly, you will see your GPU (for example NVIDIA GeForce RTX 3060) detected properly:

2. Test TensorFlow GPU

Next, run the following code to check whether TensorFlow detects your GPU:

import tensorflow as tf

print(tf.config.list_physical_devices('GPU'))

You can also check the number of GPUs detected:

print("Num GPUs Available:", len(tf.config.list_physical_devices('GPU')))

Finally, run TensorFlow’s built-in GPU validation (warnings are normal):

import tensorflow as tf

assert tf.test.is_gpu_available()
assert tf.test.is_built_with_cuda()

If TensorFlow logs show your GPU model (such as RTX 3060), then TensorFlow GPU is successfully installed and fully working inside Jupyter Notebook.

Conclusion

Thank you so much for reading all the way through. I hope you have been able to configure your Windows 11 computer properly for running almost any kind of Machine Learning-based experiments.

To get more content like this, you can follow me on LinkedIn and X. You can also check my website and follow me on GitHub if you are into open source and development.

While many services for tracking ML lineage exist, creating a comprehensive and manageable lineage often proves complicated.

In this article, I’ll walk you through integrating a comprehensive ML lineage solution for an ML application deployed on serverless AWS Lambda, covering the end-to-end pipeline stages:

• ETL pipeline

• Data drift detection

• Preprocessing

• Model tuning

• Risk and fairness evaluation.

Table of Contents

1. What is Machine Learning Lineage?

2. What We’ll Build

   • The System Architecture - AI Pricing for Retailers

   • The ML Lineage

3. Workflow in Action

4. Step 1: Initiating a DVC Project

5. Step 2: The ML Lineage

   • Stage 1: The ETL Pipeline

   • Stage 2: The Data Drift Check

   • Stage 3: Preprocessing

   • Stage 4: Tuning the Model

   • Stage 5: Performing Inference

   • Stage 6: Assessing Model Risk and Fairness

   • Test in Local

6. Step 3: Deploying the DVC Project

7. Step 4: Configuring Scheduled Run with Prefect

   • Configuring the Docker Image Registry

   • Configure Prefect Tasks and Flows

   • Test in Local

8. Step 5: Deploying the Application

   • Test in Local

9. Conclusion

Prerequisites:

• Knowledge of key Machine Learning / Deep Learning concepts including the full lifecycle: data handling, model training, tuning, and validation.

• Proficiency in Python, with experience using major ML libraries.

• Basic understanding of DevOps principles.

Tools we’ll use:

Here is a summary of the tools we’re going to use to track the ML lineage:

• DVC: An open-source version system for data. Used to track the ML lineage.

• AWS S3: A secure object storage service from AWS. Used as a remote storage.

• Evently AI: An open-source ML and LLM observability framework. Used to detect data drift.

• Prefect: A workflow orchestration engine. Used to manage the schedule run of the lineage.

What is Machine Learning Lineage?

Machine learning (ML) lineage is a framework for tracking and understanding the complete lifecycle of a machine learning model.

It contains information at different levels such as:

• Code: The scripts, libraries, and configurations for model training.

• Data: The original data, transformations, and features.

• Experiments: Training runs, hyperparameter tuning results.

• Models: The trained models and their versions.

• Predictions: The outputs of deployed models.

ML lineage is essential for multiple reasons:

• Reproducibility: Recreate the same model and prediction for validation.

• Root cause analysis: Trace back to the data, code, or configuration change when a model fails in production.

• Compliance: Some regulated industries require proof of model training to ensure fairness, transparency, and adherence to laws like GDPR and the EU AI Act.

What We’ll Build

In this project, I’ll integrate an ML lineage into this price prediction system built on AWS Lambda architecture using DVC, an open-source version control system for ML applications.

The below diagram illustrates the system architecture and the ML lineage we’ll integrate:

Figure A: A comprehensive ML lineage for an ML application on serverless Lambda (Created by Kuriko IWAI)

The System Architecture: AI Pricing for Retailers

The system operates as a containerized, serverless microservice designed to provide optimal price recommendations to maximize retailer sales.

Its core intelligence comes from AI models trained on historical purchase data to predict the quantity of the product sold at various prices, allowing sellers to determine the best price.

For consistent deployment, the prediction logic and its dependencies are packaged into a Docker container image and stored in AWS ECR (Elastic Container Registry).

The prediction is then served by an AWS Lambda function, which retrieves and runs the container from ECR and exposes the result via AWS API Gateway for the Flask application to consume.

If you want to see how to build this from the ground up, you can follow along with my tutorial How to Build a Machine Learning System on Serverless Architecture.

The ML Lineage

In the system, GitHub handles the code lineage, while DVC captures the lineage of:

• Data (blue boxes): ETL and preprocessing.

• Experiments (light orange): Hyperparamters tuning and validation.

• Models and Prediction (dark orange): Final model artifacts and prediction results.

DVC tracks the lineage through separate stages, from data extraction to fairness testing (yellow rows in Figure A).

For each stage, DVC uses an MD5 or SHA256 hash to track and push metadata like artifacts, metrics, and reports to its remote on AWS S3.

The pipeline incorporates Evently AI to handle data drift tests, which are essential for identifying shifts in data distributions that could compromise the model's generalization capabilities in production.

Only models that successfully pass both the data drift and fairness tests can serve predictions via the AWS API gateway (red box in Figure A).

Lastly, this entire lineage process is triggered weekly by the open-source workflow scheduler, Prefect.

Prefect prompts DVC to check for updates in data and scripts, and executes the full lineage process if changes are detected.

Workflow in Action

The building process involves five main steps:

1. Initiate a DVC project

2. Define the lineage stages with the DVC script dvc.yaml and corresponding Python script

3. Deploy the DVC project

4. Configure scheduled run with Prefect

5. Deploy the application

Let’s walk through each step together.

Step 1: Initiating a DVC Project

The first step is to initiate a DVC project:

$dvc init

This command automatically creates a .dvc directory at the root of the project folder:

.
.dvc/
│
└── cache/         # [.gitignore] store dvc caches (cached actual data files)
└── tmp/           # [.gitignore]
└── .gitignore     # gitignore cache, tmp, and config.local
└── config         # dvc config for production
└── config.local   # [.gitignore] dvc config for local

DVC maintains a fast, lightweight Git repository by separating the original data in large files from the repository.

The process involves caching the original data in the local .dvc/cache directory, creating a small .dvc metadata file which contains an MD5 hash and a link to the original data file path, pushing only the small metadata files to Git, and pushing the original data to the DVC remote.

Step 2: The ML Lineage

Next, we’ll configure the ML lineage with the following stages:

1. etl_pipeline: Extract, clean, impute the original data and perform feature engineering.

2. data_drift_check: Run data drift tests. If they fail, the system exits.

3. preprocess: Create training, validation, and test datasets.

4. tune_primary_model: Tune hyperparameters and train the model.

5. inference_primary_model: Perform inference on the test dataset.

6. assess_model_risk: Runs risk and fairness tests.

Each stage requires defining the DVC command and its corresponding Python script.

Let’s get started.

Stage 1: The ETL Pipeline

The first stage is to extract, clean, impute the original data, and perform feature engineering.

DVC Configuration

We’ll create the dvc.yaml file at the root of the project directory and add the etl_pipeline stage:

dvc.yaml

stages:
  etl_pipeline:
    # the main command dvc will run in this stage
    cmd: python src/data_handling/etl_pipeline.py

    # dependencies necessary to run the main command
    deps:
      - src/data_handling/etl_pipeline.py
      - src/data_handling/
      - src/_utils/

    # output paths for dvc to track
    outs:
      - data/original_df.parquet
      - data/processed_df.parquet

The dvc.yaml file defines a sequence of steps (stages) using sections like:

• cmd: The shell command to be executed for that stage

• deps: Dependencies that need to run the cmd

• prams: Default parameters for the cmd defined in the params.yaml file

• metrics: The metrics files to track

• reports: The report files to track

• plots: The DVC plot files for visualization

• outs: The output files produced by the cmd, which DVC will track

The configuration helps DVC ensure reproducibility by explicitly listing dependencies, outputs, and the commands of each stage. It also helps it manage the lineage by establishing a Directed Acyclic Graph (DAG) of the workflow, linking each stage to the next.

Python Scripts

Next, let’s add Python scripts, ensuring the data is stored using the file paths specified in the outs section of the dvc.yaml file:

src/data_handling/etl_pipeline.py:

import os
import argparse

import src.data_handling.scripts as scripts
from src._utils import main_logger

def etl_pipeline():
    # extract the entire data
    df = scripts.extract_original_dataframe()

    # load perquet file
    ORIGINAL_DF_PATH = os.path.join('data', 'original_df.parquet')
    df.to_parquet(ORIGINAL_DF_PATH, index=False) # dvc tracked

    # transform
    df = scripts.structure_missing_values(df=df)
    df = scripts.handle_feature_engineering(df=df)

    PROCESSED_DF_PATH = os.path.join('data', 'processed_df.parquet')
    df.to_parquet(PROCESSED_DF_PATH, index=False) # dvc tracked
    return df

# for dvc execution
if __name__ == '__main__':  
    parser = argparse.ArgumentParser(description="run etl pipeline")
    parser.add_argument('--stockcode', type=str, default='', help="specific stockcode to process. empty runs full pipeline.")
    parser.add_argument('--impute', action='store_true', help="flag to create imputation values")
    args = parser.parse_args()

    etl_pipeline(stockcode=args.stockcode, impute_stockcode=args.impute)

Outputs

The original and structured data in Pandas’ DataFrames are stored in the DVC cache:

• data/original_df.parquet

• data/processed_df.parquet

Stage 2: The Data Drift Check

Before jumping into preprocessing, we’ll run data drift tests to ensure any notable drift is in the data. To do this, we’ll use EventlyAI, an open-source ML and LLM observability framework.

What is Data Drift?

Data drift refers to any changes in the statistical properties like the mean, variance, or distribution of the data that the model is trained on.

There are three main types of data drift:

• Covariate Drift (Feature Drift): A change in the input feature distribution.

• Prior Probability Drift (Label Drift): A change in the target variable distribution.

• Concept Drift: A change in the relationship between the input data and the target variable.

Data drift compromises the model's generalization capabilities over time, making its detection after deployment crucial.

DVC Configuration

We’ll add the data_drift_check stage right after the etl_pipeline stage:

dvc.yaml:

stages:
  etl_pipeline:
    ###
  data_drift_check:
     # the main command dvc will run in this stage
    cmd: >
      python src/data_handling/report_data_drift.py
      data/processed/processed_df.csv 
      data/processed_df_${params.stockcode}.parquet
      reports/data_drift_report_${params.stockcode}.html
      metrics/data_drift_${params.stockcode}.json
      ${params.stockcode}

    # default values to the parameters (defined in the param.yaml file)
    params:
      - params.stockcode

    # dependencies necessary to run the main command
    deps:
      - src/data_handling/report_data_drift.py
      - src/

    # output file pathes for dvc to track
    plots:
      - reports/data_drift_report_${params.stockcode}.html:

    metrics:
      - metrics/data_drift_${params.stockcode}.json:
          type: json

Then, add default values to the parameters passed to the DVC command:

params.yaml:

params:
  stockcode: <STOCKCODE OF CHOICE>

Python Scripts

After generating an API token from the EventlyAI workplace, we’ll add a Python script to detect data drift and store the results in the metrics variable:

src/data_handling/report_data_drift.py:

import os
import sys
import json
import pandas as pd
import datetime
from dotenv import load_dotenv

from evidently import Dataset, DataDefinition, Report
from evidently.presets import DataDriftPreset
from evidently.ui.workspace import CloudWorkspace

import src.data_handling.scripts as scripts
from src._utils import main_logger


if __name__ == '__main__':
    # initiate evently cloud workspace
    load_dotenv(override=True)
    ws = CloudWorkspace(token=os.getenv('EVENTLY_API_TOKEN'), url='https://app.evidently.cloud')

    # retrieve evently project
    project = ws.get_project('EVENTLY AI PROJECT ID')

    # retrieve paths from the command line args
    REFERENCE_DATA_PATH = sys.argv[1]
    CURRENT_DATA_PATH = sys.argv[2]
    REPORT_OUTPUT_PATH = sys.argv[3]
    METRICS_OUTPUT_PATH = sys.argv[4]
    STOCKCODE = sys.argv[5]

    # create folders if not exist
    os.makedirs(os.path.dirname(REPORT_OUTPUT_PATH), exist_ok=True)
    os.makedirs(os.path.dirname(METRICS_OUTPUT_PATH), exist_ok=True)

    # extract datasets
    reference_data_full = pd.read_csv(REFERENCE_DATA_PATH)
    reference_data_stockcode = reference_data_full[reference_data_full['stockcode'] == STOCKCODE]
    current_data_stockcode = pd.read_parquet(CURRENT_DATA_PATH)

    # define data schema
    nums, cats = scripts.categorize_num_cat_cols(df=reference_data_stockcode)
    for col in nums: current_data_stockcode[col] = pd.to_numeric(current_data_stockcode[col], errors='coerce')

    schema = DataDefinition(numerical_columns=nums, categorical_columns=cats)

    # define evently dataset w/ the data schema
    eval_data_1 = Dataset.from_pandas(reference_data_stockcode, data_definition=schema)
    eval_data_2 = Dataset.from_pandas(current_data_stockcode, data_definition=schema)

    # execute drift detection
    report = Report(metrics=[DataDriftPreset()])
    data_eval = report.run(reference_data=eval_data_1, current_data=eval_data_2)
    data_eval.save_html(REPORT_OUTPUT_PATH)

    # create metrics for dvc tracking
    report_dict = json.loads(data_eval.json())
    num_drifts = report_dict['metrics'][0]['value']['count']
    shared_drifts = report_dict['metrics'][0]['value']['share']
    metrics = dict(
        drift_detected=bool(num_drifts > 0.0), num_drifts=num_drifts, shared_drifts=shared_drifts,
        num_cols=nums,
        cat_cols=cats,
        stockcode=STOCKCODE,
        timestamp=datetime.datetime.now().isoformat(),
    )

    # load metrics file
    with open(METRICS_OUTPUT_PATH, 'w') as f:
        json.dump(metrics, f, indent=4)
        main_logger.info(f'... drift metrics saved to {METRICS_OUTPUT_PATH}... ')

    # stop the system if data drift is found
    if num_drifts > 0.0: sys.exit('❌ FATAL: data drift detected. stopping pipeline')

If data drift is found, the script immediately exits using the final sys.exit command.

Outputs

The script generates two files that DVC will track:

• reports/data_drift_report.html: The data drift report in a HTML file.

• metrics/data_drift.json: The data drift metics in a JSON file including drift results along with feature columns and a timestamp:

metrics/data_drift.json:

{
    "drift_detected": false,
    "num_drifts": 0.0,
    "shared_drifts": 0.0,
    "num_cols": [
        "invoiceno",
        "invoicedate",
        "unitprice",
        "product_avg_quantity_last_month",
        "product_max_price_all_time",
        "unitprice_vs_max",
        "unitprice_to_avg",
        "unitprice_squared",
        "unitprice_log"
    ],
    "cat_cols": [
        "stockcode",
        "customerid",
        "country",
        "year",
        "year_month",
        "day_of_week",
        "is_registered"
    ],
    "timestamp": "2025-10-07T00:24:29.899495"
}

The drift test results are also available on the Evently workplace dashboard for further analysis:

Figure B. Screenshot of the Evently workspace dashboard

Stage 3: Preprocessing

If no data drift is detected, the linage moves onto the preprocessing stage.

DVC Configuration

We’ll add the preprocess stage right after the data_drift_check stage:

dvc.yaml:

stages:
  etl_pipeline:
    ###
  data_drift_check:
    ### 
  preprocess:
    cmd: >
      python src/data_handling/preprocess.py --target_col ${params.target_col} --should_scale ${params.should_scale} --verbose ${params.verbose}

    deps:
      - src/data_handling/preprocess.py
      - src/data_handling/
      - src/_utils

    # params from params.yaml
    params:
      - params.target_col
      - params.should_scale
      - params.verbose

    outs:
      # train, val, test datasets
      - data/x_train_df.parquet
      - data/x_val_df.parquet
      - data/x_test_df.parquet
      - data/y_train_df.parquet
      - data/y_val_df.parquet
      - data/y_test_df.parquet

      # preprocessed input datasets
      - data/x_train_processed.parquet
      - data/x_val_processed.parquet
      - data/x_test_processed.parquet

      # trained preprocessor and human readable feature names for shap analysis
      - preprocessors/column_transformer.pkl
      - preprocessors/feature_names.json

And then add default values of the parameters used in the cmd:

params.yaml:

params:
  target_col: "quantity"
  should_scale: True
  verbose: False

Python Scripts

Next, we’ll add a Python script to create training, validation, and test datasets and preprocess input data:

import os
import argparse
import json
import joblib
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

import src.data_handling.scripts as scripts
from src._utils import main_logger

def preprocess(stockcode: str = '', target_col: str = 'quantity', should_scale: bool = True, verbose: bool = False):
    # initiate metrics to track (dvc)
    DATA_DRIFT_METRICS_PATH = os.path.join('metrics', f'data_drift_{args.stockcode}.json')

    if os.path.exists(DATA_DRIFT_METRICS_PATH):
        with open(DATA_DRIFT_METRICS_PATH, 'r') as f:
            metrics = json.load(f)
    else: metrics = dict()

    # load processed df from dvc cache
    PROCESSED_DF_PATH = os.path.join('data', 'processed_df.parquet')
    df = pd.read_parquet(PROCESSED_DF_PATH)

    # categorize num and cat columns
    num_cols, cat_cols = scripts.categorize_num_cat_cols(df=df, target_col=target_col)
    if verbose: main_logger.info(f'num_cols: {num_cols} \ncat_cols: {cat_cols}')

    # structure cat cols
    if cat_cols:
        for col in cat_cols: df[col] = df[col].astype('string')

    # initiate preprocessor (either load from the dvc cache or create from scratch)
    PREPROCESSOR_PATH = os.path.join('preprocessors', 'column_transformer.pkl')
    try:
        preprocessor = joblib.load(PREPROCESSOR_PATH)
    except:
        preprocessor = scripts.create_preprocessor(num_cols=num_cols if should_scale else [], cat_cols=cat_cols)

    # creates train, val, test datasets
    y = df[target_col]
    X = df.copy().drop(target_col, axis='columns')

    # split
    test_size, random_state = 50000, 42
    X_tv, X_test, y_tv, y_test = train_test_split(X, y, test_size=test_size, random_state=random_state, shuffle=False)
    X_train, X_val, y_train, y_val = train_test_split(X_tv, y_tv, test_size=test_size, random_state=random_state, shuffle=False)

    # store train, val, test datasets (dvc track)
    X_train.to_parquet('data/x_train_df.parquet', index=False)
    X_val.to_parquet('data/x_val_df.parquet', index=False)
    X_test.to_parquet('data/x_test_df.parquet', index=False)
    y_train.to_frame(name=target_col).to_parquet('data/y_train_df.parquet', index=False)
    y_val.to_frame(name=target_col).to_parquet('data/y_val_df.parquet', index=False)
    y_test.to_frame(name=target_col).to_parquet('data/y_test_df.parquet', index=False)

    # preprocess
    X_train = preprocessor.fit_transform(X_train)
    X_val = preprocessor.transform(X_val)
    X_test = preprocessor.transform(X_test)

    # store preprocessed input data (dvc track)
    pd.DataFrame(X_train).to_parquet(f'data/x_train_processed.parquet', index=False)
    pd.DataFrame(X_val).to_parquet(f'data/x_val_processed.parquet', index=False)
    pd.DataFrame(X_test).to_parquet(f'data/x_test_processed.parquet', index=False)

    # save feature names (dvc track) for shap
    with open('preprocessors/feature_names.json', 'w') as f:
        feature_names = preprocessor.get_feature_names_out()
        json.dump(feature_names.tolist(), f)

    return  X_train, X_val, X_test, y_train, y_val, y_test, preprocessor


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='run data preprocessing')
    parser.add_argument('--stockcode', type=str, default='', help='specific stockcode')
    parser.add_argument('--target_col', type=str, default='quantity', help='the target column name')
    parser.add_argument('--should_scale', type=bool, default=True, help='flag to scale numerical features')
    parser.add_argument('--verbose', type=bool, default=False, help='flag for verbose logging')
    args = parser.parse_args()

    X_train, X_val, X_test, y_train, y_val, y_test, preprocessor = preprocess(
        target_col=args.target_col,
        should_scale=args.should_scale,
        verbose=args.verbose,
        stockcode=args.stockcode,
    )

Outputs

This stage generates the necessary datasets for both model training and inference:

Input features:

• data/x_train_df.parquet

• data/x_val_df.parquet

• data/x_test_df.parquet

Preprocessed input features:

• data/x_train_processed_df.parquet

• data/x_val_processed_df.parquet

• data/x_test_processed_df.parquet

Target variables:

• data/y_train_df.parquet

• data/y_val_df.parquet

• data/y_test_df.parquet

The preprocessor and human-readable feature names are also stored in cache for inference and SHAP feature impact analysis later:

• preprocessors/column_transformer.pk

• preprocessors/feature_names.json

Lastly, DVC adds the preprocess_status , x_train_processed_path, and preprocessor_path to the data summary metrics file data.json created in Step 2 to track the end-to-end process of Steps 2 and 3:

metrics/data.json:

{
    "drift_detected": false,
    "num_drifts": 0.0,
    "shared_drifts": 0.0,
    "num_cols": [
        "invoiceno",
        "invoicedate",
        "unitprice",
        "product_avg_quantity_last_month",
        "product_max_price_all_time",
        "unitprice_vs_max",
        "unitprice_to_avg",
        "unitprice_squared",
        "unitprice_log"
    ],
    "cat_cols": [
        "stockcode",
        "customerid",
        "country",
        "year",
        "year_month",
        "day_of_week",
        "is_registered"
    ],
    "timestamp": "2025-10-07T00:24:29.899495",

    # updates
    "preprocess_status": "completed",
    "x_train_processed_path": "data/x_train_processed_85123A.parquet",
    "preprocessor_path": "preprocessors/column_transformer.pkl"
}

Next, let’s move onto the model/experiment lineage.

Stage 4: Tuning the Model

Now that we’ve created the datasets, we’ll tune and train the primary model. It’s a multi-layered feedforward network on PyTorch, using training and validation datasets created in the preprocess stage.

DVC Configuration

First, we’ll add the tuning_primary_model stage right after the preprocess stage:

dvc.yaml:

stages:
  etl_pipeline:
    ###
  data_drift_check:
    ### 
  preprocess:
    ### 
  tune_primary_model:
    cmd: >
      python src/model/torch_model/main.py
      data/x_train_processed_${params.stockcode}.parquet
      data/x_val_processed_${params.stockcode}.parquet
      data/y_train_df_${params.stockcode}.parquet
      data/y_val_df_${params.stockcode}.parquet
      ${tuning.should_local_save}
      ${tuning.grid}
      ${tuning.n_trials}
      ${tuning.num_epochs}
      ${params.stockcode}

    deps:
      - src/model/torch_model/main.py
      - src/data_handling/
      - src/model/
      - src/_utils/

    params:
      - params.stockcode
      - tuning.n_trials
      - tuning.grid
      - tuning.should_local_save

    outs:
      - models/production/dfn_best_${params.stockcode}.pth # dvc track

    metrics:
      - metrics/dfn_val_${params.stockcode}.json: # dvc track

Then we’ll add default values to the parameters:

params.yaml:

params:
  target_col: "quantity"
  should_scale: True
  verbose: False

tuning:
  n_trials: 100
  num_epochs: 3000
  should_local_save: False
  grid: False

Python Scripts

Next, we’ll add the Python scripts to tune the model using Bayesian optimization and then train the optimal model on the complete X_train and y_train datasets created in the preprocess stage.

src/model/torch_model/main.py:

import os
import sys
import json
import datetime
import pandas as pd
import torch
import torch.nn as nn

import src.model.torch_model.scripts as scripts


def tune_and_train(
        X_train, X_val, y_train, y_val,
        stockcode: str = '',
        should_local_save: bool = True,
        grid: bool = False,
        n_trials: int = 50,
        num_epochs: int = 3000
    ) -> tuple[nn.Module, dict]:

    # perform bayesian optimization
    best_dfn, best_optimizer, best_batch_size, best_checkpoint = scripts.bayesian_optimization(
        X_train, X_val, y_train, y_val, n_trials=n_trials, num_epochs=num_epochs
    )

    # save the model artifact (dvc track)
    DFN_FILE_PATH = os.path.join('models', 'production', f'dfn_best_{stockcode}.pth' if stockcode else 'dfn_best.pth')
    os.makedirs(os.path.dirname(DFN_FILE_PATH), exist_ok=True)
    torch.save(best_checkpoint, DFN_FILE_PATH)

    return best_dfn, best_checkpoint



def track_metrics_by_stockcode(X_val, y_val, best_model, checkpoint: dict, stockcode: str):
    MODEL_VAL_METRICS_PATH = os.path.join('metrics', f'dfn_val_{stockcode}.json')
    os.makedirs(os.path.dirname(MODEL_VAL_METRICS_PATH), exist_ok=True)

    # validate the tuned model
    _, mse, exp_mae, rmsle = scripts.perform_inference(model=best_model, X=X_val, y=y_val)
    model_version = f"dfn_{stockcode}_{os.getpid()}"
    metrics = dict(
        stockcode=stockcode,
        mse_val=mse,
        mae_val=exp_mae,
        rmsle_val=rmsle,
        model_version=model_version,
        hparams=checkpoint['hparams'],
        optimizer=checkpoint['optimizer_name'],
        batch_size=checkpoint['batch_size'],
        lr=checkpoint['lr'],
        timestamp=datetime.datetime.now().isoformat()
    )
    # store the validation results (dvc track)
    with open(MODEL_VAL_METRICS_PATH, 'w') as f:
        json.dump(metrics, f, indent=4)
        main_logger.info(f'... validation metrics saved to {MODEL_VAL_METRICS_PATH} ...')


if __name__ == '__main__':
    # fetch command arg values
    X_TRAIN_PATH = sys.argv[1]
    X_VAL_PATH = sys.argv[2]
    Y_TRAIN_PATH = sys.argv[3]
    Y_VAL_PATH = sys.argv[4]
    SHOULD_LOCAL_SAVE = sys.argv[5] == 'True'
    GRID = sys.argv[6] == 'True'
    N_TRIALS = int(sys.argv[7])
    NUM_EPOCHS = int(sys.argv[8])
    STOCKCODE = str(sys.argv[9])

    # extract training and validation datasets from dvc cache
    X_train, X_val = pd.read_parquet(X_TRAIN_PATH), pd.read_parquet(X_VAL_PATH)
    y_train, y_val = pd.read_parquet(Y_TRAIN_PATH), pd.read_parquet(Y_VAL_PATH)

    # tuning
    best_model, checkpoint = tune_and_train(
        X_train, X_val, y_train, y_val,
        stockcode=STOCKCODE, should_local_save=SHOULD_LOCAL_SAVE, grid=GRID, n_trials=N_TRIALS, num_epochs=NUM_EPOCHS
    )

    # metrics tracking
    track_metrics_by_stockcode(X_val, y_val, best_model=best_model, checkpoint=checkpoint, stockcode=STOCKCODE)

Outputs

The stage generates two files:

• models/production/dfn_best.pth: Includes model artifacts and checkpoint like the optimal hyperparameter set.

• metrics/dfn_val.json: Contains tuning results, model version, timestamp, and validation results for MSE, MAE, and RMSLE:

metrics/dfn_val.json:

{
    "stockcode": "85123A",
    "mse_val": 0.6137686967849731,
    "mae_val": 9.092489242553711,
    "rmsle_val": 0.6953299045562744,
    "model_version": "dfn_85123A_35604",
    "hparams": {
        "num_layers": 4,
        "batch_norm": false,
        "dropout_rate_layer_0": 0.13765888061300502,
        "n_units_layer_0": 184,
        "dropout_rate_layer_1": 0.5509872409359128,
        "n_units_layer_1": 122,
        "dropout_rate_layer_2": 0.2408753527744403,
        "n_units_layer_2": 35,
        "dropout_rate_layer_3": 0.03451842588822594,
        "n_units_layer_3": 224,
        "learning_rate": 0.026240673135104406,
        "optimizer": "adamax",
        "batch_size": 64
    },
    "optimizer": "adamax",
    "batch_size": 64,
    "lr": 0.026240673135104406,
    "timestamp": "2025-10-07T00:31:08.700294"
}

Stage 5: Performing Inference

After the model tuning phase is complete, we’ll configure the test inference for a final evaluation.

The final evaluation uses the MSE, MAE, and RMSLE metrics, as well as SHAP for feature impact and interpretability analysis.

SHAP (SHapley Additive exPlanations) is a framework for quantifying how much each feature contributes to a model’s prediction by using the concept of Shapley values from game theory.

The SHAP values are leveraged for future EDA and feature engineering.

DVC Configuration

First, we’ll add the inference_primary_model stage to the DVC configuration.

This stage has the plots section where DVC will track and version the generated visualization files on the SHAP values.

dvc.yaml:

stages:
  etl_pipeline:
    ###
  data_drift_check:
    ### 
  preprocess:
    ### 
  tune_primary_model:
    ### 
  inference_primary_model:
    cmd: >
      python src/model/torch_model/inference.py
      data/x_test_processed_${params.stockcode}.parquet
      data/y_test_df_${params.stockcode}.parquet
      models/production/dfn_best_${params.stockcode}.pth
      ${params.stockcode}
      ${tracking.sensitive_feature_col}
      ${tracking.privileged_group}

    deps:
      - src/model/torch_model/inference.py
      - models/production/
      - src/

    params:
      - params.stockcode
      - tracking.sensitive_feature_col
      - tracking.privileged_group

    metrics:
      - metrics/dfn_inf_${params.stockcode}.json: # dvc track
          type: json

    plots:
      # shap summary / beeswarm plot for global interpretability
      - reports/dfn_shap_summary_${params.stockcode}.json:
          template: simple
          x: shap_value
          y: feature_name
          title: SHAP Beeswarm Plot

      # shap mean absolute vals - feature importance bar plot
      - reports/dfn_shap_mean_abs_${params.stockcode}.json:
          template: bar
          x: mean_abs_shap
          y: feature_name
          title: Mean Absolute SHAP Importance

    outs:
      - data/dfn_inference_results_${params.stockcode}.parquet
      - reports/dfn_raw_shap_values_${params.stockcode}.parquet # save raw shap vals for detailed analysis later

Python Scripts

Next, we’ll add scripts where the trained model performs inference:

src/model/torch_model/inference.py:

import os
import sys
import json
import datetime
import numpy as np
import pandas as pd
import torch
import shap

import src.model.torch_model.scripts as scripts
from src._utils import main_logger


if __name__ == '__main__':
    # load test dataset
    X_TEST_PATH = sys.argv[1]
    Y_TEST_PATH = sys.argv[2]
    X_test, y_test = pd.read_parquet(X_TEST_PATH), pd.read_parquet(Y_TEST_PATH)

    # create X_test w/ column names for shap analysis and sensitive feature tracking
    X_test_with_col_names = X_test.copy()
    FEATURE_NAMES_PATH = os.path.join('preprocessors', 'feature_names.json')
    try:
        with open(FEATURE_NAMES_PATH, 'r') as f: feature_names = json.load(f)
    except FileNotFoundError: feature_names = X_test.columns.tolist()
    if len(X_test_with_col_names.columns) == len(feature_names): X_test_with_col_names.columns = feature_names

    # reconstruct the optimal model tuned in the previous stage
    MODEL_PATH = sys.argv[3]
    checkpoint = torch.load(MODEL_PATH)
    model = scripts.load_model(checkpoint=checkpoint)

    # perform inference
    y_pred, mse, exp_mae, rmsle = scripts.perform_inference(model=model, X=X_test, y=y_test, batch_size=checkpoint['batch_size'])

    # create result df w/ y_pred, y_true, and sensitive features
    STOCKCODE = sys.argv[4]
    SENSITIVE_FEATURE = sys.argv[5]
    PRIVILEGED_GROUP = sys.argv[6]
    inference_df = pd.DataFrame(y_pred.cpu().numpy().flatten(), columns=['y_pred'])
    inference_df['y_true'] = y_test
    inference_df[SENSITIVE_FEATURE] = X_test_with_col_names[f'cat__{SENSITIVE_FEATURE}_{str(PRIVILEGED_GROUP)}'].astype(bool)
    inference_df.to_parquet(path=os.path.join('data', f'dfn_inference_results_{STOCKCODE}.parquet'))

    # record inference metrics
    MODEL_INF_METRICS_PATH = os.path.join('metrics', f'dfn_inf_{STOCKCODE}.json')
    os.makedirs(os.path.dirname(MODEL_INF_METRICS_PATH), exist_ok=True)
    model_version = f"dfn_{STOCKCODE}_{os.getpid()}"
    inf_metrics = dict(
        stockcode=STOCKCODE,
        mse_inf=mse,
        mae_inf=exp_mae,
        rmsle_inf=rmsle,
        model_version=model_version,
        hparams=checkpoint['hparams'],
        optimizer=checkpoint['optimizer_name'],
        batch_size=checkpoint['batch_size'],
        lr=checkpoint['lr'],
        timestamp=datetime.datetime.now().isoformat()
    )
    with open(MODEL_INF_METRICS_PATH, 'w') as f: # dvc track
        json.dump(inf_metrics, f, indent=4)
        main_logger.info(f'... inference metrics saved to {MODEL_INF_METRICS_PATH} ...')


    ## shap analysis
    # compute shap vals
    model.eval()

    # prepare backgdound data
    X_test_tensor = torch.from_numpy(X_test.values.astype(np.float32)).to(device_type)

    # take the small samples from x_test as background
    background = X_test_tensor[np.random.choice(X_test_tensor.shape[0], 100, replace=False)].to(device_type)

    # define deepexplainer
    explainer = shap.DeepExplainer(model, background)

    # compute shap vals
    shap_values = explainer.shap_values(X_test_tensor) # outputs = numpy array or tensor

    # convert shap array to pandas df
    if isinstance(shap_values, list): shap_values = shap_values[0]
    if isinstance(shap_values, torch.Tensor): shap_values = shap_values.cpu().numpy()
    shap_values = shap_values.squeeze(axis=-1) # type: ignore
    shap_df = pd.DataFrame(shap_values, columns=feature_names)

    # shap raw data (dvc track)
    RAW_SHAP_OUT_PATH = os.path.join('reports', f'dfn_raw_shap_values_{STOCKCODE}.parquet')
    os.makedirs(os.path.dirname(RAW_SHAP_OUT_PATH), exist_ok=True)
    shap_df.to_parquet(RAW_SHAP_OUT_PATH, index=False)
    main_logger.info(f'... shap values saved to {RAW_SHAP_OUT_PATH} ...')

    # bar plot of mean abs shap vals (dvc report)
    mean_abs_shap = shap_df.abs().mean().sort_values(ascending=False)
    shap_mean_abs_df = pd.DataFrame({'feature_name': feature_names, 'mean_abs_shap': mean_abs_shap.values })
    MEAN_ABS_SHAP_PATH = os.path.join('reports', f'dfn_shap_mean_abs_{STOCKCODE}.json')
    shap_mean_abs_df.to_json(MEAN_ABS_SHAP_PATH, orient='records', indent=4)

Outputs

This stage generates five output files:

• data/dfn_inference_result_${params_stockcode}.parquet: Stores prediction results, labeled targets, and any columns with sensitive features like gender, age, income, and more. I’ll use this file for the fairness test in the last stage.

• metrics/dfn_inf.json: Stores evaluation metrics and tuning results:

{
    "stockcode": "85123A",
    "mse_inf": 0.6841545701026917,
    "mae_inf": 11.5866117477417,
    "rmsle_inf": 0.7423332333564758,
    "model_version": "dfn_85123A_35834",
    "hparams": {
        "num_layers": 4,
        "batch_norm": false,
        "dropout_rate_layer_0": 0.13765888061300502,
        "n_units_layer_0": 184,
        "dropout_rate_layer_1": 0.5509872409359128,
        "n_units_layer_1": 122,
        "dropout_rate_layer_2": 0.2408753527744403,
        "n_units_layer_2": 35,
        "dropout_rate_layer_3": 0.03451842588822594,
        "n_units_layer_3": 224,
        "learning_rate": 0.026240673135104406,
        "optimizer": "adamax",
        "batch_size": 64
    },
    "optimizer": "adamax",
    "batch_size": 64,
    "lr": 0.026240673135104406,
    "timestamp": "2025-10-07T00:31:12.946405"
}

• reports/dfn_shap_mean_abs.json: Stores the mean SHAP values:

[
    {
        "feature_name":"num__invoicedate",
        "mean_abs_shap":0.219255722
    },
    {
        "feature_name":"num__unitprice",
        "mean_abs_shap":0.1069829418
    },
    {
        "feature_name":"num__product_avg_quantity_last_month",
        "mean_abs_shap":0.1021453096
    },
    {
        "feature_name":"num__product_max_price_all_time",
        "mean_abs_shap":0.0855356899
    },
...
]

• reports/dfn_shap_summary.json: Contains the data points necessary to draw the beeswarm/bar plots.

• reports/dfn_raw_shap_values.parquet: Stores raw SHAP values.

Stage 6: Assessing Model Risk and Fairness

The last stage is to assess risk and fairness of the final inference results.

The Fairness Testing

Fairness testing in ML is the process of systematically evaluating a model’s predictions to ensure they are not unfairly biased toward specific groups defined by sensitive attributes like race and gender.

In this project, we’ll use the registration status is_registered column as a sensitive feature and make sure the Mean Outcome Difference (MOD) is within the specified threshold of 0.1.

The MOD is calculated as the absolute difference between the mean prediction values of the privileged (registered) and unprivileged (unregistered) groups.

DVC Configuration

First, we’ll add the assess_model_risk stage right after the inference_primary_model stage:

dvc.yaml:

stages:
  etl_pipeline:
    ###
  data_drift_check:
    ### 
  preprocess:
    ### 
  tune_primary_model:
    ### 
  inference_primary_model:
    ###
  assess_model_risk:
    cmd: >
      python src/model/torch_model/assess_risk_and_fairness.py
      data/dfn_inference_results_${params.stockcode}.parquet
      metrics/dfn_risk_fairness_${params.stockcode}.json
      ${tracking.sensitive_feature_col}
      ${params.stockcode}
      ${tracking.privileged_group}
      ${tracking.mod_threshold}

    deps:
      - src/model/torch_model/assess_risk_and_fairness.py
      - src/_utils/
      - data/dfn_inference_results_${params.stockcode}.parquet # ensure the result df as dependency

    params:
      - params.stockcode
      - tracking.sensitive_feature_col
      - tracking.privileged_group
      - tracking.mod_threshold

    metrics:
      - metrics/dfn_risk_fairness_${params.stockcode}.json:
          type: json

Then we’ll add default values to the parameters:

param.yaml:

params:
  target_col: "quantity"
  should_scale: True
  verbose: False

tuning:
  n_trials: 100
  num_epochs: 3000
  should_local_save: False
  grid: False

# adding default values to the tracking metrics
tracking:
  sensitive_feature_col: "is_registered"
  privileged_group: 1 # member
  mod_threshold: 0.1

Python Script

The corresponding Python script contains the calculate_fairness_metrics function which performs the risk and fairness assessment:

src/model/torch_model/assess_risk_and_fairness.py:

import os
import json
import datetime
import argparse
import pandas as pd
from sklearn.metrics import mean_absolute_error, mean_squared_error, root_mean_squared_log_error

from src._utils import main_logger


def calculate_fairness_metrics(
        df: pd.DataFrame,
        sensitive_feature_col: str,
        label_col: str = 'y_true',
        prediction_col: str = 'y_pred',
        privileged_group: int = 1,
        mod_threshold: float = 0.1,
    ) -> dict:

    metrics = dict()
    unprivileged_group = 0 if privileged_group == 1 else 1

    ## 1. risk assessment - predictive performance metrics by group
    for group, name in zip([unprivileged_group, privileged_group], ['unprivileged', 'privileged']):
        subset = df[df[sensitive_feature_col] == group]
        if len(subset) == 0: continue

        y_true = subset[label_col].values
        y_pred = subset[prediction_col].values

        metrics[f'mse_{name}'] = float(mean_squared_error(y_true, y_pred)) # type: ignore
        metrics[f'mae_{name}'] = float(mean_absolute_error(y_true, y_pred)) # type: ignore
        metrics[f'rmsle_{name}'] = float(root_mean_squared_log_error(y_true, y_pred)) # type: ignore

        # mean prediction (outcome disparity component)
        metrics[f'mean_prediction_{name}'] = float(y_pred.mean()) # type: ignore

    ## 2. bias assessment - fairness metrics
    # absolute mean error difference
    mae_diff = metrics.get('mae_unprivileged', 0) - metrics.get('mae_privileged', 0)
    metrics['mae_diff'] = float(mae_diff)

    # mean outcome difference
    mod = metrics.get('mean_prediction_unprivileged', 0) - metrics.get('mean_prediction_privileged', 0)
    metrics['mean_outcome_difference'] = float(mod)
    metrics['is_mod_acceptable'] = 1 if abs(mod) <= mod_threshold else 0

    return metrics


def main():
    parser = argparse.ArgumentParser(description='assess bias and fairness metrics on model inference results.')
    parser.add_argument('inference_file_path', type=str, help='parquet file path to the inference results w/ y_true, y_pred, and sensitive feature cols.')
    parser.add_argument('metrics_output_path', type=str, help='json file path to save the metrics output.')
    parser.add_argument('sensitive_feature_col', type=str, help='column name of sensitive features')
    parser.add_argument('stockcode', type=str)
    parser.add_argument('privileged_group', type=int, default=1)
    parser.add_argument('mod_threshold', type=float, default=.1)
    args = parser.parse_args()

    try:
        # load inf df
        df_inference = pd.read_parquet(args.inference_file_path)
        LABEL_COL = 'y_true'
        PREDICTION_COL = 'y_pred'
        SENSITIVE_COL = args.sensitive_feature_col

        # compute fairness metrics
        metrics = calculate_fairness_metrics(
            df=df_inference,
            sensitive_feature_col=SENSITIVE_COL,
            label_col=LABEL_COL,
            prediction_col=PREDICTION_COL,
            privileged_group=args.privileged_group,
            mod_threshold=args.mod_threshold,
        )

        # add items to metrics
        metrics['model_version'] = f'dfn_{args.stockcode}_{os.getpid()}'
        metrics['sensitive_feature'] = args.sensitive_feature_col
        metrics['privileged_group'] = args.privileged_group
        metrics['mod_threshold'] = args.mod_threshold
        metrics['stockcode'] = args.stockcode
        metrics['timestamp'] = datetime.datetime.now().isoformat()

        # load metrics (dvc track)
        with open(args.metrics_output_path, 'w') as f:
            json_metrics = { k: (v if pd.notna(v) else None) for k, v in metrics.items() }
            json.dump(json_metrics, f, indent=4)

    except Exception as e:
        main_logger.error(f'... an error occurred during risk and fairness assessment: {e} ...')
        exit(1)

if __name__ == '__main__':
    main()

Outputs

The final stage generates a metrics file which contains test results and model version:

metrics/dfn_risk_fairness.json:

{
    "mse_unprivileged": 3.5370739412593575,
    "mae_unprivileged": 1.48263614013523,
    "rmsle_unprivileged": 0.6080000224747837,
    "mean_prediction_unprivileged": 1.8507767915725708,
    "mae_diff": 1.48263614013523,
    "mean_outcome_difference": 1.8507767915725708,
    "is_mod_acceptable": 1,
    "model_version": "dfn_85123A_35971",
    "sensitive_feature": "is_registered",
    "privileged_group": 1,
    "mod_threshold": 0.1,
    "timestamp": "2025-10-07T00:31:15.998590"
}

That’s all for the lineage configuration. Now, we’ll test it in local.

Test in Local

We’ll run the entire ML lineage with this command:

$dvc repro -f

-f forces DVC to rerun all the stages with or without any updates.

The command will automatically create the dvc.lock file at the root of the project directory:

schema: '2.0'
stages:
  etl_pipeline_full:
    cmd: python src/data_handling/etl_pipeline.py
    deps:
    - path: src/_utils/
      hash: md5
      md5: ae41392532188d290395495f6827ed00.dir
      size: 15870
      nfiles: 10
    - path: src/data_handling/
      hash: md5
      md5: a8a61a4b270581a7c387d51e416f4e86.dir
      size: 95715
...

The dvc.lock file must be published in Git to make sure DVC will load the latest files:

$git add dvc.lock .dvc dvc.yaml params.yaml
$git commit -m'updated dvc config'
$git push

Step 3: Deploying the DVC Project

Next, we’ll deploy the DVC project to ensure the AWS Lambda function can access the cached files in production.

We’ll start by configuring the DVC remote where the cached files are stored.

DVC offers various storage types like AWS S3 and Google Cloud. We’ll use AWS S3 for this project but your choice depend on the project ecosystem, your familiarity with the tool, and any resource constraints.

First, we’ll create a new S3 bucket in the selected AWS region:

$aws s3 mb s3://<PROJECT NAME>/<BUCKET NAME>  --region <AWS REGION>

Make sure the IAM role has the following permissions: s3:ListBucket, s3:GetObject, s3:PutObject, and s3:DeleteObject.

Then, add theURI of the S3 bucket to the DVC remote:

$dvc remote add -d <DVC REMOTE NAME> ss3://<PROJECT NAME>/<BUCKET NAME>

Next, push the cache files to the DVC remote:

$dvc push

Now, all cache files are stored in the S3 bucket:

Figure C. Screenshot of the DVC remote in AWS S3 bucket

As shown in Figure A, this deployment step is necessary for the AWS Lambda function to access the DVC cache in production.

Step 4: Configuring Scheduled Run with Prefect

The next step is to configure the scheduled run of the entire lineage with Prefect.

Prefect is an open-source workflow orchestration tool for building, scheduling, and monitoring pipelines. It uses a concept called a work pool to effectively decouple the orchestration logic from the execution infrastructure.

Then, the work pool serves as a standardized base configuration by running a Docker container image to guarantee a consistent execution environment for all flows.

Configuring the Docker Image Registry

The first step is to configure the Docker image registry for the Prefect work pool:

• For local deployment: A container registry in the Docker Hub.

• For production deployment: AWS ECR.

For local deployment, we’ll first authenticate the Docker client:

$docker login

And grant a user permission to run Docker commands without sudo:

$sudo dscl . -append /Groups/docker GroupMembership $USER

For production deployment, we’ll create a new ECR:

$aws ecr create-repository --repository-name <REGISTORY NAME> --region <AWS REGION>

(Make sure the IAM role has access to this new ECR URI.)

Configure Prefect Tasks and Flows

Next, we’ll configure the Prefect task and flow in the project:

• The Prefect task executes the dvc repro and dvc push commands

• The Prefect flow weekly executes the Prefect task.

src/prefect_flows.py:

import os
import sys
import subprocess
from datetime import timedelta, datetime
from dotenv import load_dotenv
from prefect import flow, task
from prefect.schedules import Schedule
from prefect_aws import AwsCredentials

from src._utils import main_logger

# add project root to the python path - enabling prefect to find the script
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))

# define the prefect task
@task(retries=3, retry_delay_seconds=30)
def run_dvc_pipeline():
    # execute the dvc pipeline 
    result = subprocess.run(["dvc", "repro"], capture_output=True, text=True, check=True)

    # push the updated data
    subprocess.run(["dvc", "push"], check=True)


# define the prefect flow
@flow(name="Weekly Data Pipeline")
def weekly_data_flow():
    run_dvc_pipeline()

if __name__ == '__main__':
    # docker image registry (either docker hub or aws ecr)
    load_dotenv(override=True)
    ENV = os.getenv('ENV', 'production')
    DOCKER_HUB_REPO = os.getenv('DOCKER_HUB_REPO')
    ECR_FOR_PREFECT_PATH = os.getenv('S3_BUCKET_FOR_PREFECT_PATH')
    image_repo = f'{DOCKER_HUB_REPO}:ml-sales-pred-data-latest' if ENV == 'local' else f'{ECR_FOR_PREFECT_PATH}:latest'

    # define weekly schedule
    weekly_schedule = Schedule(
        interval=timedelta(weeks=1),
        anchor_date=datetime(2025, 9, 29, 9, 0, 0),
        active=True,
    )

    # aws credentials to access ecr
    AwsCredentials(
        aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),
        aws_secret_access_key=os.getenv('AWS_SECRET_ACCESS_KEY'),
        region_name=os.getenv('AWS_REGION_NAME'),
    ).save('aws', overwrite=True)

    # deploy the prefect flow
    weekly_data_flow.deploy(
        name='weekly-data-flow',
        schedule=weekly_schedule, # schedule
        work_pool_name="wp-ml-sales-pred", # work pool where the docker image (flow) runs
        image=image_repo, # create a docker image at docker hub (local) or ecr (production)
        concurrency_limit=3,
        push=True # push the docker image to the image_repo
    )

Test in Local

Next, we’ll test the workflow locally with the Prefect server:

$uv run prefect server start

$export PREFECT_API_URL="http://127.0.0.1:4200/api"

Run the prefect_flows.py script:

$uv run src/prefect_flows.py

Upon the successful execution, the Prefect dashboard indicates the workflow is scheduled to run:

Figure D. As screenshot of the Prefect dashboard

Step 5: Deploying the Application

The final step is to deploy the entire application as a containerized Lambda by configuring the Dockerfile and the Flask application scripts.

The specific process in this final deployment step depends on the infrastructure.

But the common point is that DVC eliminates the need to store the large Parquet or CSV files directly in the feature store or model store because it caches them as lightweight hashed files.

So, first, we’ll simplify the loading logic of the Flask application script by using the dvc.api framework:

app.py:

### ... the rest components remain the same  ...

import dvc.api

DVC_REMOTE_NAME=<REMOTE NAME IN .dvc/config file>


def configure_dvc_for_lambda():
    # set dvc directories to /tmp
    os.environ.update({
        'DVC_CACHE_DIR': '/tmp/dvc-cache',
        'DVC_DATA_DIR': '/tmp/dvc-data',
        'DVC_CONFIG_DIR': '/tmp/dvc-config',
        'DVC_GLOBAL_CONFIG_DIR': '/tmp/dvc-global-config',
        'DVC_SITE_CACHE_DIR': '/tmp/dvc-site-cache'
    })
    for dir_path in ['/tmp/dvc-cache', '/tmp/dvc-data', '/tmp/dvc-config']:
        os.makedirs(dir_path, exist_ok=True)


def load_x_test():
    global X_test
    if not os.environ.get('PYTEST_RUN', False):
        main_logger.info("... loading x_test ...")

        # config dvc directories
        configure_dvc_for_lambda()
        try:
            with dvc.api.open(X_TEST_PATH, remote=DVC_REMOTE_NAME, mode='rb') as fd:
                X_test = pd.read_parquet(fd)
                main_logger.info('✅ successfully loaded x_test via dvc api')
        except Exception as e:
            main_logger.error(f'❌ general loading error: {e}', exc_info=True)


def load_preprocessor():
    global preprocessor
    if not os.environ.get('PYTEST_RUN', False):
        main_logger.info("... loading preprocessor ...")
        configure_dvc_for_lambda()
        try:
            with dvc.api.open(PREPROCESSOR_PATH, remote=DVC_REMOTE_NAME, mode='rb') as fd:
                preprocessor = joblib.load(fd)
                main_logger.info('✅ successfully loaded preprocessor via dvc api')

        except Exception as e:
            main_logger.error(f'❌ general loading error: {e}', exc_info=True)

### ... the rest components remain the same  ...

Then, update the Dockerfile to enable Docker to correctly reference the DVC components:

Dockerfile.lambda.production:

# use an official python runtime
FROM public.ecr.aws/lambda/python:3.12

# set environment variables (adding dvc related env variables)
ENV JOBLIB_MULTIPROCESSING=0
ENV DVC_HOME="/tmp/.dvc"
ENV DVC_CACHE_DIR="/tmp/.dvc/cache"
ENV DVC_REMOTE_NAME="storage"
ENV DVC_GLOBAL_SITE_CACHE_DIR="/tmp/dvc_global"

# copy requirements file and install dependencies
COPY requirements.txt ${LAMBDA_TASK_ROOT}
RUN python -m pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt
RUN pip install --no-cache-dir dvc dvc-s3

# setup dvc
RUN dvc init --no-scm
RUN dvc config core.no_scm true

# copy the code to the lambda task root
COPY . ${LAMBDA_TASK_ROOT}
CMD [ "app.handler" ]

Lastly, ensure the large files are ignored from the Docker container image:

.dockerignore:

### ... the rest components remain the same  ...

# dvc cache contains large files
.dvc/cache
.dvcignore

# add all folders that DVC will track
data/
preprocessors/
models/
reports/
metrics/

Test in Local

Finally, we’ll build and test the Docker image:

$docker build -t my-app -f Dockerfile.lambda.local .
$docker run -p 5002:5002 -e ENV=local my-app app.py

Upon the successful configuration, the waitress server will run the Flask application.

After confirming the changes, push the code to Git:

$git add .
$git commit -m'updated dockerfiles and flask app scripts'
$git push

This push command triggers the CI/CD pipeline via GitHub Actions, which generates a Docker container image and pushes it to AWS ECR.

And then after a successful pipeline flow and verification, we can manually run the deployment workflow using GitHub Actions.

And that’s it!

You can learn more here: Integrating the infrastructure CI/CD pipeline to an ML application

All code is available in my GitHub repository.

The mock app is available here.

Conclusion

Building robust ML applications requires comprehensive ML lineage to ensure reliability and traceability.

In this article, you learned how to build an ML lineage by integrating open-source services like DVC and Prefect.

In practice, initial planning matters. Specifically, defining how metrics are tracked and at which stages leads directly to a cleaner, more maintainable code structure and the extensibility in the future.

Moving forward, we can consider adding more stages to the lineage and integrating advanced logic for data drift detection or fairness tests.

This will further ensure continued model performance and data integrity in the production environment.

You can check out my Portfolio / Github.

All images, unless otherwise noted, are by the author.

This is where Transformers come in. Originally built for language, they’ve become state-of-the-art in vision tasks thanks to models like the Vision Transformer (ViT) and video-focused variants such as TimeSformer.

In this tutorial, we’ll use a Transformer backbone to create a lightweight real-time gesture recognition tool, optimized for small datasets and deployable on a regular laptop webcam.

Table of Contents

• Why Transformers for Gestures?

• What You’ll Learn

• Prerequisites

• Project Setup

• Generate a Gesture Dataset

• Option 1: Generate a Synthetic Dataset

• Training Script: train.py

• Export the Model to ONNX

• Evaluate Accuracy + Latency

• Option 2: Use Small Samples from Public Gesture Datasets

• Accessibility Notes & Ethical Limits

• Next Steps

• Conclusion

Why Transformers for Gestures?

Transformers are powerful because they use self-attention to model relationships across a sequence. For gestures, this means the model doesn’t just see isolated frames, but also learns how movements evolve over time. A wave, for example, looks different from a raised hand only when viewed as a sequence.

Vision Transformers process images as patches, while video Transformers extend this to multiple frames with temporal attention. Even a simple approach, like applying ViT to each frame and pooling across time, can outperform traditional CNN-based methods for small datasets.

Combined with Hugging Face’s pre-trained models and ONNX Runtime for optimization, Transformers make it possible to train on a modest dataset and still achieve smooth real-time recognition.

What You’ll Learn

In this tutorial, you’ll build a gesture recognition system using Transformers. By the end, you’ll know how to:

• Create (or record) a tiny gesture dataset

• Train a Vision Transformer (ViT) with temporal pooling

• Export the model to ONNX for faster inference

• Build a real-time Gradio app that classifies gestures from your webcam

• Evaluate your model’s accuracy and latency with simple scripts

• Understand the accessibility potential and ethical limits of gesture recognition

Prerequisites

To follow along, you should have:

• Basic Python knowledge (functions, scripts, virtual environments)

• Familiarity with PyTorch (tensors, datasets, training loops) – helpful but not required

• Python 3.8+ installed on your system

• A webcam (for the live demo in Gradio)

• Optionally: GPU access (training on CPU works, but is slower)

Project Setup

Create a new project folder and install the required libraries.

# Create a new project directory and navigate into it
mkdir transformer-gesture && cd transformer-gesture

# Set up a Python virtual environment
python -m venv .venv

# Activate the virtual environment
# Windows PowerShell
.venv\Scripts\Activate.ps1

# macOS/Linux
source .venv/bin/activate

The provided code snippet is a set of commands for setting up a new Python project with a virtual environment. Here's a breakdown of each part:

1. mkdir transformer-gesture && cd transformer-gesture: This command creates a new directory named "transformer-gesture" and then navigates into it.

2. python -m venv .venv: This command creates a new virtual environment in the current directory. The virtual environment is stored in a folder named ".venv".

3. Activating the virtual environment:

   • For Windows PowerShell, you can use .venv\Scripts\Activate.ps1 to activate the virtual environment.

   • For macOS/Linux, use source .venv/bin/activate to activate the virtual environment.

Activating a virtual environment ensures that the Python interpreter and any packages you install are isolated to this specific project, preventing conflicts with other projects or system-wide packages.

Create a requirements.txt file:

torch>=2.0
torchvision
torchaudio
timm
huggingface_hub

onnx
onnxruntime

gradio

numpy
opencv-python
pillow

matplotlib
seaborn
scikit-learn

The list provided is a set of package dependencies typically found in a requirements.txt file for a Python project. Here's a brief explanation of each package:

1. torch>=2.0: PyTorch is a popular open-source deep learning framework that provides a flexible and efficient platform for building and training neural networks. Version 2.0 and above includes improvements in performance and new features.

2. torchvision: This library is part of the PyTorch ecosystem and provides tools for computer vision tasks, including datasets, model architectures, and image transformations.

3. torchaudio: Also part of the PyTorch ecosystem, Torchaudio provides audio processing tools and datasets, making it easier to work with audio data in deep learning projects.

4. timm: The PyTorch Image Models (timm) library offers a collection of pre-trained models and utilities for computer vision tasks, facilitating quick experimentation and deployment.

5. huggingface_hub: This library allows easy access to models and datasets hosted on the Hugging Face Hub, a platform for sharing and collaborating on machine learning models and datasets.

6. onnx: The Open Neural Network Exchange (ONNX) format is used to represent machine learning models, enabling interoperability between different frameworks.

7. onnxruntime: This is a high-performance runtime for executing ONNX models, allowing for efficient deployment across various platforms.

8. gradio: Gradio is a library for creating user interfaces for machine learning models, making them accessible through a web interface for easy interaction and testing.

9. numpy: A fundamental package for numerical computing in Python, providing support for arrays and a wide range of mathematical functions.

10. opencv-python: OpenCV is a library for computer vision and image processing tasks, widely used for real-time applications.

11. pillow: A Python Imaging Library (PIL) fork, Pillow provides tools for opening, manipulating, and saving many different image file formats.

12. matplotlib: A plotting library for Python, Matplotlib is used for creating static, interactive, and animated visualizations in Python.

13. seaborn: Built on top of Matplotlib, Seaborn provides a high-level interface for drawing attractive and informative statistical graphics.

14. scikit-learn: A machine learning library in Python that provides simple and efficient tools for data analysis and modeling, including classification, regression, clustering, and dimensionality reduction.

Install dependencies:

pip install -r requirements.txt

The command pip install -r requirements.txt is used to install all the Python packages listed in a file named requirements.txt. This file typically contains a list of package dependencies required for a Python project, each specified with a package name and optionally a version number.

By running this command, pip, which is the Python package installer, reads the file and installs each package listed, ensuring that the project has all the necessary dependencies to run properly. This is a common practice in Python projects to manage and share dependencies easily.

Generate a Gesture Dataset

To train our Transformer-based gesture recognizer, we need some data. Instead of downloading a huge dataset, we’ll start with a tiny synthetic dataset you can generate in seconds. This makes the tutorial lightweight and ensures that everyone can follow along without dealing with multi-gigabyte downloads.

Option 1: Generate a Synthetic Dataset

We’ll use a small Python script that creates short .mp4 clips of a moving (or still) coloured box. Each class represents a gesture:

• swipe_left – box moves from right to left

• swipe_right – box moves from left to right

• stop – box stays still in the center

Save this script as generate_synthetic_gestures.py in your project root:

import os, cv2, numpy as np, random, argparse

def ensure_dir(p): os.makedirs(p, exist_ok=True)

def make_clip(mode, out_path, seconds=1.5, fps=16, size=224, box_size=60, seed=0, codec="mp4v"):
    rng = random.Random(seed)
    frames = int(seconds * fps)
    H = W = size

    # background + box color
    bg_val = rng.randint(160, 220)
    bg = np.full((H, W, 3), bg_val, dtype=np.uint8)
    color = (rng.randint(20, 80), rng.randint(20, 80), rng.randint(20, 80))

    # path of motion
    y = rng.randint(40, H - 40 - box_size)
    if mode == "swipe_left":
        x_start, x_end = W - 20 - box_size, 20
    elif mode == "swipe_right":
        x_start, x_end = 20, W - 20 - box_size
    elif mode == "stop":
        x_start = x_end = (W - box_size) // 2
    else:
        raise ValueError(f"Unknown mode: {mode}")

    fourcc = cv2.VideoWriter_fourcc(*codec)
    vw = cv2.VideoWriter(out_path, fourcc, fps, (W, H))
    if not vw.isOpened():
        raise RuntimeError(
            f"Could not open VideoWriter with codec '{codec}'. "
            "Try --codec XVID and use .avi extension, e.g. out.avi"
        )

    for t in range(frames):
        alpha = t / max(1, frames - 1)
        x = int((1 - alpha) * x_start + alpha * x_end)
        # small jitter to avoid being too synthetic
        jitter_x, jitter_y = rng.randint(-2, 2), rng.randint(-2, 2)
        frame = bg.copy()
        cv2.rectangle(frame, (x + jitter_x, y + jitter_y),
                      (x + jitter_x + box_size, y + jitter_y + box_size),
                      color, thickness=-1)
        # overlay text
        cv2.putText(frame, mode, (8, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 0), 2, cv2.LINE_AA)
        cv2.putText(frame, mode, (8, 24), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 1, cv2.LINE_AA)
        vw.write(frame)

    vw.release()

def write_labels(labels, out_dir):
    with open(os.path.join(out_dir, "labels.txt"), "w", encoding="utf-8") as f:
        for c in labels:
            f.write(c + "\n")

def main():
    ap = argparse.ArgumentParser(description="Generate a tiny synthetic gesture dataset.")
    ap.add_argument("--out", default="data", help="Output directory (default: data)")
    ap.add_argument("--classes", nargs="+",
                    default=["swipe_left", "swipe_right", "stop"],
                    help="Class names (default: swipe_left swipe_right stop)")
    ap.add_argument("--clips", type=int, default=16, help="Clips per class (default: 16)")
    ap.add_argument("--seconds", type=float, default=1.5, help="Seconds per clip (default: 1.5)")
    ap.add_argument("--fps", type=int, default=16, help="Frames per second (default: 16)")
    ap.add_argument("--size", type=int, default=224, help="Frame size WxH (default: 224)")
    ap.add_argument("--box", type=int, default=60, help="Box size (default: 60)")
    ap.add_argument("--codec", default="mp4v", help="Codec fourcc (mp4v or XVID)")
    ap.add_argument("--ext", default=".mp4", help="File extension (.mp4 or .avi)")
    args = ap.parse_args()

    ensure_dir(args.out)
    write_labels(args.classes, ".")  # writes labels.txt to project root

    print(f"Generating synthetic dataset -> {args.out}")
    for cls in args.classes:
        cls_dir = os.path.join(args.out, cls)
        ensure_dir(cls_dir)
        mode = "stop" if cls == "stop" else ("swipe_left" if "left" in cls else ("swipe_right" if "right" in cls else "stop"))
        for i in range(args.clips):
            filename = os.path.join(cls_dir, f"{cls}_{i+1:03d}{args.ext}")
            make_clip(
                mode=mode,
                out_path=filename,
                seconds=args.seconds,
                fps=args.fps,
                size=args.size,
                box_size=args.box,
                seed=i + 1,
                codec=args.codec
            )
        print(f"  {cls}: {args.clips} clips")

    print("Done. You can now run: python train.py, python export_onnx.py, python app.py")

if __name__ == "__main__":
    main()

The script generates a synthetic gesture dataset by creating video clips of a moving or stationary coloured box, simulating gestures like "swipe left," "swipe right," and "stop," and saves them in a specified output directory.

Now run it inside your virtual environment:

python generate_synthetic_gestures.py --out data --clips 16 --seconds 1.5

The command above runs a Python script named generate_synthetic_gestures.py, which generates a synthetic gesture dataset with 16 clips per gesture, each lasting 1.5 seconds, and saves the output in a directory named "data".

This creates a dataset like:

data/
  swipe_left/*.mp4
  swipe_right/*.mp4
  stop/*.mp4
labels.txt

Each folder contains short clips of a moving (or still) box that simulate gestures. This is perfect for testing the pipeline.

Training Script: train.py

Now that we have our dataset, let’s fine-tune a Vision Transformer with temporal pooling. This model applies ViT frame-by-frame, averages embeddings across time, and trains a classification head on your gestures.

Here’s the full training script:

# train.py
import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import DataLoader
import timm
from dataset import GestureClips, read_labels

class ViTTemporal(nn.Module):
    """Frame-wise ViT encoder -> mean pool over time -> linear head."""
    def __init__(self, num_classes, vit_name="vit_tiny_patch16_224"):
        super().__init__()
        self.vit = timm.create_model(vit_name, pretrained=True, num_classes=0, global_pool="avg")
        feat_dim = self.vit.num_features
        self.head = nn.Linear(feat_dim, num_classes)

    def forward(self, x):  # x: (B,T,C,H,W)
        B, T, C, H, W = x.shape
        x = x.view(B * T, C, H, W)
        feats = self.vit(x)                  # (B*T, D)
        feats = feats.view(B, T, -1).mean(dim=1)  # (B, D)
        return self.head(feats)

def train():
    device = "cuda" if torch.cuda.is_available() else "cpu"
    labels, _ = read_labels("labels.txt")
    n_classes = len(labels)

    train_ds = GestureClips(train=True)
    val_ds   = GestureClips(train=False)
    print(f"Train clips: {len(train_ds)} | Val clips: {len(val_ds)}")

    # Windows/CPU friendly
    train_dl = DataLoader(train_ds, batch_size=2, shuffle=True,  num_workers=0, pin_memory=False)
    val_dl   = DataLoader(val_ds,   batch_size=2, shuffle=False, num_workers=0, pin_memory=False)

    model = ViTTemporal(num_classes=n_classes).to(device)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05)

    best_acc = 0.0
    epochs = 5
    for epoch in range(1, epochs + 1):
        # ---- Train ----
        model.train()
        total, correct, loss_sum = 0, 0, 0.0
        for x, y in train_dl:
            x, y = x.to(device), y.to(device)
            optimizer.zero_grad()
            logits = model(x)
            loss = criterion(logits, y)
            loss.backward()
            optimizer.step()

            loss_sum += loss.item() * x.size(0)
            correct += (logits.argmax(1) == y).sum().item()
            total += x.size(0)

        train_acc = correct / total if total else 0.0
        train_loss = loss_sum / total if total else 0.0

        # ---- Validate ----
        model.eval()
        vtotal, vcorrect = 0, 0
        with torch.no_grad():
            for x, y in val_dl:
                x, y = x.to(device), y.to(device)
                vcorrect += (model(x).argmax(1) == y).sum().item()
                vtotal += x.size(0)
        val_acc = vcorrect / vtotal if vtotal else 0.0

        print(f"Epoch {epoch:02d} | train_loss {train_loss:.4f} "
              f"| train_acc {train_acc:.3f} | val_acc {val_acc:.3f}")

        if val_acc > best_acc:
            best_acc = val_acc
            torch.save(model.state_dict(), "vit_temporal_best.pt")

    print("Best val acc:", best_acc)

if __name__ == "__main__":
    train()

Running the command python train.py initiates the training process for your gesture recognition model. Here's a breakdown of what happens:

1. Load your dataset from data/: The script will access and load the gesture dataset stored in the "data" directory. This dataset is used to train the model.

2. Fine-tune a pre-trained Vision Transformer: The training script will take a Vision Transformer model that has been pre-trained on a larger dataset and fine-tune it using your specific gesture dataset. Fine-tuning helps the model adapt to the nuances of your data, improving its performance on the specific task of gesture recognition.

3. Save the best checkpoint as vit_temporal_best.pt: During training, the script will evaluate the model's performance on a validation set. The best-performing version of the model (based on some metric like accuracy) will be saved as a checkpoint file named "vit_temporal_best.pt". This file can later be used for inference or further training.

What Training Looks Like

You should see logs similar to this:

Train clips: 38 | Val clips: 10
Epoch 01 | train_loss 1.4508 | train_acc 0.395 | val_acc 0.200
Epoch 02 | train_loss 1.2466 | train_acc 0.263 | val_acc 0.200
Epoch 03 | train_loss 1.1361 | train_acc 0.368 | val_acc 0.200
Best val acc: 0.200

Don’t worry if your accuracy is low at first, as with the synthetic dataset that’s normal. The key is proving that the Transformer pipeline works. You can boost results later by:

• Adding more clips per class

• Training for more epochs

• Switching to real recorded gestures

Figure 1. Example training logs from train.py, where the Vision Transformer with temporal pooling is fine-tuned on a tiny synthetic dataset.

Export the Model to ONNX

To make our model easier to run in real time (and lighter on CPU), we’ll export it to the ONNX format.

Note: ONNX, which stands for Open Neural Network Exchange, is an open-source format designed to facilitate the interchange of deep learning models between different frameworks. It lets you train a model in one framework, such as PyTorch or TensorFlow, and then deploy it in another, like Caffe2 or MXNet, without needing to completely rewrite the model. This interoperability is achieved by providing a standardized representation of the model's architecture and parameters.

ONNX supports a wide range of operators and is continually updated to include new features, making it a versatile choice for deploying machine learning models across various platforms and devices.

Create a file called export_onnx.py:

import torch
from train import ViTTemporal
from dataset import read_labels

labels, _ = read_labels("labels.txt")
n_classes = len(labels)

# Load trained model
model = ViTTemporal(num_classes=n_classes)
model.load_state_dict(torch.load("vit_temporal_best.pt", map_location="cpu"))
model.eval()

# Dummy input: batch=1, 16 frames, 3x224x224
dummy = torch.randn(1, 16, 3, 224, 224)

# Export
torch.onnx.export(
    model, dummy, "vit_temporal.onnx",
    input_names=["video"], output_names=["logits"],
    dynamic_axes={"video": {0: "batch"}},
    opset_version=13
)

print("Exported vit_temporal.onnx")

Run it with python export_onnx.py.

This generates a file vit_temporal.onnx in your project folder. ONNX lets us use onnxruntime, which is much faster for inference.

Create a file called app.py:

import os, tempfile, cv2, torch, onnxruntime, numpy as np
import gradio as gr
from dataset import read_labels

T = 16
SIZE = 224
MODEL_PATH = "vit_temporal.onnx"

labels, _ = read_labels("labels.txt")

# --- ONNX session + auto-detect names ---
ort_session = onnxruntime.InferenceSession(MODEL_PATH, providers=["CPUExecutionProvider"])
# detect first input and first output names to avoid mismatches
INPUT_NAME = ort_session.get_inputs()[0].name   # e.g. "input" or "video"
OUTPUT_NAME = ort_session.get_outputs()[0].name # e.g. "logits" or something else

def preprocess_clip(frames_rgb):
    if len(frames_rgb) == 0:
        frames_rgb = [np.zeros((SIZE, SIZE, 3), dtype=np.uint8)]
    if len(frames_rgb) < T:
        frames_rgb = frames_rgb + [frames_rgb[-1]] * (T - len(frames_rgb))
    frames_rgb = frames_rgb[:T]
    clip = [cv2.resize(f, (SIZE, SIZE), interpolation=cv2.INTER_AREA) for f in frames_rgb]
    clip = np.stack(clip, axis=0)                                    # (T,H,W,3)
    clip = np.transpose(clip, (0, 3, 1, 2)).astype(np.float32) / 255 # (T,3,H,W)
    clip = (clip - 0.5) / 0.5
    clip = np.expand_dims(clip, 0)                                   # (1,T,3,H,W)
    return clip

def _extract_path_from_gradio_video(inp):
    if isinstance(inp, str) and os.path.exists(inp):
        return inp
    if isinstance(inp, dict):
        for key in ("video", "name", "path", "filepath"):
            v = inp.get(key)
            if isinstance(v, str) and os.path.exists(v):
                return v
        for key in ("data", "video"):
            v = inp.get(key)
            if isinstance(v, (bytes, bytearray)):
                tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".mp4")
                tmp.write(v); tmp.flush(); tmp.close()
                return tmp.name
    if isinstance(inp, (list, tuple)) and inp and isinstance(inp[0], str) and os.path.exists(inp[0]):
        return inp[0]
    return None

def _read_uniform_frames(video_path):
    cap = cv2.VideoCapture(video_path)
    frames = []
    total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 1
    idxs = np.linspace(0, total - 1, max(T, 1)).astype(int)
    want = set(int(i) for i in idxs.tolist())
    j = 0
    while True:
        ok, bgr = cap.read()
        if not ok: break
        if j in want:
            rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
            frames.append(rgb)
        j += 1
    cap.release()
    return frames

def predict_from_video(gradio_video):
    video_path = _extract_path_from_gradio_video(gradio_video)
    if not video_path or not os.path.exists(video_path):
        return {}
    frames = _read_uniform_frames(video_path)

    # If OpenCV choked on the codec (common with recorded webm), re-encode once:
    if len(frames) == 0:
        tmp = tempfile.NamedTemporaryFile(delete=False, suffix=".mp4"); tmp_name = tmp.name; tmp.close()
        cap = cv2.VideoCapture(video_path)
        fourcc = cv2.VideoWriter_fourcc(*"mp4v")
        w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) or 640
        h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) or 480
        out = cv2.VideoWriter(tmp_name, fourcc, 20.0, (w, h))
        while True:
            ok, frame = cap.read()
            if not ok: break
            out.write(frame)
        cap.release(); out.release()
        frames = _read_uniform_frames(tmp_name)

    clip = preprocess_clip(frames)
    # >>> use the detected ONNX input/output names <<<
    logits = ort_session.run([OUTPUT_NAME], {INPUT_NAME: clip})[0]  # (1, C)
    probs = torch.softmax(torch.from_numpy(logits), dim=1)[0].numpy().tolist()
    return {labels[i]: float(probs[i]) for i in range(len(labels))}

def predict_from_image(image):
    if image is None:
        return {}
    clip = preprocess_clip([image] * T)
    logits = ort_session.run([OUTPUT_NAME], {INPUT_NAME: clip})[0]
    probs = torch.softmax(torch.from_numpy(logits), dim=1)[0].numpy().tolist()
    return {labels[i]: float(probs[i]) for i in range(len(labels))}

with gr.Blocks() as demo:
    gr.Markdown("# Gesture Classifier (ONNX)\nRecord or upload a short video, then click **Classify Video**.")
    with gr.Tab("Video (record or upload)"):
        vid_in = gr.Video(label="Record from webcam or upload a short clip")
        vid_out = gr.Label(num_top_classes=3, label="Prediction")
        gr.Button("Classify Video").click(fn=predict_from_video, inputs=vid_in, outputs=vid_out)
    with gr.Tab("Single Image (fallback)"):
        img_in = gr.Image(label="Upload an image frame", type="numpy")
        img_out = gr.Label(num_top_classes=3, label="Prediction")
        gr.Button("Classify Image").click(fn=predict_from_image, inputs=img_in, outputs=img_out)

if __name__ == "__main__":
    demo.launch()

Running the command python app.py launches a Gradio application in your web browser. Here's what happens:

1. Webcam feed streams live: The application accesses your webcam to provide a live video feed. This allows you to perform gestures in front of the camera in real-time.

2. Predictions update continuously: As you perform gestures, the model processes the video frames continuously, updating its predictions in real-time.

3. Top 3 gesture classes displayed with probabilities: The application displays the top three predicted gesture classes along with their probabilities, giving you an idea of the model's confidence in its predictions.

When you open the app in your browser, you'll find two tabs. In the Video tab, you can click Record from webcam to capture a short clip of your gesture, typically lasting 2–4 seconds. After recording, click Classify Video. The model will then process the captured frames using the Transformer model and display the predicted gesture probabilities. This setup allows for interactive testing and demonstration of the gesture recognition system.

Here’s an example where I raised my hand for a stop gesture, and the model predicts “stop” as the top class:

Figure 2. The Gradio app running locally. After recording a short clip, the Transformer model predicts the gesture with class probabilities.

Evaluate Accuracy + Latency

Now that the model runs in a demo app, let’s check how well it performs. There are two sides to this:

• Accuracy: does the model predict the right gesture class?

• Latency: how fast does it respond, especially on CPU vs GPU?

1. Quick Accuracy Check

Save this as eval.py in the same folder as your other scripts:

import torch
from dataset import GestureClips, read_labels
from train import ViTTemporal

labels, _ = read_labels("labels.txt")
n_classes = len(labels)

# Load validation data
val_ds = GestureClips(train=False)
val_dl = torch.utils.data.DataLoader(val_ds, batch_size=2, shuffle=False)

# Load trained model
model = ViTTemporal(num_classes=n_classes)
model.load_state_dict(torch.load("vit_temporal_best.pt", map_location="cpu"))
model.eval()

correct, total = 0, 0
all_preds, all_labels = [], []

with torch.no_grad():
    for x, y in val_dl:
        logits = model(x)
        preds = logits.argmax(dim=1)
        correct += (preds == y).sum().item()
        total += y.size(0)
        all_preds.extend(preds.tolist())
        all_labels.extend(y.tolist())

print(f"Validation accuracy: {correct/total:.2%}")

2. Confusion Matrix

Let’s also visualize which gestures are confused. Add this snippet at the bottom of eval.py:

import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(all_labels, all_preds)

plt.figure(figsize=(6,6))
sns.heatmap(cm, annot=True, fmt="d", xticklabels=labels, yticklabels=labels, cmap="Blues")
plt.xlabel("Predicted")
plt.ylabel("True")
plt.title("Confusion Matrix")
plt.tight_layout()
plt.show()

When you run python eval.py, a heatmap like this will pop up:

Figure 3. Confusion matrix on the validation set. Correct predictions appear along the diagonal. Off-diagonal counts show gesture confusions.

3. Latency Benchmark

Finally, let’s see how fast inference runs. Save the following as benchmark.py:

import time, numpy as np, onnxruntime
from dataset import read_labels

labels, _ = read_labels("labels.txt")

ort = onnxruntime.InferenceSession("vit_temporal.onnx", providers=["CPUExecutionProvider"])
INPUT_NAME = ort.get_inputs()[0].name
OUTPUT_NAME = ort.get_outputs()[0].name

dummy = np.random.randn(1, 16, 3, 224, 224).astype(np.float32)

# Warmup
for _ in range(3):
    ort.run([OUTPUT_NAME], {INPUT_NAME: dummy})

# Benchmark
t0 = time.time()
for _ in range(50):
    ort.run([OUTPUT_NAME], {INPUT_NAME: dummy})
t1 = time.time()

print(f"Average latency: {(t1 - t0)/50:.3f} seconds per clip")

Run: python benchmark.py

On CPU, you might see ~0.05–0.15s per clip; on GPU it’s much faster.

Note: If latency is high, you can enable quantization in ONNX to shrink the model and speed up inference.

Option 2: Use Small Samples from Public Gesture Datasets

If you’d prefer to see your model trained on real gesture clips instead of synthetic moving boxes, you can grab a handful of videos from open datasets. You don’t need to download the entire dataset (which can be several GB) just a few .mp4 samples are enough to follow along.

Recommended sources

• 20BN Jester Dataset: Contains short clips of hand gestures like swiping, clapping, and pointing.

• WLASL: A large-scale dataset of isolated sign language words.

Both projects provide small .mp4 videos you can use as realistic training examples. I’ve linked them below.

Setting up your dataset folder

Once you download a few clips, place them in the data/ folder under subfolders named after each gesture class. For example:

data/
├── swipe_left/
│   ├── clip1.mp4
│   └── clip2.mp4
├── swipe_right/
│   ├── clip1.mp4
│   └── clip2.mp4
└── stop/
    ├── clip1.mp4
    └── clip2.mp4

And update labels.txt to match the folder names:

swipe_left
swipe_right
stop

Now your dataset is ready, and the same training scripts from earlier (train.py, eval.py) will work without modification.

Why choose this option?

• Gives more realistic results than synthetic coloured boxes

• Lets you see how the model handles actual human hand movements

• It just requires a bit more effort (downloading clips, trimming them if needed)

Tip: If downloading from these datasets feels too heavy, you can also record your own short gestures using your laptop webcam. Just save them as .mp4 files and organize them in the same folder structure.

Accessibility Notes & Ethical Limits

While this project shows the technical workflow for gesture recognition with Transformers, it’s important to step back and consider the human context:

• Accessibility first: Tools like this can help students with speech or motor difficulties, but they should always be co-designed with the people who will use them. Don’t assume one-size-fits-all.

• Dataset sensitivity: Using publicly available sign or gesture datasets is fine for prototyping, but deploying such a system requires careful consideration of consent and representation.

• Error tolerance: Even small misclassifications can have big consequences in accessibility contexts (for example, confusing stop with go). Always plan for fallback options (like manual input or confirmation).

• Bias and inclusivity: Models trained on narrow datasets may fail for different skin tones, lighting conditions, or cultural gesture variations. Broad and diverse training data is essential for fairness.

In other words: this demo is a teaching scaffold, not a production-ready accessibility tool. Responsible deployment requires collaboration with educators, therapists, and end users.

Next Steps

If you’d like to push this project further, here are some directions to explore:

• Better models: Try video-focused Transformers like TimeSformer or VideoMAE for stronger temporal reasoning.

• Larger vocabularies: Add more gesture classes, build your own dataset, or use portions of public datasets like 20BN Jester or WLASL.

• Pose fusion: Combine gesture video with human pose keypoints from MediaPipe or OpenPose for more robust predictions.

• Real-time smoothing: Implement temporal smoothing or debounce logic in the app so predictions are more stable during live use.

• Quantization + edge devices: Convert your ONNX model to an INT8 quantized version and deploy it on a Raspberry Pi or Jetson Nano for classroom-ready prototypes.

Conclusion

In this tutorial, you learned how to create a gesture recognition system using Transformer models, demonstrating the potential of cutting-edge machine learning techniques. By preparing a small dataset, training a Vision Transformer with temporal pooling, exporting the model to ONNX for efficient inference, and deploying a real-time Gradio app, you showcased a practical application of these technologies. The evaluation of accuracy and latency further highlighted the system's effectiveness and responsiveness.

This project illustrates how you can leverage advanced ML methods to enhance accessibility and communication, paving the way for more inclusive learning environments.

Remember: while this demo works with small datasets, real-world applications need larger, more diverse data and careful consideration of accessibility, inclusivity, and ethics.

Here’s the GitHub repo for full source code: transformer-gesture.

Before I began learning about MLOps, I tried to understand the technologies behind LLMs and how they work. In this article, I’ll share my understanding of machine learning, deep learning, and generative AI, along with their potential applications.

Table of Contents

• Artificial Intelligence (AI)

• Machine Learning (ML): The Foundation

• Deep Learning: Adding Complexity

• Generative AI: Write New

• Summary of Differences Between Machine Learning vs Deep Learning vs Generative AI

• Conclusion

Artificial Intelligence (AI)

Artificial Intelligence (AI) is a form of technology that lets machines solve problems in a way that is identical to how people do it. It helps businesses make better decisions on a large scale by helping them recognize images, create content, and make predictions based on data. Artificial intelligence includes machine learning, deep learning, and generative AI.

Machine Learning (ML): The Foundation

When we give computers many examples, they learn how to make their own decisions or guesses. It's like teaching a kid to tell the difference between animals. You show them a lot of pictures of cats and dogs and say things like "This is a cat" and "This is a dog." In the end, they learn to tell the difference between cats and dogs on their own. Machine learning is similar in that you give a computer a lot of data with examples, and it learns how to make predictions about new data.

How Does Machine Learning Work?

Machine Learning (ML) is the process of teaching computers to find patterns in data and make decisions or predictions without being instructed what to do. There are usually six main steps in this process:

Data Collection: Get many examples, like thousands of emails, photos, or sales records. The more training data you have, the more accurate your predictions will be.

Data Preparation: At this stage, you clean the data by getting rid of mistakes and adding missing labels.

Selecting Algorithm (Models): It's like choosing the right tools for the job. Models can find patterns in data or make predictions. You can find machine learning models for your data here.

Training Phase: After you pick the right model for your cleaned-up data, you teach it. This is like getting ready for a test.

Evaluation: Use the test data to assess the model's performance and see if it can make accurate predictions on unseen data.

Deployment: Put the trained model to work in the real world.

Training Phase: Teach the computer with 10,000 house sales with details like size (2,000 sq ft), number of bedrooms (3), and location (downtown). Cost: $300,000.

Learning: The algorithm finds patterns, such as the fact that bigger houses cost more and places in the city center cost more. More bedrooms make a house worth more.

Prediction: Think about a new house with 1,800 square feet, two bedrooms, and a location in the suburbs. It guesses a figure based on what it has learned.

Types of Machine Learning

1. Supervised Learning: Give algorithms labeled and defined training data to look for patterns. The sample data tells the algorithm what to do and what to expect as an output. For instance, millions of X-ray reports that say someone is healthy or sick would need to be tagged. Then, machine learning programs could use this training data to guess if a new X-ray shows signs of illness.

2. Unsupervised Learning: Algorithms that use unsupervised learning learn from data that doesn't have labels. The algorithm must find patterns in untagged data without outside help. For instance, finding groups of people on Facebook or Twitter who have similar interests.

3. Reinforcement Learning: This technique is a kind of machine learning in which an agent learns how to make choices by interacting with the world around it. The agent receives points for doing things right and loses points for doing things wrong. Its goal is to get as many points as possible. For instance, cars learn how to drive safely by making mistakes in simulations. They get rewards for staying in their lane, following traffic rules, and not hitting other cars.

Machine Learning—Real-World Examples

Email Spam Detection

You can show the computer thousands of emails that say "spam" or "not spam." It learns patterns, like how emails with "FREE MONEY" are usually spam. It can now automatically sort your inbox.

Photo Recognition

Give the computer millions of pictures with labels that say what's in them. It learns that apples are likely to be round and have stems. Your phone can now tell what things are in your pictures.

Movie Recommendations

Netflix keeps track of the movies you've seen and rated. It finds people who like the same things you do. It suggests movies that other people like.

Deep Learning: Adding Complexity

Deep learning is a type of artificial intelligence. It helps computers understand data like humans do. Deep learning can identify complex images, text, sound, and other data patterns to make accurate predictions. It uses artificial neural networks that work like the human brain. Neural networks are connected nodes that handle information.

How Does Deep Learning Work?

Artificial neural networks are used in deep learning to learn from data. These networks consist of interconnected layers of nodes. Each node learns a different thing about the data.

For instance, when you show a computer a picture of a cat, the picture goes through a lot of steps. The first layer looks for shapes and edges. The second layer puts these shapes together to make ears, eyes, and whiskers. The last layers say things like "This picture looks like a cat." Deep learning can make a lot of mistakes when learning, but it gets better and better after each piece of feedback.

Deep Learning—Real-World Examples

• Tesla Autopilot: Processes eight cameras simultaneously to navigate roads, recognize traffic signs, and avoid obstacles.

• Google's DeepMind: Detects over fifty eye diseases from retinal scans with 94% accuracy.

• ChatGPT: Helps with writing, coding, and problem-solving.

Generative AI: Write New

Generative AI is a subset of deep learning that makes new things, like stories, pictures, music, or code, instead of just looking at or sorting through things that are already there. Generative AI systems learn patterns from a lot of training data and then use those patterns to make new content.

Real-World Examples

• Chatbots help institutions give better customer service by making product suggestions and answering questions.

• Automatically generate technical documents from the source code.

• Auto-generate quizzes, practice problems, and explanations

Summary of Differences Between Machine Learning vs Deep Learning vs Generative AI

Conclusion

To sum it up, anyone who is keen to learn more about artificial intelligence needs to know the differences between machine learning, deep learning, and generative AI.

Machine learning is the basis for this because it lets computers learn from data and make predictions. Deep learning takes this a step further by using neural networks to process complicated data patterns in a way that is similar to how humans understand things.

Generative AI goes a step further by making new things, which shows how creative AI can be. As these technologies get better, they open up a lot of new opportunities in many fields, such as improving customer service, making medical diagnoses more accurate, and making new content. To maximize AI's benefits in your life, stay current on new developments.

But a model isn’t truly valuable until it’s in production, serving real users and solving real problems.

In this article, you’ll learn how to ship a production-ready ML application built on serverless architecture.

Table of Contents

• Prerequisites

• What We’re Building

  • AI Pricing for Retailers

  • The Models

  • Tuning and Training

  • The Prediction

  • Performance Validation

• The System Architecture

  • Core AWS Resources in the Architecture

• The Deployment Workflow in Action

  • Step 1: Draft Python Scripts

  • Step 2: Configure Feature/Model Stores in S3

  • Step 3: Create a Flask Application with API Endpoints

  • Step 4: Publish a Docker Image to ECR

  • Step 5: Create a Lambda Function

  • Step 6: Configure AWS Resources

• Building a Client Application (Optional)

  • The React Application

• Final Results

• Conclusion

Prerequisites

This project requires some basic experience with:

• Machine Learning / Deep Learning: The full lifecycle, including data handling, model training, tuning, and validation.

• Coding: Proficiency in Python, with experience using major ML libraries such as PyTorch and Scikit-Learn.

• Full-stack deployment: Experience deploying applications using RESTful APIs.

What We’re Building

AI Pricing for Retailers

This project aims to help a middle-sized retailer compete with large players like Amazon.

Smaller companies often can’t afford significant price discounts, so they can face challenges finding optimal price points as they expand their product lines.

Our goal is to leverage AI models to recommend the best price for a selected product to maximize sales for the retailer, and display it on a client-side user interface (UI):

You can explore the UI from here.

The Models

I’ll train and tune multiple models so that when the primary model fails, a backup model gets loaded to serve predictions.

• Primary Model: Multi-layered feedforward network (on the PyTorch library)

• Backup Models (Backups): LightGBM, SVR, and Elastic Net (on the Scikit-Learn library)

The backup models are prioritized based on learning capabilities.

Tuning and Training

The primary model was trained on a dataset of around 500,000 samples (source) and fine-tuned using Optuna's Bayesian Optimization, with grid search available for further refinement.

The backups are also trained on the same samples and tuned using the Scikit-Optimize framework.

The Prediction

All models serve predictions on logged quantity values.

Logarithmic transformations of the quantity data make the distribution denser, which helps models learn patterns more effectively. This is because logarithms reduce the impact of extreme values, or outliers, and can help normalize skewed data.

Performance Validation

We’ll evaluate model performance using different metrics for the transformed and original data, with a lower value always indicating better performance.

• Logged values: Mean Squared Error (MSE)

• Actual values: Root Mean Squared Log Error (RMSLE) and Mean Absolute Error (MAE)

The System Architecture

We’re going to build a complete ecosystem around an AWS Lambda function to create a scalable ML system:

Fig. The system architecture (Created by Kuriko IWAI)

AWS Lambda is a serverless production where a service provider can run the application without managing servers. Once they upload the code, AWS takes on the responsibility of managing the underlying infrastructure.

In the serverless production, the code is deployed as a stateless function that runs only when it’s triggered by an event like HTTP requests or scheduled tasks.

This event-driven nature makes serverless production extremely efficient in resource allocation because:

• There’s no server management: The cloud provider takes care of operational tasks.

• You have automatic scaling: Serverless applications automatically scale up or down based on demand.

• You have pay-per-use billing: Charged for the exact amount of compute resources the application consumes.

Note that other cloud ecosystems like Google Cloud Platform (GCP) and Microsoft Azure offer comprehensive alternatives to AWS. Which one you choose depends on your budget, project type, and familiarity with each ecosystem.

Core AWS Resources in the Architecture

The system architecture focuses on the following points:

• The application is fully containerized on Docker for universal accessibility.

• The container image is stored in AWS Elastic Container Registry (ECR).

• The API Gateway’s REST API endpoints trigger an event to invoke the Lambda function.

• The Lambda function loads the container image from ECR and perform inference.

• Trained models, processors, and input features are stored in AWS S3 buckets.

• A Redis client serves cached analytical data and past predictions stored in the ElastiCache.

And to build the system, we’ll use the following AWS resources:

• Lamda: Serves a function to perform inference.

• API Gateway: Routes API calls to the Lambda function.

• S3 Storage: Serves feature store and model store.

• ElastiCache: Store cached predictions and analytical data.

• ECR: Stores Docker container images to allow Lambda to pull the image.

Each resource requires configuration. I’ll explore those details in the next section.

The Deployment Workflow in Action

The deployment workflow involves the following steps:

1. Draft data preparation, model training, and serialization scripts

2. Configure designated feature store and model store in S3

3. Create a Flask application with API endpoints

4. Publish a Docker image to ECR

5. Create a Lambda function

6. Configure related AWS resources

We’ll now walk through each of these steps to help you fully understand the process.

For your reference, here is the repository structure:

.
.venv/                  [.gitignore]    # stores uv venv
│
└── data/               [.gitignore]
│     └──raw/                           # stores raw data
│     └──preprocessed/                  # stores processed data after imputation and engineering
│
└── models/             [.gitignore]    # stores serialized model after training and tuning
│     └──dfn/                           # deep feedforward network
│     └──gbm/                           # light gbm
│     └──en/                            # elastic net
│     └──production/                    # models to be stored in S3 for production use
|
└── notebooks/                          # stores experimentation notebooks
│
└── src/                                # core functions
│     └──_utils/                        # utility functions
│     └──data_handling/                 # functions to engineer features
│     └──model/                         # functions to train, tune, validate models
│     │     └── sklearn_model
│     │     └── torch_model
│     │     └── ...
│     └──main.py                        # main script to run the inference locally
│
└──app.py                               # Flask application (API endpoints)
└──pyproject.toml                       # project configuration
└──.env                [.gitignore]     # environment variables
└──uv.lock                              # dependency locking
└──Dockerfile                           # for Docker container image
└──.dockerignore
└──requirements.txt
└──.python-version                      # python version locking (3.12)

Step 1: Draft Python Scripts

The first step is to draft Python scripts for data preparation, model training and tuning.

We’ll run these scripts in a batch process because these are resource-intensive and stateful tasks that aren’t suitable for serverless functions optimized for short-lived, stateless, and event-driven tasks.

Serverless functions also can experience cold starts. With heavy tasks in the function, the API gateway would timeout before serving predictions.

src/main.py

import os
import torch
import warnings
import pickle
import joblib
import numpy as np
import lightgbm as lgb
from sklearn.linear_model import ElasticNet
from sklearn.svm import SVR
from skopt.space import Real, Integer, Categorical
from dotenv import load_dotenv

import src.data_handling as data_handling
import src.model.torch_model as t
import src.model.sklearn_model as sk


if __name__ == '__main__': 
    load_dotenv(override=True)
    os.makedirs(PRODUCTION_MODEL_FOLDER_PATH, exist_ok=True)

    # create train, validation, test datasets
    X_train, X_val, X_test, y_train, y_val, y_test, preprocessor = data_handling.main_script()

    # store the trained preprocessor in local storage
    joblib.dump(preprocessor, PREPROCESSOR_PATH)

    # model tuning and training
    best_dfn_full_trained, checkpoint = t.main_script(X_train, X_val, y_train, y_val)

    # serialize the trained model
    torch.save(checkpoint, DFN_FILE_PATH)

    # svr
    best_svr_trained, best_hparams_svr = sk.main_script(
        X_train, X_val, y_train, y_val, **sklearn_models[1]
    )
    if best_svr_trained is not None:
        with open(SVR_FILE_PATH, 'wb') as f:
            pickle.dump({ 'best_model': best_svr_trained, 'best_hparams': best_hparams_svr }, f)

    # elastic net
    best_en_trained, best_hparams_en = sk.main_script(
        X_train, X_val, y_train, y_val, **sklearn_models[0]
    )
    if best_en_trained is not None:
        with open(EN_FILE_PATH, 'wb') as f:
            pickle.dump({ 'best_model': best_en_trained, 'best_hparams': best_hparams_en }, f)

    # light gbm
    best_gbm_trained, best_hparams_gbm = sk.main_script(
        X_train, X_val, y_train, y_val, **sklearn_models[2]
    )

    if best_gbm_trained is not None:
        with open(GBM_FILE_PATH, 'wb') as f:
            pickle.dump({'best_model': best_gbm_trained, 'best_hparams': best_hparams_gbm }, f)

Run the script to train and serialize the models using the uv package management:

$uv venv
$source .venv/bin/activate
$uv run src/main.py

The main.py script includes several key components.

Scripts for Data Handling

These scripts involve loading original data, structure missing values, and engineer features necessary for the future prediction.

src/data_handling/main.py

import os
import joblib
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

import src.data_handling.scripts as scripts
from src._utils import main_logger


# load and save the original data frame in parquet
df = scripts.load_original_dataframe()
df.to_parquet(ORIGINAL_DF_PATH, index=False)

# imputation
df = scripts.structure_missing_values(df=df)

# feature engineering
df = scripts.handle_feature_engineering(df=df)

# save processed df in csv and parquet
scripts.save_df_to_csv(df=df)
df.to_parquet(PROCESSED_DF_PATH, index=False)


# for preprocessing, classify numerical and categorical columns
num_cols, cat_cols = scripts.categorize_num_cat_cols(df=df, target_col=target_col)
if cat_cols:
    for col in cat_cols: df[col] = df[col].astype('string')

# creates training, validation, and test datasets (test dataset is for inference only)
y = df[target_col]
X = df.copy().drop(target_col, axis='columns')
test_size, random_state = 50000, 42
X_tv, X_test, y_tv, y_test = train_test_split(
    X, y, test_size=test_size, random_state=random_state
)
X_train, X_val, y_train, y_val = train_test_split(
    X_tv, y_tv, test_size=test_size, random_state=random_state
)

# transform the input datasets
X_train, X_val, X_test, preprocessor = scripts.transform_input(
    X_train, X_val, X_test, num_cols=num_cols, cat_cols=cat_cols
)

# retrain and serialize the preprocessor
if preprocessor is not None: preprocessor.fit(X)
joblib.dump(preprocessor, PREPROCESSOR_PATH)

Scripts for Model Training and Tuning (PyTorch Model)

The scripts involve initiating the model, searching optimal neural architecture and hyperparameters, and serializing the fully-trained model so that the system can load the trained model when performing inference.

Because the primary model is built on PyTorch and the backups use Scikit-Learn, we’re drafting the scripts separately.

1. PyTorch Models

The training script contains training the model with the validation over a subset of training data.

It contains the early stopping logic when the loss history is not improved for a given consecutive epochs (that is, 10 epochs).

src/model/torch_model/scripts/training.py

import torch
import torch.nn as nn
import optuna # type: ignore
from sklearn.model_selection import train_test_split

from src._utils import main_logger

# device
device_type = device_type if device_type else 'cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu'
device = torch.device(device_type)

# gradient scaler for stability (only applicable for cuba)
scaler = torch.GradScaler(device=device_type) if device_type == 'cuba' else None

# start training
best_val_loss = float('inf')
epochs_no_improve = 0
for epoch in range(num_epochs):
    model.train()
    for batch_X, batch_y in train_data_loader:
        batch_X, batch_y = batch_X.to(device), batch_y.to(device)
        optimizer.zero_grad()

        try:
            # pytorch's AMP system automatically handles the casting of tensors to Float16 or Float32
            with torch.autocast(device_type=device_type):
                outputs = model(batch_X)
                loss = criterion(outputs, batch_y)

                # break the training loop when models return nan or inf
                if torch.any(torch.isnan(outputs)) or torch.any(torch.isinf(outputs)):
                    main_logger.error(
                        'pytorch model returns nan or inf. break the training loop.'
                    )
                    break

            # create scaled gradients of losses
            if scaler is not None:
                scaler.scale(loss).backward()
                scaler.unscale_(optimizer)  # cliping grad
                nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
                scaler.step(optimizer)  # unscales the gradients
                scaler.update()  # updates the scale

            else:
                loss.backward()
                nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # cliping grad
                optimizer.step()

        except:
            outputs = model(batch_X)
            loss = criterion(outputs, batch_y)
            loss.backward()
            optimizer.step()


    # run validation on a subset of the training dataset
    model.eval()
    val_loss = 0.0

    # switch the torch mode
    with torch.inference_mode():
        for batch_X_val, batch_y_val in val_data_loader:
            batch_X_val, batch_y_val = batch_X_val.to(device), batch_y_val.to(device)
            outputs_val = model(batch_X_val)
            val_loss += criterion(outputs_val, batch_y_val).item()

    val_loss /= len(val_data_loader)

    # check if early stop
    if val_loss < best_val_loss - min_delta:
        best_val_loss = val_loss
        epochs_no_improve = 0
    else:
        epochs_no_improve += 1
        if epochs_no_improve >= patience:
            main_logger.info(f'early stopping at epoch {epoch + 1}')
            break

The tuning script uses the study component from the Optuna library to run the Bayesian Optimization.

The study component choose a neural architecture and hyperparameter set to test from the global search space.

Then, it builds, trains, and validates the model to find the optimal neural architecture that can minimize the loss (MSE, for instance).

src/model/torch_model/scripts/tuning.py

import itertools
import pandas as pd
import numpy as np
import optuna
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.model_selection import train_test_split

from src.model.torch_model.scripts.pretrained_base import DFN
from src.model.torch_model.scripts.training import train_model
from src._utils import main_logger

# device
device_type = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
device = torch.device(device_type)

# loss function
criterion = nn.MSELoss()

# define objective function for optuna
def objective(trial):
    # model
    num_layers = trial.suggest_int('num_layers', 1, 20)
    batch_norm = trial.suggest_categorical('batch_norm', [True, False])
    dropout_rates = []
    hidden_units_per_layer = []
    for i in range(num_layers):
        dropout_rates.append(trial.suggest_float(f'dropout_rate_layer_{i}', 0.0, 0.6))
        hidden_units_per_layer.append(trial.suggest_int(f'n_units_layer_{i}', 8, 256)) # hidden units per layer

    model = DFN(
        input_dim=X_train.shape[1],
        num_layers=num_layers,
        dropout_rates=dropout_rates,
        batch_norm=batch_norm,
        hidden_units_per_layer=hidden_units_per_layer
    ).to(device)

    # optimizer
    learning_rate = trial.suggest_float('learning_rate', 1e-10, 1e-1, log=True)
    optimizer_name = trial.suggest_categorical('optimizer', ['adam', 'rmsprop', 'sgd', 'adamw', 'adamax', 'adadelta', 'radam'])
    optimizer = _handle_optimizer(optimizer_name=optimizer_name, model=model, lr=learning_rate)

    # data loaders
    batch_size = trial.suggest_categorical('batch_size', [32, 64, 128, 256])
    test_size = 10000 if len(X_train) > 15000 else int(len(X_train) * 0.2)
    X_train_search, X_val_search, y_train_search, y_val_search = train_test_split(X_train, y_train, test_size=test_size, random_state=42)
    train_data_loader = create_torch_data_loader(X=X_train_search, y=y_train_search, batch_size=batch_size)
    val_data_loader = create_torch_data_loader(X=X_val_search, y=y_val_search, batch_size=batch_size)

    # training
    num_epochs = 3000 # ensure enough epochs (early stopping would stop the loop when overfitting)
    _, best_val_loss = train_model(
        train_data_loader=train_data_loader,
        val_data_loader=val_data_loader,
        model=model,
        optimizer=optimizer,
        criterion = criterion,
        num_epochs=num_epochs,
        trial=trial,
    )
    return best_val_loss


# start to optimize hyperparameters and architecture
study = optuna.create_study(direction='minimize', sampler=optuna.samplers.TPESampler())
study.optimize(objective, n_trials=50, timeout=600)

# best 
best_trial = study.best_trial
best_hparams = best_trial.params

# construct the model based on the tuning results
best_lr = best_hparams['learning_rate']
best_batch_size = best_hparams['batch_size']
input_dim = X_train.shape[1]
best_model = DFN(
    input_dim=input_dim,
    num_layers=best_hparams['num_layers'],
    hidden_units_per_layer=[v for k, v in best_hparams.items() if 'n_units_layer_' in k],
    batch_norm=best_hparams['batch_norm'],
    dropout_rates=[v for k, v in best_hparams.items() if 'dropout_rate_layer_' in k],
).to(device)

# construct an optimizer based on the tuning results
best_optimizer_name = best_hparams['optimizer']
best_optimizer = _handle_optimizer(
    optimizer_name=best_optimizer_name, model=best_model, lr=best_lr
)

# create torch data loaders
train_data_loader = create_torch_data_loader(
    X=X_train, y=y_train, batch_size=best_batch_size
)
val_data_loader = create_torch_data_loader(
    X=X_val, y=y_val, batch_size=best_batch_size
)

# retrain the best model with full training dataset applying the optimal batch size and optimizer
best_model, _ = train_model(
    train_data_loader=train_data_loader,
    val_data_loader=val_data_loader,
    model=best_model,
    optimizer=best_optimizer,
    criterion = criterion,
    num_epochs=1000
)

# create a checkpoint for serialization (reconstruct the model using the checkpoint)
checkpoint = {
    'state_dict': best_model.state_dict(),
    'hparams': best_hparams,
    'input_dim': X_train.shape[1],
    'optimizer': best_optimizer,
    'batch_size': best_batch_size
}

# serialize the model w/ checkpoint
torch.save(checkpoint, FILE_PATH)

2. Scikit-Learn Models (Backups)

For Scikit-Learn models, we’ll run k-fold cross validation during training to prevent overfitting.

K-fold cross-validation is a technique for evaluating a machine learning model's performance by training and testing it on different subsets of training data.

We define the run_kfold_validation function where the model is trained and validated using 5-fold cross-validation.

src/model/sklearn_model/scripts/tuning.py

from sklearn.model_selection import KFold
from sklearn.metrics import mean_squared_error

def run_kfold_validation(
        X_train,
        y_train,
        base_model,
        hparams: dict,
        n_splits: int = 5, # the number of folds 
        early_stopping_rounds: int = 10,
        max_iters: int = 200
    ) -> float:

    mses = 0.0

    # create k-fold component
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)

    for fold, (train_index, val_index) in enumerate(kf.split(X_train)):
        # create a subset of training and validation datasets from the entire training data
        X_train_fold, X_val_fold = X_train.iloc[train_index], X_train.iloc[val_index]
        y_train_fold, y_val_fold = y_train.iloc[train_index], y_train.iloc[val_index]

        # reconstruct a model
        model = base_model(**hparams)

        # start the cross validation
        best_val_mse = float('inf')
        patience_counter = 0
        best_model_state = None
        best_iteration = 0

        for iteration in range(max_iters):
            # train on a subset of the training data
            try:
                model.train_one_step(X_train_fold, y_train_fold, iteration)
            except:
                model.fit(X_train_fold, y_train_fold)

            # make a prediction on validation data 
            y_pred_val_kf = model.predict(X_val_fold)

            # compute validation loss (MSE)
            current_val_mse = mean_squared_error(y_val_fold, y_pred_val_kf)

            # check if epochs should be stopped (early stopping)
           if current_val_mse < best_val_mse:
                best_val_mse = current_val_mse
                patience_counter = 0
                best_model_state = model.get_params()
                best_iteration = iteration
           else:
                patience_counter += 1

           # execute early stopping when patience_counter exceeds early_stopping_rounds
           if patience_counter >= early_stopping_rounds:
                main_logger.info(f"Fold {fold}: Early stopping triggered at iteration {iteration} (best at {best_iteration}). Best MSE: {best_val_mse:.4f}")
                break


        # after training epochs, reconstruct the best performing model 
        if best_model_state: model.set_params(**best_model_state)

        # make prediction
        y_pred_val_kf = model.predict(X_val_fold)

        # add MSEs
        mses += mean_squared_error(y_pred_val_kf, y_val_fold)

    # compute the final loss (avarage of MSEs across folds)
    ave_mse = mses / n_splits
    return ave_mse

Then, for the tuning script, we use the gp_minimize function from the Scikit-Optimize library.

The gp_minimize function is used to tune hyperparameters with Bayesian optimization.

This function intelligently searches the best hyperparameter set that can minimize the model's error, which is calculated using the run_kfold_validation function defined earlier.

The best-performing hyperparameters are then used to reconstruct and train the final model.

src/model/sklearn_model/scripts/tuning.py

from functools import partial
from skopt import gp_minimize


# define the objective function for Bayesian Optimization using Scikit-Optimize
def objective(params, X_train, y_train, base_model, hparam_names):
    hparams = {item: params[i] for i, item in enumerate(hparam_names)}
    ave_mse = run_kfold_validation(X_train=X_train, y_train=y_train, base_model=base_model, hparams=hparams)
    return ave_mse

# create the search space
hparam_names = [s.name for s in space]
objective_partial = partial(objective, X_train=X_train, y_train=y_train, base_model=base_model, hparam_names=hparam_names)

# search the optimal hyperparameters
results = gp_minimize(
    func=objective_partial,
    dimensions=space,
    n_calls=n_calls,
    random_state=42,
    verbose=False,
    n_initial_points=10,
)
# results
best_hparams = dict(zip(hparam_names, results.x))
best_mse = results.fun

# reconstruct the model with the best hyperparameters
best_model = base_model(**best_hparams)

# retrain the model with full training dataset
best_model.fit(X_train, y_train)

Step 2: Configure Feature/Model Stores in S3

The trained models and processed data are stored in the S3 bucket as a Parquet file.

We’ll draft the s3_upload function where the Boto3 client, a low-level interface to an AWS service, initiates the connection to S3:

import os
import boto3
from dotenv import load_dotenv

from src._utils import main_logger

def s3_upload(file_path: str):
    # initiate the boto3 client
    load_dotenv(override=True)
    S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME') # the bucket created in s3
    s3_client = boto3.client('s3', region_name=os.environ.get('AWS_REGION_NAME')) # your default region

    if s3_client:
        # create s3 key and upload the file to the bucket
        s3_key = file_path if file_path[0] != '/' else file_path[1:]
        s3_client.upload_file(file_path, S3_BUCKET_NAME, s3_key)
        main_logger.info(f"file uploaded to s3://{S3_BUCKET_NAME}/{s3_key}")
    else:
        main_logger.error('failed to create an S3 client.')

Model Store

Trained PyTorch models are serialized (converted) into .pth files.

Then, these files are uploaded to the S3 bucket, enabling the system to load the trained model when it performs inference in production.

import torch

from src._utils import s3_upload

# model serialization, store in local
torch.save(trained_model.state_dict(), MODEL_FILE_PATH)

# upload to s3 model store
s3_upload(file_path=MODEL_FILE_PATH)

Feature Store

The processed data is converted into a CSV and Parquet file format.

Then, the Parquet files are uploaded to the S3 bucket, enabling the system to load the lightweight data when it creates prediction data to perform inference in production.

from src._utils import s3_upload

# store csv and parquet files in local
df.to_csv(file_path, index=False)
df.to_parquet(DATA_FILE_PATH, index=False)

# store in s3 feature store
s3_upload(file_path=DATA_FILE_PATH)

# trained preprocessor is also stored to transform the prediction data
s3_upload(file_path=PROCESSOR_PATH)

Step 3: Create a Flask Application with API Endpoints

Next, we’ll create a Flask application with API endpoints.

Flask needs to configure Python scripts in the app.py file located at the root of the project repository.

As showed in the code snippets, the app.py file needs to contain the components in order of:

1. AWS Boto3 client setup,

2. Flask app configuration and API endpoint setup,

3. Loading the trained preprocessor, processed input data X_test, and trained models,

4. Invoke the Lambda function via API Gateway, and

5. The local test section.

Note that X_test should never be used during model training to avoid data leakage.

app.py

from flask import Flask
from flask_cors import cross_origin
from waitress import serve
from dotenv import load_dotenv

from src._utils import main_logger

# global variables (will be loaded from the S3 buckets)
_redis_client = None
X_test = None
preprocessor = None
model = None
backup_model = None

# load env if local else skip (lambda refers to env in production)
AWS_LAMBDA_RUNTIME_API = os.environ.get('AWS_LAMBDA_RUNTIME_API', None)
if AWS_LAMBDA_RUNTIME_API is None: load_dotenv(override=True)


#### <---- 1. AWS BOTO3 CLIENT ---->
# boto3 client 
S3_BUCKET_NAME = os.environ.get('S3_BUCKET_NAME', 'ml-sales-pred')
s3_client = boto3.client('s3', region_name=os.environ.get('AWS_REGION_NAME', 'us-east-1'))
try:
    # test connection to boto3 client
    sts_client = boto3.client('sts')
    identity = sts_client.get_caller_identity()
    main_logger.info(f"Lambda is using role: {identity['Arn']}")
except Exception as e:
    main_logger.error(f"Lambda credentials/permissions error: {e}")

#### <---- 2. FLASK CONFIGURATION & API ENDPOINTS ---->
# configure the flask app
app = Flask(__name__)
app.config['CORS_HEADERS'] = 'Content-Type'

# add a simple API endpoint to serve the prediction by price point to test
@app.route('/v1/predict-price/<string:stockcode>', methods=['GET', 'OPTIONS'])
@cross_origin(origins=origins, methods=['GET', 'OPTIONS'], supports_credentials=True)
def predict_price(stockcode):
    df_stockcode = None

    # fetch request params
    data = request.args.to_dict()

    try:
        # fetch cache
        if _redis_client is not None:
            # returns cached prediction results if any without performing inference
            cached_prediction_result = _redis_client.get(cache_key_prediction_result_by_stockcode)
            if cached_prediction_result: 
                return jsonify(json.loads(json.dumps(cached_prediction_result)))

            # historical data of the selected product
            cached_df_stockcode = _redis_client.get(cache_key_df_stockcode)
            if cached_df_stockcode: df_stockcode = json.loads(json.dumps(cached_df_stockcode))


        # define the price range to make predictions. can be a request param, or historical min/max prices
        min_price = float(data.get('unitprice_min', df_stockcode['unitprice_min'][0]))
        max_price = float(data.get('unitprice_max', df_stockcode['unitprice_max'][0]))

        # create bins in the price range. when the number of the bins increase, the prediction becomes more smooth, but requires more computational cost
        NUM_PRICE_BINS = int(data.get('num_price_bins', 100))
        price_range = np.linspace(min_price, max_price, NUM_PRICE_BINS)

        # create a prediction dataset by merging X_test (dataset never used in model training) and df_stockcode
        price_range_df = pd.DataFrame({ 'unitprice': price_range })
        test_sample = X_test.sample(n=1000, random_state=42)
        test_sample_merged = test_sample.merge(price_range_df, how='cross') if X_test is not None else price_range_df
        test_sample_merged.drop('unitprice_x', axis=1, inplace=True)
        test_sample_merged.rename(columns={'unitprice_y': 'unitprice'}, inplace=True)

        # preprocess the dataset
        X = preprocessor.transform(test_sample_merged) if preprocessor else test_sample_merged

        # perform inference
        y_pred_actual = None
        epsilon = 0
        # try using the primary model
        if model:
            input_tensor = torch.tensor(X, dtype=torch.float32)
            model.eval()
            with torch.inference_mode():
                y_pred = model(input_tensor)
                y_pred = y_pred.cpu().numpy().flatten()
                y_pred_actual = np.exp(y_pred + epsilon)

        # if not, use backups
        elif backup_model:
            y_pred = backup_model.predict(X)
            y_pred_actual = np.exp(y_pred + epsilon)


        # finalize the outcome for client app
        df_ = test_sample_merged.copy()
        df_['quantity'] = np.floor(y_pred_actual) # quantity must be an integer
        df_['sales'] = df_['quantity'] * df_['unitprice'] # compute sales
        df_ = df_.sort_values(by='unitprice')

        # aggregate the results by the unitprice in the price range
        df_results = df_.groupby('unitprice').agg(
            quantity=('quantity', 'median'),
            quantity_min=('quantity', 'min'),
            quantity_max=('quantity', 'max'),
            sales=('sales', 'median'),
        ).reset_index()

        # find the optimal price point
        optimal_row = df_results.loc[df_results['sales'].idxmax()]
        optimal_price = optimal_row['unitprice']
        optimal_quantity = optimal_row['quantity']
        best_sales = optimal_row['sales']

        all_outputs = []
        for _, row in df_results.iterrows():
            current_output = {
                "stockcode": stockcode,
                "unit_price": float(row['unitprice']),
                'quantity': int(row['quantity']),
                'quantity_min': int(row['quantity_min']),
                'quantity_max': int(row['quantity_max']),
                "predicted_sales": float(row['sales']),
            }
            all_outputs.append(current_output)

        # store the prediction results in cache
        if all_outputs and _redis_client is not None:
             serialized_data = json.dumps(all_outputs)
            _redis_client.set(
                cache_key_prediction_result_by_stockcode, 
                serialized_data,
                ex=3600     # expire in an hour
            )

        # return a list of all outputs
        return jsonify(all_outputs)

    except Exception as e: return jsonify([])


# request header management (for the process from API gateway to the Lambda)
@app.after_request
def add_header(response):
    response.headers['Cache-Control'] = 'public, max-age=0'
    response.headers['Access-Control-Allow-Origin'] = CLIENT_A
    response.headers['Access-Control-Allow-Headers'] = 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,Origin'
    response.headers['Access-Control-Allow-Methods'] = 'GET, POST, OPTIONSS'
    response.headers['Access-Control-Allow-Credentials'] = 'true'
    return response

#### <---- 3. LOADING PROCESSOR, DATASET, AND MODELS ---->
load_processor()
load_x_test()
load_model()

#### <---- 4. INVOKE LAMBDA ---->
def handler(event, context):
    logger.info("lambda handler invoked.")
    try:
        # connecting the redis client after the lambda is invoked
        get_redis_client()
    except Exception as e:
        logger.critical(f"failed to establish initial Redis connection in handler: {e}")
        return {
            'statusCode': 500,
            'body': json.dumps({'error': 'Failed to initialize Redis client. Check environment variables and network config.'})
        }

    # use the awsgi package to convert JSON to WSGI
    return awsgi.response(app, event, context)


#### <---- 5. FOR LOCAL TEST ---->
# serve the application locally on WSGI server, waitress
# lambda will ignore this section.
if __name__ == '__main__':   
    if os.getenv('ENV') == 'local':
        main_logger.info("...start the operation (local)...")
        serve(app, host='0.0.0.0', port=5002)
    else:
        app.run(host='0.0.0.0', port=8080)

I’ll test the endpoint locally using the uv package manager:

$uv run app.py --cache-clear

$curl http://localhost:5002/v1/predict-price/{STOCKCODE}

The system provided a list of sales predictions for each price point:

Fig. Screenshot of the Flask app local response

Key Points on Flask App Configuration

There are various points you should take into consideration when configuring a Flask application with Lambda. Let’s go over them now:

1. A Few API Endpoints Per Container

Adding many API endpoints to a single serverless instance can lead to monolithic function concern where issues in one endpoint impact others.

In this project, we’ll focus on a single endpoint per container – and if needed, we can add separate Lambda functions to the system.

2. Understanding the handler Function and the role of AWSGI

The handler function is invoked every time the Lambda function receives a client request from the API Gateway.

The function takes the event argument that includes the request details in a JSON dictionary and passes it to the Flask application.

AWSGI acts as an adapter, translating a Lambda event in JSON format into a WSGI request that a Flask application can understand, and converts the application’s response back into a JSON format that Lambda and API Gateway can process.

3. Using Cache Storage

The get_redis_client function is called once the handler function is called by the API Gateway. This allows the Flask application to store or fetch a cache from the Redis client:

import redis
import redis.cluster
from redis.cluster import ClusterNode

_redis_client = None

def get_redis_client():
    global _redis_client
    if _redis_client is None:
        REDIS_HOST = os.environ.get("REDIS_HOST")
        REDIS_PORT = int(os.environ.get("REDIS_PORT", 6379))
        REDIS_TLS = os.environ.get("REDIS_TLS", "true").lower() == "true"
        try:
            startup_nodes = [ClusterNode(host=REDIS_HOST, port=REDIS_PORT)]
            _redis_client = redis.cluster.RedisCluster(
                startup_nodes=startup_nodes,
                decode_responses=True,
                skip_full_coverage_check=True,
                ssl=REDIS_TLS,                  # elasticache has encryption in transit: enabled -> must be true
                ssl_cert_reqs=None,
                socket_connect_timeout=5,
                socket_timeout=5,
                health_check_interval=30,
                retry_on_timeout=True,
                retry_on_error=[
                    redis.exceptions.ConnectionError,
                    redis.exceptions.TimeoutError
                ],
                max_connections=10,            # limit connections for Lambda
                max_connections_per_node=2     # limit per node
            )
            _redis_client.ping()
            main_logger.info("successfully connected to ElastiCache Redis Cluster (Configuration Endpoint)")
        except Exception as e:
            main_logger.error(f"an unexpected error occurred during Redis Cluster connection: {e}", exc_info=True)
            _redis_client = None
    return _redis_client

4. Handling Heavy Tasks Outside of the handler Function

Serverless functions can experience a cold start duration.

While a Lambda function can run for up to 15 minutes, its associated API Gateway has a timeout of 29 seconds (29,000 ms) for a RESTful API.

So, any heavy tasks like loading preprocessors, input data, or models should be performed once outside of the handler function, ensuring they are ready before the API endpoint is called.

Here are the loading functions called in app.py.

app.py

import joblib

from src._utils import s3_load, s3_load_to_temp_file

preprocessor = None
X_test = None
model = None
backup_model = None


# load processor
def load_preprocessor():
    global preprocessor
    preprocessor_tempfile_path = s3_load_to_temp_file(PREPROCESSOR_PATH)
    preprocessor = joblib.load(preprocessor_tempfile_path)
    os.remove(preprocessor_tempfile_path)


# load input data
def load_x_test():
    global X_test
    x_test_io = s3_load(file_path=X_TEST_PATH)
    X_test = pd.read_parquet(x_test_io)


# load model
def load_model():
    global model, backup_model
    # try loading & reconstructing the primary model
    try:
        # first load io file from the s3 bucket
        model_data_bytes_io_ = s3_load(file_path=DFN_FILE_PATH)
        # convert to checkpoint dictionary (containing hyperparameter set)
        checkpoint_ = torch.load(
            model_data_bytes_io_, 
            weights_only=False, 
            map_location=device
        )
        # reconstruct the model
        model = t.scripts.load_model(checkpoint=checkpoint_, file_path=DFN_FILE_PATH)
        # set the model evaluation mode
        model.eval()

    # else, backup model
     except:
        load_artifacts_backup_model()

Step 4: Publish a Docker Image to ECR

After configuring the Flask application, we’ll containerize the entire application on Docker.

Containerization makes a package of the application, including models, its dependencies, and configuration in machine learning context, as a container.

Docker creates a container image based on the instructions defined in a Dockerfile, and the Docker engine uses the image to run the isolated container.

In this project, we’ll upload the Docker container image to ECR, so the Lambda function can access it in production.

After this, we’ll define the .dockerignore file to optimize the container image:

.dockerignore

# any irrelevant data
__pycache__/
.ruff_cache/
.DS_Store/
.venv/
dist/
.vscode
*.psd
*.pdf
[a-f]*.log
tmp/
awscli-bundle/

# add any experimental models, unnecessary data
dfn_bayesian/
dfn_grid/
data/
notebooks/

Dockerfile

# serve from aws ecr 
FROM public.ecr.aws/lambda/python:3.12

# define a working directory in the container
WORKDIR /app

# copy the entire repository (except .dockerignore) into the container at /app
COPY . /app/

# install dependencies defined in the requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# define commands
ENTRYPOINT [ "python" ]
CMD [ "-m", "awslambdaric", "app.handler" ]

Test in Local

Next, we’ll test the Docker image by building the container named my-app locally:

$docker build -t my-app -f Dockerfile .

Then, we’ll run the container with the waitress server in local:

$docker run -p 5002:5002 -e ENV=local my-app app.py

The -e ENV=local flag sets the environment variable inside the container, which will trigger the waitress.serve() call in the app.py.

In the terminal, you’ll find a message saying the following:

You can also call the endpoint created to see the results returned:

$uv run app.py --cache-clear

$curl http://localhost:5002/v1/predict-price/{STOCKCODE}

Publish the Docker Image to ECR

To publish the Docker image, we first need to configure the default AWS credentials and region:

• From the AWS account console, issue an access token and check the default region.

• Store them in the ~/aws/credentials and ~/aws/config files:

~/aws/credentials

[default] 
aws_secret_access_key=
aws_access_key_id=

~/aws/config

[default]
region=

After the configuration, we’ll publish the Docker image to ECR.

# authenticate the docker client to ECR
$aws ecr get-login-password --region <your-aws-region> | docker login --username AWS --password-stdin <your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com

# create repository
$aws ecr create-repository --repository-name <your-repo-name> --region <your-aws-region>

# tag the docker image
$docker tag <your-repo-name>:<your-app-version>  <your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com/<your-app-name>:<your-app-version>

# push
$docker push <your-aws-account-id>.dkr.ecr.<your-aws-region>.amazonaws.com/<your-repo-name>:<your-app-version>

Here’s what’s going on:

• <your-aws-region>: Your default AWS region (for example, us-east-1 ).

• <your-aws-account-id>: 12-digit AWS account ID.

• <your-repo-name>: Your desired repository name.

• <your-app-version>: Your desired tag name (for example, v1.0).

Now, the Docker image is stored in ECR with the tag:

Fig. Screenshot of the AWS ECR console

Step 5: Create a Lambda Function

Next, we’ll create a Lambda function.

From the Lambda console, choose:

• The Container Image option,

• The container image URL from the pull down list,

• A function name of our choice, and

• An architecture type (arm64 is recommended for a better price-performance).

Fig. Screenshot of AWS Lambda function configuration

The Lambda function my-app was successfully launched.

Connect the Lambda function to API Gateway

Next, we’ll add API gateway as an event trigger to the Lambda function.

First, visit the API Gateway console and create REST API methods using the ARN of the Lambda function (press enter or click to view image in full size):

Fig. Screenshot of the AWS API Gateway configuration

Then, add resources to the created API gateway to create an endpoint:
API Gateway > APIs > Resources > Create Resource

• Align the resource endpoint with the API endpoint defined in the app.py.

• Configure CORS (for example, accept specific origins).

• Deploy the resource to the stage.

Going back to the Lambda console, you’ll find the API Gateway is connected as an event trigger:
Lambda > Function > my-app (your function name)

Fig. Screenshot of the AWS Lambda dashboard

Step 6: Configure AWS Resources

Lastly, we’ll configure the related AWS resources to make the system work in production.

This process involves the following steps:

1. The IAM Role: Controls Who to Access Resources

AWS requires IAM roles to grant temporary, secure permissions to users, mitigating security risks related to long-term credentials like passwords.

The IAM role leverages policies to grant accesses to the selected service. Policies can be issued by AWS or customized by the user by defining the inline policy.