But there was still a major limitation. Even though GPT-2 could generalize across tasks, it still struggled to adapt reliably. Performance often depended on carefully written prompts, and for many real-world applications, fine-tuning was still necessary. AI systems were becoming more flexible, but they still were not truly learning tasks from context the way humans do.

Then GPT-3 pushed the idea much further. Instead of asking whether language models could perform tasks without fine-tuning, the paper explored something even more ambitious:

What happens if we scale language models to an extreme size? The answer surprised almost everyone in the AI community.

GPT-3 showed that a sufficiently large language model could often learn new tasks directly from examples inside the prompt itself. No retraining. No gradient updates. Just a few demonstrations written in natural language.

For example, if you showed the model a few English-to-French translations, it could continue the pattern correctly for a new sentence. If you gave it examples of questions and answers, it could often infer the task immediately and generate reasonable responses.

This became known as few-shot learning and in-context learning.

More importantly, GPT-3 suggested a completely different way of interacting with AI systems. Instead of training a separate model for every task, the same model could dynamically adapt depending on the instructions and examples it received.

That idea eventually became the foundation for modern AI systems like ChatGPT.

Now, like many influential AI papers, the GPT-3 paper can be difficult to read because of its scale, technical experiments, and long benchmark evaluations. So in this article, I’ll break everything down in a clear and practical way.

We’ll explore what problem the paper was trying to solve, how few-shot learning works, why scaling became so important, how GPT-3 was trained, and why this paper fundamentally changed the direction of modern AI research.

By the end, you should understand the core ideas behind GPT-3 and why this paper became one of the most important milestones in the history of large language models LLM.

Paper Overview

In this article, we’ll review the paper Language Models are Few-Shot Learners by Tom Brown et al. from Open AI.

This paper introduced GPT-3 and demonstrated something that changed the direction of modern AI research: large language models could learn tasks directly from prompts and examples without task-specific fine-tuning like the methodology of GPT-1.

Instead of retraining the model for every new task, GPT-3 could often adapt dynamically through natural language instructions, one-shot examples, or few-shot prompting.

The paper also introduced the idea of in-context learning, where the model effectively learns from patterns inside the prompt itself during inference.

Here’s the original paper if you want to explore it directly: Language Models are Few-Shot Learners (PDF)

And here’s a quick infographic of what we’ll cover throughout this review:

Table of Content:

• Executive Summary

• Goals of the Paper

• Core Idea

• Methodology

• Fine-tuning vs Zero-Shot vs Few-Shot

• Model Architecture

• Experiments

• Key Findings

• Task-Specific Observations

• Generalization vs Memorization

• Discussion

• Limitations

• Conclusion

• Final Insight

• GPT-1 vs GPT-2 vs GPT-3: Key Differences

• PyTorch Implementations of the GPT Architecture Evolution

• Resources:

Prerequisites

To get the most out of this breakdown, it helps to already be familiar with a few foundational ideas.

Reading the previous reviews in this series will be especially helpful:

• AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1)

• AI Paper Review: Language Models are Unsupervised Multitask Learners (GPT-2)

GPT-3 directly builds on many of the ideas introduced in those earlier papers, especially pre-training, zero-shot learning, and large-scale language modeling.

It also helps to have:

• A general understanding of natural language processing (NLP) and how machines work with text

• A high-level idea of what a Transformer model is (you do not need deep mathematical details)

• Familiarity with supervised learning, unsupervised learning, and zero-shot learning

• A basic understanding of prompts and how language models generate text

• General machine learning concepts like training data, parameters, scaling, and inference

You do not need to be an AI researcher to follow this article, though.

I’ll keep the explanations practical and intuitive, focusing more on understanding the core ideas behind GPT-3 rather than getting lost in dense mathematical details or academic terminology.

Executive Summary

Before GPT-3, models like GPT-2 had already shown something surprising: a language model trained only to predict the next word could still perform many tasks it was never directly trained for. Translation, summarization, question answering somehow these abilities started appearing naturally as models became larger.

But there was still a limitation.

Even with GPT-2, strong performance often depended on careful prompting or additional fine-tuning. In practice, most NLP systems still followed the same pattern: train a large model first, then retrain or fine-tune it separately for every new task.

GPT-3 challenges that entire workflow.

According to the authors, if a language model becomes large enough, it can begin learning tasks directly from context alone. Instead of updating the model’s parameters, you simply show it a few examples inside the prompt, and the model continues the pattern.

This idea is what the paper calls few-shot learning.

For example, rather than training a separate translation model, you could write something like:

• dog → chien

• cat → chat

• house → ?

And GPT-3 would often continue with the correct answer: maison.

What makes this important is that the model is not learning through gradient updates during inference. There is no retraining happening in the traditional sense. The learning happens inside the context window itself, through the examples provided in the prompt.

This marks a major shift in how language models are used.

Instead of building a specialized system for every task, GPT-3 suggests that a single sufficiently large model can adapt dynamically just by reading instructions and examples. The paper refers to this behavior as in-context learning, and much of GPT-3’s contribution revolves around showing how powerful this idea becomes at scale.

Goals of the Paper

According to the authors, one of the biggest limitations of existing NLP systems is that they depend too heavily on task-specific training. Even though models had become increasingly powerful by the time GPT-3 was introduced, most systems still required a separate fine-tuning process for every new task.

In practice, this created several problems.

First, every task needed labeled data. If you wanted a model to summarize articles, answer questions, classify sentiment, or translate text, you usually needed thousands, or sometimes millions of carefully prepared examples. Collecting that data was expensive, time-consuming, and often unrealistic for smaller or niche tasks.

Second, every new capability required additional training. Even when the underlying model was already pretrained on massive amounts of text, developers still had to retrain or fine-tune it again and again for specific use cases.

The paper argues that this workflow is fundamentally inefficient. More importantly, the authors point out that it does not resemble how humans learn. Humans can often understand a task after seeing only a few demonstrations or simple instructions. We do not usually need thousands of labeled examples to figure out what is being asked.

This becomes the central question behind GPT-3:

Can a language model learn new tasks directly from context instead of relying on parameter updates and task-specific retraining?

That question drives nearly every experiment in the paper. Rather than testing whether GPT-3 can master one carefully optimized benchmark, the authors are exploring something broader: whether scaling language models can produce systems that adapt dynamically just from prompts, examples, and natural language instructions.

Core Idea

At its core, GPT-3 is still built around the same fundamental idea used in GPT-2: train a language model to predict the next token in a sequence. The training objective itself is surprisingly simple. Given some text, the model learns to guess what comes next, one token at a time.

On the surface, GPT-3 may look like nothing more than a much larger version of GPT-2. And in some ways, that is true. The model scales dramatically in size, growing to 175 billion parameters, and it is trained on a far larger and more diverse dataset gathered from sources like Common Crawl, WebText, books, and Wikipedia.

But the paper argues that something more interesting begins to happen as language models scale.

Instead of simply memorizing text patterns better, GPT-3 starts showing the ability to learn tasks directly from prompts. When the model sees examples inside the input itself, it can often continue the pattern correctly without any additional training or parameter updates.

For example, if the prompt contains a few question-answer pairs or translation examples, GPT-3 can infer the structure of the task and generate similar outputs for new inputs. In other words, the prompt becomes a temporary learning environment.

This is the key conceptual shift in the paper.

Traditional machine learning usually separates training from inference. First the model learns by updating its weights, then later it is deployed to make predictions. GPT-3 blurs that boundary. The model still learns during pretraining, of course, but during inference it can also adapt behavior dynamically based on the context it receives.

The authors describe this behavior as in-context learning.

What makes this idea important is that the model is not retrained for each task. There are no gradient updates happening while the prompt is processed. Instead, GPT-3 learns from the examples embedded inside the context window itself.

This marks a subtle but important change in how we think about language models. The prompt is no longer just an input. It effectively becomes a lightweight interface for teaching the model what to do.

Methodology

One reason GPT-3 became so influential is that the underlying training process is actually very familiar. Unlike many research papers that introduce entirely new architectures or complicated learning algorithms, GPT-3 mostly builds on ideas that already existed before it. The difference is how aggressively those ideas are scaled.

According to the authors, the core training objective remains standard autoregressive language modeling. In simple terms, the model reads text and repeatedly learns to predict the next token in the sequence. This is the same general approach used in GPT-2.

The process itself is conceptually straightforward:

• Train a very large Transformer model

• Feed it enormous amounts of internet text

• Optimize it to predict the next word over and over again

What changes dramatically is the scale.

GPT-3 is trained on hundreds of billions of tokens collected from sources such as Common Crawl, WebText, books, and Wikipedia. The paper also explains that OpenAI filtered and cleaned large portions of the Common Crawl dataset to improve quality and reduce duplication.

But the most important part of the methodology is not just how the model is trained. It is how the model is used after training.

Traditionally, NLP systems relied heavily on fine-tuning. After pretraining a language model, developers would train it again on a smaller labeled dataset for each individual task. GPT-3 experiments with a different approach entirely.

Instead of retraining the model, tasks are described directly inside the prompt.

The paper studies three main settings:

• Zero-shot learning: the model receives only a natural language instruction

• One-shot learning: the model receives a single example of the task

• Few-shot learning: the model receives several examples before solving a new case

For example, a translation prompt might look like this:

dog → chien
cat → chat
house → ?

GPT-3 then continues the pattern and predicts:

maison

What makes this remarkable is that no retraining happens during this process. The model’s weights remain completely unchanged. It is simply using the information inside the prompt to infer what kind of task is being requested.

In practice, this transforms the prompt into something much more powerful than an ordinary input. It becomes a temporary workspace where the model can recognize patterns, adapt behavior, and apply learned knowledge dynamically.

The paper repeatedly emphasizes that this behavior emerges through scale rather than task-specific engineering. GPT-3 is not trained separately for translation, summarization, reasoning, or question answering. Instead, the same general language modelinqag objective appears to produce all of these abilities when the model becomes sufficiently large.

Fine-tuning vs Zero-Shot vs Few-Shot

Model Architecture

Architecturally, GPT-3 does not introduce a radically new design. In fact, one of the most interesting aspects of the paper is that the core architecture is almost identical to GPT-2. OpenAI continues using a decoder-only Transformer model trained with an autoregressive objective.

At a high level, the Transformer architecture processes text using a mechanism called attention. Instead of reading words strictly one at a time like older recurrent models, Transformers can look across the entire sequence and determine which words are most relevant to each other.

More specifically, GPT-3 relies on self-attention, which allows the model to weigh different parts of the context while generating text. This helps the model capture long-range relationships between words, sentences, and ideas.

The model is also autoregressive, meaning it generates text sequentially by predicting the next token based on everything that came before it. This next-token prediction objective remains the foundation of GPT-3, just as it was for GPT-2.

So if the architecture is mostly the same, what actually changed?

The answer is scale.

GPT-3 dramatically increases the size of the model, the amount of training data, and the computational resources used during training. The largest version of GPT-3 contains 175 billion parameters, making it far larger than GPT-2’s 1.5 billion parameter model.

The paper also experiments with multiple model sizes ranging from 125 million parameters all the way to 175 billion. This was important because the authors wanted to study how capabilities evolve as models grow larger.

The architecture includes:

• A decoder-only Transformer design

• A context window of 2048 tokens

• Multiple model scales trained under similar objectives

• Attention mechanisms that allow the model to process contextual relationships efficiently

One of the paper’s most important observations is that performance improves smoothly as scale increases. Larger models consistently perform better across a wide range of tasks, including translation, question answering, reasoning, and few-shot learning.

This idea becomes central to the entire GPT-3 paper.

Rather than relying on handcrafted task-specific systems, the authors suggest that many advanced capabilities emerge naturally when language models become sufficiently large and are trained on enough diverse data. In other words, scaling itself starts acting like a research strategy.

What makes this shift important is that GPT-3 does not achieve its results through complicated architectural innovations. The paper’s argument is much simpler, and in some ways more surprising:

A relatively standard Transformer architecture, when scaled aggressively enough, begins to display entirely new behaviors.

Note: The original figure illustrates the complete Transformer architecture (Encoder–Decoder) from Attention Is All You Need. For clarity and relevance to GPT-style models, the image used here was cropped to focus only on the decoder side of the architecture, since GPT models are based on a decoder-only Transformer design.

Reference: Brownlee, J. Encoders and Decoders in Transformer Models Machine Learning Mastery.

Experiments

To understand whether GPT-3 could truly learn from context alone, the authors evaluated the model across a very broad range of NLP tasks. Rather than focusing on a single benchmark, the paper tests whether the same pretrained model can adapt to many different kinds of problems using only prompts and examples.

The experiments cover a wide variety of domains, including:

• Language modeling and text completion

• Question answering

• Translation between languages

• Reading comprehension

• Commonsense reasoning

• Winograd-style reasoning tasks

• Cloze and sentence completion tasks

• Synthetic reasoning problems such as arithmetic and word manipulation

What makes these experiments especially important is the evaluation setup itself.

Instead of fine-tuning GPT-3 separately for each benchmark, the model is tested entirely through prompting. The authors evaluate GPT-3 in three different settings:

• Zero-shot learning, where the model receives only a task description

• One-shot learning, where it receives a single example

• Few-shot learning, where several demonstrations are included inside the prompt

For example, in translation tasks, the prompt may contain a few English-to-French examples before asking the model to continue the pattern. In question-answering tasks, the model might see several example questions and answers before attempting a new one.

Importantly, the model’s parameters never change during these evaluations. There are no gradient updates, no retraining steps, and no task-specific optimization. GPT-3 performs every task using the exact same pretrained weights.

This is one of the paper’s biggest departures from traditional NLP systems.

At the time, most state-of-the-art models achieved strong benchmark results through supervised fine-tuning on carefully prepared datasets. GPT-3 instead tests whether a single large language model can generalize across tasks simply by understanding patterns inside prompts.

The paper also evaluates how performance changes as model size increases. OpenAI trained multiple versions of GPT-3, ranging from 125 million parameters up to 175 billion parameters, then compared how scaling affected zero-shot, one-shot, and few-shot behavior.

According to the authors, larger models become noticeably better at using contextual information. Few-shot learning improves especially strongly with scale, suggesting that bigger models are not just memorizing more information. They are becoming better at adapting to new tasks dynamically.

Key Findings

This is the section where GPT-3 stops feeling like “just a bigger language model” and starts looking like something fundamentally different.

According to the paper, one of the clearest patterns across nearly all experiments is that performance improves consistently as model size increases. As GPT-3 scales from millions of parameters to hundreds of billions, the model becomes dramatically better at understanding prompts, adapting to context, and performing tasks it was never explicitly trained for.

But the most surprising result is not simply higher benchmark scores.

The real breakthrough is that few-shot learning actually works at scale.

Across many tasks, GPT-3’s few-shot performance approaches strong fine-tuned systems, and in some cases even matches or surpasses them. This is remarkable because GPT-3 achieves these results without updating its weights for individual tasks. Everything happens through prompting alone.

One of the strongest examples appears in question answering benchmarks.

On TriviaQA, GPT-3 improves significantly as more examples are provided in the prompt. The paper reports that zero-shot performance is already competitive, but one-shot and few-shot prompting push results even further, eventually reaching or exceeding some state-of-the-art fine-tuned systems in the same closed-book setting.

Source: Brown et al. (2020), Language Models are Few-Shot Learners, Figure 1.2.

The same pattern appears repeatedly throughout the paper:

• Few-shot prompting consistently outperforms zero-shot prompting

• Larger models make better use of contextual examples

• Scaling improves not only accuracy, but adaptability itself

This last point is especially important.

The paper suggests that scaling does more than help the model memorize facts or generate more fluent text. As models become larger, they appear to develop stronger in-context learning abilities. In other words, bigger models become better at inferring patterns and task structures directly from prompts.

The authors even observe that the gap between zero-shot and few-shot performance grows with model size. Smaller models struggle to learn effectively from prompts, while larger models can often infer the task from only a handful of examples.

What makes this finding historically important is that it changes how researchers think about capability growth in AI systems.

Before GPT-3, scaling was often viewed mainly as a way to improve existing performance metrics. GPT-3 introduces a different possibility: that entirely new behaviors can emerge as models become sufficiently large.

This is why the paper became so influential. It was not just reporting better benchmark numbers. It was presenting evidence that scale itself can unlock qualitatively new forms of learning behavior.

Task-Specific Observations

When you look beyond the headline results, the paper reveals something more nuanced about GPT-3: its abilities are highly uneven. The model performs surprisingly well in some areas, yet still struggles badly in others.

GPT-3 shows particularly strong performance on tasks that align closely with pattern recognition and language continuation.

Translation is one notable example. While GPT-3 was never trained specifically as a translation system, the model can still produce impressive results when given a few examples in the prompt. According to the paper, few-shot translation performance improves substantially as model size increases, especially when translating into English.

The model also performs well on question answering benchmarks, especially in closed-book settings where the answer must come directly from information stored inside the model’s parameters. Tasks like TriviaQA show strong gains as GPT-3 moves from zero-shot to few-shot prompting.

Text completion and cloze-style tasks are another major strength. GPT-3 demonstrates a strong ability to continue patterns, complete paragraphs, and infer missing words from context. On datasets like LAMBADA, the few-shot setup produces especially large improvements.

But the paper is also careful about documenting weaknesses.

GPT-3 struggles noticeably on certain reasoning-heavy benchmarks, particularly tasks involving natural language inference. Datasets like ANLI remain difficult even for the largest model.

Some reading comprehension tasks also expose limitations. In several cases, GPT-3 generates answers that sound plausible but fail to demonstrate deep understanding of the passage. This becomes a recurring theme throughout the paper: fluent language generation does not always mean reliable reasoning.

One of the most interesting observations is how sensitive GPT-3 is to prompt design.

Performance often changes dramatically depending on how examples are written, formatted, or ordered inside the context window. In many tasks, adding just a few demonstrations significantly improves accuracy.

This suggests something important about how GPT-3 operates.

The model is not simply retrieving fixed knowledge from memory. Instead, it relies heavily on contextual cues to infer what kind of behavior is expected. Small prompt changes can reshape the model’s interpretation of the task itself.

In practice, this paper helped introduce an entirely new idea to the AI community: that how you ask the model can matter almost as much as the model itself.

That insight eventually evolves into what we now call prompt engineering.

Generalization vs Memorization

One of the biggest questions surrounding GPT-3 is whether the model is genuinely learning useful patterns, or simply memorizing enormous portions of the internet.

This concern becomes especially important because GPT-3 is trained on massive web-scale datasets, including Common Crawl. With a model this large, it is reasonable to ask whether strong benchmark performance comes from real generalization or from accidentally seeing parts of the evaluation data during training.

The authors take this issue seriously and dedicate an entire section of the paper to studying what they call data contamination.

According to the paper, OpenAI searched for overlaps between the training data and benchmark datasets used during evaluation. They discovered that some contamination did exist. In other words, portions of certain evaluation datasets appeared somewhere inside the model’s training corpus.

However, the authors argue that this overlap is not large enough to fully explain GPT-3’s results.

For many benchmarks, performance improvements remain consistent even after accounting for contamination effects. The paper also notes that some tasks specifically designed to test adaptation and reasoning still show strong few-shot behavior despite being unlikely to appear directly in the training data.

Another important observation is that GPT-3 still underfits the training data. This means the model has not perfectly memorized everything it has seen, even after extremely large-scale training.

That detail matters because it suggests the model is learning statistical structures and linguistic patterns rather than storing an exact copy of the dataset.

Of course, memorization does still happen to some extent. Large language models can reproduce fragments of training text, especially when rare or repeated data appears frequently during training. The paper does not deny this. Instead, the authors argue that memorization alone cannot explain GPT-3’s broad performance across translation, reasoning, question answering, and in-context learning tasks.

In practice, the evidence points toward something more complex.

GPT-3 appears to absorb patterns, relationships, and task structures from large-scale text data, then reuse those patterns flexibly in new contexts. That is very different from simply copying stored answers.

This distinction becomes one of the central debates in modern AI research. GPT-3 forced researchers to think more carefully about what it actually means for a language model to “understand” something, and where the boundary lies between memorization, pattern recognition, and genuine generalization.

Discussion

This is the point in the paper where the broader implications of GPT-3 start becoming clear.

According to the authors, large language models may be doing something more general than simply predicting text. By training on enormous amounts of language data, the model appears to learn patterns associated with tasks themselves.

That idea changes how we think about language modeling.

Traditionally, NLP systems were designed around explicit supervision. If you wanted a model to translate text, answer questions, summarize documents, or classify sentiment, you trained it specifically for that task using labeled examples.

GPT-3 suggests a different possibility.

The paper argues that many tasks are already implicitly embedded inside natural language data. During pretraining, the model encounters countless examples of explanations, translations, conversations, reasoning patterns, instructions, and question-answer pairs scattered across the internet. As scale increases, the model begins learning these behaviors indirectly.

In practice, this means the model does not always require explicit retraining to perform a new task. Instead, prompts and examples can activate behaviors the model has already absorbed during pretraining.

This is why prompting becomes so powerful in GPT-3.

The prompt is not merely providing information. It is guiding the model toward a behavior pattern that already exists somewhere inside its learned representations.

At the same time, the authors are careful not to overstate the results.

Throughout the paper, they repeatedly acknowledge that GPT-3 is still inconsistent. Some outputs are remarkably convincing, while others are obviously incorrect, nonsensical, or logically flawed.

This becomes one of GPT-3’s defining characteristics.

The model often sounds far more confident than it actually is. It can generate fluent explanations and persuasive answers even when the underlying reasoning is weak or factually wrong. In some tasks, especially deeper reasoning and reading comprehension benchmarks, GPT-3 still struggles significantly.

So the paper does not present GPT-3 as a solved form of intelligence.

Instead, it presents evidence that scaling language models unlocks new capabilities that were previously weak or absent. The results are impressive enough to suggest a major shift in direction, but not strong enough to eliminate the need for further research.

That balance is part of what makes the paper influential. It is ambitious, but also surprisingly honest about the limitations that still remain.

Limitations

One reason the GPT-3 paper remained credible despite the excitement surrounding it is that the authors were unusually open about the model’s weaknesses. The paper does not claim that few-shot learning solves NLP, nor does it pretend that GPT-3 works reliably on every task.

In many cases, traditional fine-tuned systems still perform better.

Although GPT-3 achieves impressive few-shot results across a wide range of benchmarks, the model continues to struggle on several reasoning-heavy tasks, especially natural language inference and certain reading comprehension datasets.

The paper also emphasizes that GPT-3’s success depends heavily on scale. Smaller versions of the model show far weaker few-shot capabilities, while the strongest results appear only at extremely large parameter counts.

This creates a major practical problem.

Training GPT-3 required enormous computational resources, specialized infrastructure, and vast amounts of data. The largest model contains 175 billion parameters and was trained using large GPU clusters over massive datasets.

In practice, very few organizations in the world could realistically reproduce this work at the time.

The paper also discusses broader concerns around bias and fairness. Since GPT-3 learns from large internet datasets, it inevitably absorbs social biases, stereotypes, and problematic language patterns present in the data itself.

This becomes especially concerning because the model can generate highly convincing text. Incorrect or biased outputs may sound authoritative even when they are misleading or harmful.

Another issue the authors examine is data contamination. Because GPT-3 is trained on web-scale corpora, parts of benchmark datasets may accidentally appear in the training data. The paper investigates this directly and acknowledges that some overlap exists, although the authors argue that contamination alone does not explain the overall results.

There is also an environmental and economic cost to scaling models this aggressively.

Training systems at the scale of GPT-3 consumes enormous amounts of compute and energy, raising questions about sustainability and accessibility in AI research. As models become larger, cutting-edge progress increasingly depends on access to industrial-scale infrastructure.

This creates a tension that still exists today.

GPT-3 demonstrated that scaling works extraordinarily well, but it also highlighted how concentrated advanced AI research was becoming. The future of large language models was clearly promising, but also increasingly expensive.

Conclusion

The paper ends with a surprisingly simple conclusion: scaling language models changes what they are capable of doing.

According to the authors, GPT-3 demonstrates that a sufficiently large language model can learn tasks directly from context without requiring gradient updates or task-specific fine-tuning.

That idea represents a major shift in the direction of NLP.

For years, the standard workflow in machine learning looked something like this:

• Pretrain a model

• Fine-tune it for a specific task

• Deploy the specialized system

GPT-3 introduces a different paradigm.

Instead of retraining the model repeatedly for new tasks, the same pretrained model can often adapt through prompts alone. Instructions and examples inside the context window become enough to guide the model toward useful behavior.

In other words, the workflow starts looking more like this:

• Train once

• Adapt dynamically through prompting

What makes this important is not just convenience. It changes how researchers think about generalization itself.

The paper suggests that many capabilities traditionally associated with supervised learning can emerge naturally from large-scale language modeling. Translation, question answering, reasoning, summarization, and even task adaptation begin appearing inside a single unified system trained only with next-token prediction.

At the same time, the authors remain careful in their conclusions.

GPT-3 is clearly powerful, but it is not reliable enough to be considered a complete solution to intelligence or reasoning. The paper repeatedly acknowledges weaknesses involving logic, factual accuracy, bias, and consistency.

Still, the broader message is difficult to ignore.

GPT-3 showed that scaling language models does not simply improve fluency. It can produce entirely new behaviors that were weak or absent in smaller systems. That realization reshaped the trajectory of modern AI research and laid the foundation for the prompt-driven systems that would soon follow.

Final Insight

If GPT-1 introduced the idea of large-scale pretraining followed by fine-tuning, and GPT-2 showed that language models could generalize surprisingly well without task-specific training, then GPT-3 pushes the idea even further.

It suggests that language models can begin learning during inference itself.

That is the real conceptual shift behind this paper.

Before GPT-3, most AI systems were still fundamentally task-specific. Even powerful pretrained models usually needed additional supervised training before they became useful for a particular application.

GPT-3 starts breaking that pattern.

Instead of building a separate model for translation, summarization, question answering, or reasoning, the same model can adapt dynamically depending on the prompt it receives. Examples inside the context window effectively become temporary instructions for behavior.

In practice, this moves AI systems away from narrow specialization and toward something more flexible:

• From task-specific systems

• To general-purpose models that adapt on the fly

What makes this especially important is that GPT-3 did not achieve this through complicated symbolic reasoning systems or handcrafted pipelines. The model was still trained using a relatively simple next-token prediction objective. Yet at sufficient scale, entirely new behaviors started emerging.

Looking back, this paper feels less like the end of the GPT series and more like the beginning of a new era.

Many ideas that now define modern AI trace directly back to GPT-3:

• Prompt engineering

• Instruction-following systems

• In-context learning

• Conversational AI assistants

• General-purpose foundation models

And ultimately, systems like ChatGPT exist because GPT-3 demonstrated that prompting itself could become a powerful interface for interacting with intelligence.

That is why this paper became historically important.

It did not just scale language models. It changed how people imagined using them.

GPT-1 vs GPT-2 vs GPT-3: Key Differences

PyTorch Implementations of the GPT Architecture Evolution

GPT-1: Pre-training + Fine-Tuning Architecture

class GPT1(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(512, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(d_model)
            for _ in range(n_layers)
        ])

        self.ln_f = nn.LayerNorm(d_model)

        # Language modeling head
        self.lm_head = nn.Linear(d_model, vocab_size)

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.ln_f(x)

        logits = self.lm_head(x)

        return logits

GPT1 inherits from nn.Module, which is the base class used to build neural networks in PyTorch. The constructor (init) defines all trainable layers used by the model.

nn.Embedding(vocab_size, d_model) creates a learnable lookup table that converts token IDs into dense vectors. Each token in the vocabulary is mapped to a vector of size d_model.

The positional embedding layer adds information about token order. Since Transformers process tokens in parallel, they need explicit positional information to understand sequence structure.

nn.ModuleList([...]) stores multiple Transformer blocks while ensuring PyTorch properly tracks their parameters during training. Each TransformerBlock typically contains masked self-attention and a feed-forward network.

nn.LayerNorm(d_model) applies layer normalization before the output projection. This helps stabilize training and improves gradient flow in deep Transformer architectures.

The language modeling head (nn.Linear) projects the hidden representations back into vocabulary space. The output size equals vocab_size, producing prediction scores for every possible next token.

Inside the forward() method, input_ids.size(1) retrieves the sequence length, and torch.arange(...) generates positional indices for each token position.

The token embeddings and positional embeddings are added together to produce the initial Transformer input representation.

The model then passes the representation through each Transformer block sequentially:

for block in self.transformer_blocks:
    x = block(x)

This iterative stacking is what allows GPT models to learn increasingly abstract contextual representations.

After normalization, the final hidden states are passed into lm_head, producing logits. These logits are unnormalized prediction scores used to compute probabilities for next-token generation.

The model finally returns the logits tensor, which is typically passed through softmax during inference or used directly with CrossEntropyLoss during training.

GPT-2: Zero-Shot Multitask Architecture

class GPT2(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(1024, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                pre_layer_norm=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Like GPT-1, the model begins with token embeddings and positional embeddings. nn.Embedding converts token IDs into dense vectors, while positional embeddings provide information about token order in the sequence.

One noticeable difference is the larger positional embedding size (1024 instead of 512), allowing GPT-2 to process longer contexts.

The Transformer layers are stored using nn.ModuleList, but each TransformerBlock now uses:

pre_layer_norm=True

This means layer normalization is applied before attention and feed-forward operations rather than after them. This “Pre-LN” design significantly improves gradient flow and training stability in deeper Transformer models.

The forward pass follows the same overall pipeline:

1. Generate positional indices with torch.arange()

2. Add token and positional embeddings

3. Pass representations through stacked Transformer blocks

4. Apply final normalization

5. Project outputs into vocabulary space

The sequential block processing happens here:

for block in self.transformer_blocks:
    x = block(x)

GPT-2 also introduces a small optimization in the output layer:

self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

self.lm_head = nn.Linear(d_model, vocab_size, bias=False)

The bias term is removed because it provides little benefit in large language modeling setups and slightly reduces parameter count.

Finally, the model returns logits, which contain prediction scores for every token in the vocabulary at each sequence position.

GPT-3: Few-Shot / In-Context Learning Architecture

class GPT3(nn.Module):
    def __init__(
        self,
        vocab_size=50257,
        d_model=12288,
        n_layers=96,
        n_heads=96,
        context_length=2048
    ):
        super().__init__()

        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(context_length, d_model)

        self.transformer_blocks = nn.ModuleList([
            TransformerBlock(
                d_model=d_model,
                n_heads=n_heads,
                pre_layer_norm=True,
                sparse_attention=True
            )
            for _ in range(n_layers)
        ])

        self.final_layer_norm = nn.LayerNorm(d_model)

        self.lm_head = nn.Linear(
            d_model,
            vocab_size,
            bias=False
        )

    def forward(self, input_ids):
        positions = torch.arange(input_ids.size(1))

        x = (
            self.token_embedding(input_ids)
            + self.position_embedding(positions)
        )

        for block in self.transformer_blocks:
            x = block(x)

        x = self.final_layer_norm(x)

        logits = self.lm_head(x)

        return logits

Compared to earlier GPT versions, this model dramatically increases scale. The embedding size (d_model=12288) and the number of Transformer layers (96) allow the network to learn highly complex language patterns and long-range dependencies.

The model also uses 96 attention heads:

n_heads=96

Multi-head attention allows the model to focus on different relationships between tokens simultaneously, improving contextual understanding.

The positional embedding length is expanded to 2048, enabling the model to process much longer sequences than GPT-2.

Each Transformer block is configured with:

pre_layer_norm=True,
sparse_attention=True

Pre-layer normalization improves training stability in very deep networks, while sparse attention reduces the computational cost of attention by limiting how many tokens attend to each other. This becomes important at GPT-3 scale, where full attention over long sequences is extremely expensive.

The forward pass follows the standard GPT pipeline:

1. Convert token IDs into embeddings

2. Add positional information

3. Pass representations through stacked Transformer blocks

4. Apply final layer normalization

5. Generate vocabulary logits

The core iterative processing happens here:

for block in self.transformer_blocks:
    x = block(x)

Finally, the output layer projects the hidden states into vocabulary space, producing logits used for next-token prediction during training and text generation.

Resources:

• Pytorch Projects for GPT series

• Attention Is All You Need

• Improving Language Understanding by Generative Pre-Training (GPT-1)

• Language Models are Unsupervised Multitask Learners (GPT-2)

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• XLNet: Generalized Autoregressive Pretraining for Language Understanding

• RoBERTa: A Robustly Optimized BERT Pretraining Approach

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• Turing-NLG: A 17-Billion-Parameter Language Model by Microsoft

• Sparse Transformers

• Scaling Laws for Neural Language Models

Contact Me

• Github

• X

• Linkedin

If you wanted a model to translate text, summarize an article, or answer questions, you usually had to collect labeled data and train it separately for each task. AI was powerful, but still very narrow.

Then GPT-2 introduced a different idea.

Instead of teaching a model every task individually, researchers explored whether simply training a model to predict the next word on a massive amount of internet text could be enough for useful abilities to emerge on their own.

And surprisingly, it worked.

The model began showing early signs of generalization. It could answer questions, summarize text, translate between languages, and complete prompts – all without task-specific training or fine tuning them toward down stream tasks.

Now, research papers like the one that introduced these new ideas can be difficult and time-consuming to read, especially when they’re filled with technical terminology and experimental details. So in this article, I’ll break the paper down in a simple and practical way.

We’ll look at what problem the paper was trying to solve, the main ideas behind GPT-2, how zero-shot learning works, and why this paper became such an important step toward modern large language models.

By the end, you should understand the key insights of GPT-2 without needing to read the full paper yourself.

Paper Overview

In this article, we’ll review the paper Language Models are Unsupervised Multitask Learners by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.

The paper introduced GPT-2 and showed how a language model trained on massive amounts of text could perform multiple tasks without task-specific training.

Here’s the actual paper if you want to read it yourself:

Language Models are Unsupervised Multitask Learners (PDF)

And here’s a quick infographic of what we’ll cover in this review:

Table of Contents:

• Executive Summary

• Goals of the Paper

• Core Idea

• Methodology

• Zero-Shot Setup

• Fine-tuning vs Zero-Shot Learning

• Training Data (Web Text)

• Input Representation

• Model Architecture

• Experiments

• Key Findings

• Task-Specific

• Generalization vs Memorization

• Discussion

• Limitations

• Conclusion

• Final Insight

• GPT-1 vs GPT-2 — Key Differences

• Resources

Prerequisites

To get the most out of this breakdown, it helps to be familiar with a few basic ideas:

• Reading the previous review, AI Paper Review: Improving Language Understanding by Generative Pre-Training (GPT-1), will be helpful and will give you some solid background info and context (since GPT-2 directly builds on many of the ideas introduced there).

• A general understanding of natural language processing (NLP) and how machines work with text

• A high-level idea of what a Transformer model is (you don’t need deep technical details, just the basic concept)

• The difference between supervised learning, unsupervised learning, and zero-shot learning

• Basic machine learning concepts like training data, models, and scaling

If you’re not fully comfortable with all of these, that’s completely okay. I’ll keep the explanations as simple and intuitive as possible, focusing more on understanding the ideas than getting lost in heavy technical details.

Executive Summary

Before GPT-2, most NLP systems depended heavily on supervised learning. Each task, whether it was translation, question answering, or summarization, typically required its own labeled dataset and a model trained specifically for it.

This paper challenges that approach.

According to the authors, a single large language model, trained only to predict the next word in a sequence of text, can learn to perform many different tasks without any task-specific training.

Instead of being explicitly taught how to solve each problem, the model picks up these abilities from patterns in the data.

In simple terms, the model is not directly trained to translate, answer questions, or summarize. Rather, it learns to do these things implicitly through exposure to large amounts of text.

This marks an important shift. Rather than relying on supervised learning for every task, the paper shows that models can begin to generalize across tasks in what is now known as a zero-shot setting.

Goals of the Paper

To understand the motivation behind this work, it helps to look at the limitations of traditional NLP systems.

According to the authors, most existing approaches rely heavily on labeled datasets, require separate training for each task, and struggle to generalize beyond the specific problems they were designed for.

In practice, this makes systems powerful but narrow: they perform well on what they are trained for, but don’t easily transfer that knowledge elsewhere.

This paper explores a different direction.

The authors ask whether a model can learn to perform multiple tasks without explicit supervision, simply by training on large amounts of text.

They also investigate whether language modeling alone is enough to capture general capabilities, and whether increasing the size of the model and the amount of data can improve this behavior.

At its core, the goal is to move toward more general systems that learn from language itself, rather than from carefully labeled datasets.

Core Idea

At the heart of the paper is a simple but powerful idea: instead of training models in the traditional supervised way (mapping inputs directly to outputs), the authors train a model to do just one thing: predict the next word in a sequence of text.

At first, this might sound limited. But the key insight is that natural language already contains many examples of tasks embedded within it.

Text on the internet includes questions followed by answers, translations between languages, summaries of longer content, and detailed explanations.

According to the paper, by learning to predict and generate text, the model is indirectly learning how these tasks work. In other words, it begins to model relationships like p(output | input, task) without ever being explicitly told what the task is.

This is what allows the model to move beyond a single objective and start behaving like a general system.

Methodology

To understand how this idea works in practice, it helps to look at how the model is trained.

According to the authors, everything starts with a standard language modeling objective.

The model is trained to predict the next token in a sequence based on the tokens that come before it.

While this may seem simple, it allows the model to learn the underlying structure of language over time.

Formally, this means the model is learning probabilities over sequences of text. In practice, this ability enables it to generate coherent text, complete sentences, and even mimic patterns that resemble specific tasks.

This is what makes the approach powerful. Even though the model is only trained to predict the next word, it ends up capturing much richer behavior that can be applied to a variety of tasks.

Zero-Shot Setup

One of the most important differences from earlier approaches is how the model is used after training.

Unlike GPT-1, there's no fine-tuning or task-specific training. The model isn't adapted or retrained for each new task. Instead, everything is handled through the input itself.

According to the authors, tasks are expressed directly as text prompts. For example, you might write something like “Translate to French:” followed by a sentence, or “Answer the question:” followed by a prompt. The model then continues the text in a way that reflects the task.

In practice, this means the model isn't explicitly told what to do through training – it infers the task from the structure of the input and responds accordingly.

Fine-tuning vs Zero-Shot Learning

Training Data (Web Text)

Another key part of this work is the dataset used to train the model.

Instead of relying on traditional sources like Wikipedia, books, or news articles alone, the authors created a new dataset called Web Text.

It consists of millions of documents – around 40 GB of text – collected from links shared on Reddit that received a certain level of engagement.

According to the paper, this filtering step helps improve the overall quality of the data, since the content is more likely to be interesting or useful to readers.

What makes this dataset important is its diversity. It contains real-world language from many domains, and more importantly, it includes natural examples of tasks, such as explanations, question–answer pairs, and translations, embedded within the text itself.

Input Representation

To process text, the model uses a technique called Byte Pair Encoding (BPE).

According to the authors, BPE works as a middle ground between word-level and character-level representations.

Instead of treating text strictly as full words or individual characters, it breaks it into smaller units that can adapt depending on how frequently patterns appear in the data.

In practice, this allows the model to handle a wide range of text more effectively, including rare words and different languages. It also improves generalization, since the model isn't limited to a fixed vocabulary of complete words.

Model Architecture

The model used in this paper is based on a Transformer (decoder-only) architecture, similar to GPT-1 but significantly scaled up.

According to the authors, the model relies on masked self-attention, which allows it to look at previous tokens in a sequence while predicting the next one.

This means it processes text step by step, always using past context to generate the next token.

Compared to GPT-1, several important changes were introduced.

The model can handle longer context, with sequences of up to 1024 tokens, and uses a larger vocabulary of around 50,000 tokens. It's also much deeper, with more layers and significantly more parameters.

The authors trained multiple versions of the model, ranging from 117 million to 1.5 billion parameters.

The largest of these is what we now refer to as GPT-2, and it's the one responsible for most of the strong results reported in the paper.

Transformer (decoder-only)

Note: The original figure illustrates the complete Transformer architecture (Encoder–Decoder) from Attention Is All You Need. For clarity and relevance to GPT-style models, the image used here was cropped to focus only on the decoder side of the architecture, since GPT models are based on a decoder-only Transformer design.

Reference: Brownlee, J. Encoders and Decoders in Transformer Models Machine Learning Mastery.

Experiments

To evaluate the model, the authors tested it across a wide range of tasks – but with an important constraint: according to the paper, the model wasn't trained or fine-tuned on any of these tasks.

Instead, everything was evaluated in a zero-shot setting, where the model is simply given a prompt and asked to continue the text.

They applied this setup to different types of problems, including language modeling benchmarks, reading comprehension, translation, summarization, question answering, and commonsense reasoning.

The goal here was not just to measure performance, but to see how far a single model (trained only on raw text) could generalize across tasks without any additional training.

Key Findings

After evaluating the model across different tasks, the results were stronger than many would have expected.

According to the authors, GPT-2 achieves state-of-the-art results on 7 out of 8 language modeling benchmarks in a zero-shot setting.

One of the most important observations is that performance consistently improves as the model size increases, following a roughly log-linear trend.

In other words, scaling up the model leads to better results across tasks.

The paper also shows that larger models display more consistent multitask behavior.

For example, GPT-2 performs well on tasks that require long-range understanding, such as LAMBADA, and shows competitive results in reading comprehension on datasets like CoQA.

It even demonstrates early capabilities in translation and can answer factual questions without being explicitly trained for those tasks.

In practice, the key takeaway is clear: increasing model size and data plays a major role in unlocking these capabilities.

Task-Specific

Looking more closely at individual tasks, the paper gives a clearer picture of where the model performs well and where it still struggles.

GPT-2 shows surprisingly strong results in reading comprehension, even without any task-specific training. But its performance on summarization is still limited.

While it can generate summaries that look reasonable, they're often less accurate compared to supervised approaches.

For translation, the model demonstrates some ability, but the results are still far from competitive.

On the other hand, question answering improves noticeably as the model size increases, suggesting that scale plays an important role in this capability.

Overall, the model is far from perfect. But what stands out is that it's clearly beginning to learn general skills across tasks, even without being explicitly trained for them.

Generalization vs Memorization

A natural question that comes up is whether the model is actually learning useful patterns or simply memorizing the training data.

The authors address this directly. They analyze overlap between the training dataset and evaluation benchmarks using n-gram comparisons, looking for signs that the model might be copying rather than generalizing.

According to the paper, while some overlap does exist (as is common in large datasets), it's not enough to explain the model’s performance.

They also observe that the model still underfits the data, meaning it hasn’t fully captured everything in the training set.

This is an important point: if the model was mainly memorizing, we would expect it to fit the data much more closely.

In practice, this suggests that the improvements are coming from genuine learning rather than simple memorization, even though some overlap is unavoidable.

Discussion

This section is where the authors step back and reflect on what these results actually mean.

According to the paper, language models trained on large and diverse datasets aren't just learning representations of text. They're beginning to learn how to perform tasks directly, even without supervision.

In other words, pre-training is doing more than providing useful features: it's capturing patterns that resemble real task behavior.

At the same time, the authors are careful not to overstate the results.

While the zero-shot capabilities are impressive, performance is still far from practical on many tasks.

Some outputs look convincing on the surface but lack accuracy when measured more carefully.

In practice, this section highlights both sides of the story. The approach is clearly promising, but it's still an early step toward more general systems.

Limitations

Despite the progress shown in the paper, the approach still has several important limitations.

According to the authors, zero-shot performance, while impressive, is generally weaker than fully supervised models on many tasks.

The results also depend heavily on scale, both in terms of model size and the amount of data used. This means that smaller models don't show the same level of capability.

In addition, some tasks, such as summarization, remain relatively weak.

The model can produce outputs that look plausible, but they often lack accuracy or consistency when evaluated more carefully.

Another practical challenge is the cost. Training these models requires significant computational resources and large datasets, which makes this approach difficult to reproduce or scale for many researchers.

Conclusion

The paper ends with a simple but powerful idea.

According to the authors, when a language model is trained on a sufficiently large and diverse dataset – and with enough capacity – it begins to generalize across tasks and perform them without explicit training.

This suggests that the model isn't just learning language, but also the structure of the tasks embedded within it.

In practice, this points to a different way of thinking about AI systems. Instead of designing and training a model for each specific task, we can focus on training a single model on large-scale language data – and allow useful capabilities to emerge naturally from that process.

Final Insight

If GPT-1 introduced the idea of combining pre-training with fine-tuning, GPT-2 takes that idea a step further.

According to the paper, pre-training alone - when done at a large enough scale – can already produce models that begin to perform a wide range of tasks without any additional training.

This is a subtle but important shift, because it suggests that general capabilities can emerge directly from exposure to large amounts of text.

In my view, this is the point where things start to change direction.

The focus moves away from designing task-specific systems and toward building more general models that can adapt on their own.

This idea directly sets the stage for what comes next: models like GPT-3, ChatGPT, and modern large language systems that build on this same principle.

GPT-1 vs GPT-2 — Key Differences

Resources:

• Pytorch Projects for GPT series

• Attention Is All You Need

• Improving Language Understanding by Generative Pre-Training

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

• Semi-supervised Sequence Learning

• Universal Language Model Fine-tuning for Text Classification

• Deep Contextualized Word Representations

• Neural Machine Translation of Rare Words with Subword Units

• Distributed Representations of Words and Phrases and Their Compositionality

• GloVe: Global Vectors for Word Representation

Contact Me

• Github

• X

• Linkedin

Now, not everyone enjoys reading research papers or has the time to comb through and digest all that (sometimes very dense) info. So I decided to do the hard work for you and share the key insights in a series of AI paper reviews.

The goal isn’t to turn this into a heavy academic discussion, but to explain the main ideas in a clear and practical way. You'll learn what problem the paper was trying to solve, what approach it introduced, and why it mattered.

In each article, you’ll get a simple breakdown of the paper, how it works, and what you should take away from it. By the end, you should understand the key idea without needing to go through the full research paper yourself.

Paper Overview

The first paper I'll be reviewing is "Improving Language Understanding by Generative Pre-Training", by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.

Here's the actual paper if you want to read it yourself: Read the paper.

And here's a little infographic of what we'll cover here:

Table of Contents

• Executive Summary

• Goals of the Paper

• Methodology

• Transformer vs. BERT vs. GPT

• Model Architecture

• Key Techniques

• Key Findings

• Conclusions

• Limitations

• Related Work & Context

• Final Insight

• Resources

Prerequisites

To get the most out of this breakdown, it helps to be familiar with a few basic ideas:

• A general understanding of natural language processing (NLP) and how machines work with text

• A high-level idea of what a Transformer model is (you don’t need deep details, just the concept)

• The difference between supervised and unsupervised learning

• Basic machine learning concepts like training data and models

If you’re not fully comfortable with all of these, that’s okay, you can still follow along. The goal here is to keep things clear and intuitive.

Executive Summary

Before models like GPT became what we know today, there was a key limitation: AI systems were good at specific tasks, but struggled with general understanding.

In this paper, the authors introduce a simple but powerful idea. Instead of training a model separately for each task, they first train it on a large amount of unlabeled text to learn the structure of language. Then, they adapt it to specific tasks using smaller labeled datasets.

According to the authors, this two-step approach (pre-training followed by fine-tuning) allows a single model to handle many different tasks with minimal changes.

In practice, this marked a major shift: rather than building a new model for every problem, we can train one general model that learns language itself and then reuse it across tasks.

Goals of the Paper

To understand the motivation behind this work, it helps to look at the main limitations in NLP at the time.

Most models depended heavily on large labeled datasets, which weren’t always available. Many tasks simply didn’t have enough labeled data to train effective systems. On top of that, existing models were usually designed for a single task, making them hard to reuse or adapt.

Because of this, the authors aimed to reduce the reliance on labeled data and move toward a more general approach. Their goal was to build a language model that could learn from large amounts of raw text and then be applied across different tasks.

According to the paper, they also wanted to enable transfer learning: the ability to take knowledge learned from one task and apply it to others. They also wanted to improve performance without needing to redesign a new model each time.

Methodology

To understand how the authors approached this problem, let’s look at the core idea behind their method.

Pre-Training

At the heart of the paper is a simple but powerful approach built in two stages. The first stage is pre-training, where the model learns directly from raw text.

According to the authors, the model is trained on a large corpus of unlabeled text using a language modeling objective (predicting the next word in a sequence) – specifically, predicting the next word based on the previous ones to solve the intractable problem of high dimension probabilities. Through this process, the model gradually learns important aspects of language, such as grammar, context, structure, and general patterns.

The paper highlights that datasets like BooksCorpus are used in this stage because they contain long, continuous text. This is important, since it helps the model understand relationships across sentences rather than just short fragments.

Fine-Tuning (Adapting to Tasks)

Once the model has learned general language patterns, the next step is fine-tuning, where it is adapted to specific tasks using labeled data.

According to the authors, this includes tasks like question answering, text classification, natural language inference, and semantic similarity. Instead of building a new model for each task, the same pre-trained model is reused with only small adjustments.

In practice, this is what makes the approach powerful: the model already understands language at a general level, so it can quickly adapt to different tasks without needing to be redesigned from scratch.

Transformer vs. BERT vs. GPT

Before diving into GPT-1, it helps to understand how modern language models are structured. Most of them are based on the Transformer architecture, but they use it in different ways: encoder-only models (like BERT), decoder-only models (like GPT), or full encoder–decoder models.

The original encoder–decoder Transformer was mainly used for tasks like machine translation. Encoder-only models are typically used for understanding tasks such as text classification and sentiment analysis, while decoder-only models are designed for generation tasks like text creation, powering systems such as ChatGPT, Gemini, and Claude.

Illustration comparing Transformer, GPT, and BERT architectures, adapted from Comparing Large Language Models: GPT vs. BERT vs. T5 showing encoder-decoder, decoder-only, and encoder-only designs

Transformer vs BERT vs GPT: Key Differences

Model Architecture

To support this pre-training and fine-tuning approach, the GPT-1 model is built on a Transformer (decoder) architecture.

According to the authors, this choice is important for a few reasons. Unlike older models such as LSTMs, Transformers handle long-range dependencies more effectively, meaning they can better understand relationships between words that are far apart in a sentence.

They also rely on self-attention, a mechanism that allows the model to focus on the most relevant parts of the text when processing each word. This helps the model capture context more accurately.

Another key advantage is that Transformers make transfer learning more effective, since the same learned representations can be reused across different tasks with minimal changes.

The paper highlights that, in these transfer learning scenarios, Transformers outperform LSTM-based models.

Figure 1 from “Improving Language Understanding by Generative Pre-Training” (Radford et al., 2018), showing the Transformer architecture and task-specific input transformations.

Key Techniques

Along with the main approach, the authors introduce a few practical techniques that make the model more flexible across tasks.

According to the paper, different tasks are handled by converting them into text-based formats, so they can all be processed in a similar way. This makes it easier to use the same model across multiple problems without redesigning it each time.

Another important point is that the model requires only minimal architectural changes when switching between tasks. Most of the knowledge learned during pre-training is reused as-is.

The authors also include an auxiliary language modeling objective during fine-tuning, which helps the model retain its general understanding of language while adapting to specific tasks.

Key Findings

After training and evaluation, the results weren't just strong – they were surprisingly competitive.

According to the authors, the model outperformed state-of-the-art systems in 9 out of 12 tasks. It also showed clear improvements, including +8.9% in commonsense reasoning and +5.7% in question answering.

Another important observation is that the model performed well across datasets of different sizes, although performance was weaker on some smaller datasets.

This suggests that the pre-training step helped it generalize better, even when labeled data was limited.

In practice, what makes these results significant is that a single model was able to compete with specialized systems that were specifically designed for each individual task.

Figure 2 from “Improving Language Understanding by Generative Pre-Training” (Radford et al., 2018), illustrating performance gains from layer transfer and zero-shot learning behavior.

Conclusions

To wrap things up, this paper introduced a major shift in how AI systems are built.

According to the authors, instead of training a new model from scratch for every task, we can first teach a model the structure of language through pre-training, and then adapt it to specific tasks through fine-tuning. This simple idea turns out to be highly effective.

The key takeaway is that language models can develop a general understanding of text, especially when combined with Transformer architectures and large-scale data. This makes transfer learning practical across many different tasks.

In my view, this is what makes the paper so impactful. It doesn’t just improve performance on a few benchmarks. It changes the overall approach to building AI systems.

This idea later became the foundation for models like GPT-2, GPT-3, and ChatGPT, and continues to shape modern large language models today.

Limitations

Like any approach, this method comes with its own limitations.

According to the paper, one of the main challenges is the need for large amounts of unlabeled data during the pre-training stage, which may not always be easy to get. The model’s performance also depends heavily on how well the fine-tuning step is done.

The authors also note that multi-task learning was not fully explored in this work, leaving some open questions about how well the model can handle multiple tasks at the same time.

In practice, another limitation is that performance can be weaker when working with very small datasets, especially if the fine-tuning process is not carefully handled.

Related Work & Context

To better understand where this paper fits, it helps to look at the ideas it builds on.

According to the authors, earlier approaches such as word embeddings (like Word2Vec and GloVe), LSTM-based language models, and semi-supervised learning had already made progress in understanding language. But these methods were often limited to learning representations at the word level or required more task-specific design.

What this paper does differently is move beyond that. Instead of focusing only on individual words, it learns broader language representations that capture context and meaning across entire sequences. This shift is what enables the model to generalize better across different tasks.

Final Insight

If there’s one idea to take away from this paper, it’s this: you don’t need to teach an AI system every task separately.

According to the authors, once a model learns the structure of language, it can adapt to a wide range of tasks with minimal changes. That shift – from task-specific models to general language understanding – is what makes this work so important.

In my view, this is the moment where things really changed. What started here with GPT-1 became the foundation for the systems we use today, including ChatGPT and other modern language models.

Resources:

• Pytorch Projects for GPT series

• Word2Vec (Mikolov et al., 2013)

• GloVe (Pennington et al., 2014)

• Attention Is All You Need (Vaswani et al., 2017)

• Semi-supervised Sequence Learning (Dai and Le, 2015)

• Universal Language Model Fine-tuning for Text Classification (Howard and Ruder, 2018)

• Deep Contextualized Word Representations (Peters et al., 2018)

• Semi-supervised Multitask Learning for Sequence Labeling (Rei, 2017)

• Skip-Thought Vectors (Kiros et al., 2015)

• Supervised Learning of Universal Sentence Representations (Conneau et al., 2017)

Contact Me

• Github

• X

• Linkedin

This guide will teach you how to create NLP projects from scratch. It includes details on how to organize your workflow, utilize the appropriate tools, and perform typical NLP tasks.

After reading this article, you will understand how to:

• Configure your environment for NLP development.

• Select the appropriate frameworks and libraries for your project.

• Execute fundamental NLP tasks such as sentiment analysis and text classification.

• Create and implement a functional NLP application.

• Diagnose and fix common problems in NLP projects.

Before beginning, you should have some basics at hand already. They include a solid understanding of Python programming and knowledge of the general ideas of machine learning. You should also know how to build algorithms and data structures. Finally, your system should have Python 3.8 or higher installed so you can try running the example snippets.

Table of Contents

• What is Natural Language Processing?

• How NLP Systems Interpret Speech

• Typical NLP tasks

• Conventional Machine Learning Methods for NLP

• How to Use NLP in Various Industries

• How to Choose the Most Effective NLP Tools and Libraries

• How to Prepare and Train NLP systems

• Establishing and Labeling Datasets

• Conclusion

What is Natural Language Processing?

NLP (natural language processing) is a set of methodologies that allow computers to learn to comprehend human language and produce relevant outputs.

NLP manages the intricacy of human communication. In contrast to conventional machine learning, which operates with structured data only, NLP handles unstructured text data.

Specifically, to more accurately comprehend language, NLP systems simultaneously analyze the syntax (which is the arrangement of words and grammar), the semantics (the meanings of specific words and phrases), and interpret context (how adjacent information affects meaning). This allows them to differentiate between various interpretations of identical words, grasp implied messages, and produce responses as relevant as possible.

The ability of machines to process language was demonstrated by early experiments such as the Georgetown-IBM translation in 1954 and the ELIZA chatbot in 1966 (Sources: Szmurlo and Akhtar, MDPI; Hutchins, ResearchGate). With today's tools, any developer can access and use the capabilities of NLP tools.

So why is this important for you? In 2025, the market for NLP, which currently powers chatbots, translation software, and content creation platforms, has reached $42.47 billion. (Source: Precedence Research)

The growth is only accelerating. By 2030, the global NLP market is expected to grow to $439.85 billion. (Source: GrandviewResearch).

Important NLP Concepts to Know

Five interconnected layers generally make up NLP systems. Every layer addresses a distinct language processing problem. (Source: Khatri and others, ResearchGate).

• Analysis of morphology is where you break down words into their most meaningful components by this layer. Words will be broken down into prefixes, roots, and suffixes. For instance, "working" becomes "work" plus "ing." This makes it easier for your system to comprehend word relationships even when they change form.

• Analysis of syntactic structure is where you use grammar rules to determine sentence structure. Here, you construct parse trees that map the grammatical relationships between words. Individual words are represented as leaves, phrases as intermediate nodes, and sentences as roots in the tree.

• Analysis of semantics is where, from the parsed structure, you derive the true meaning.

• You deal with synonyms, antonyms, and homophones as well as word ambiguity. This transforms grammatical structure into meaning.

• Analysis of discourse is where you connect sentences within longer text structures. You'll observe how ideas flow from one paragraph to the next and spot recurring themes. This connects meaning at the sentence level to meaning at the document level.

• Analysis of pragmatics is where you decipher intent and context. You will be able to resolve references, comprehend dialogue structure, and decipher implied meanings. You can process sarcasm, cultural background, and other aspects of everyday communication at this layer.

Understanding these layers gives you the ability to build NLP systems that can manage challenging language tasks in a variety of contexts.

How NLP Systems Interpret Speech

NLP systems use a pipeline to convert raw text into computational meaning. Each step builds on its predecessor, allowing for better analysis of unstructured language data. In this section, I’ll provide real snippets of code you can insert into an editor for training.

Step 1: Text Input

To start, your system will take in raw text that can come in various forms. Potential sources for raw input include emails, social media posts, articles, documents, or transcripts of speeches. The raw data will contain misspellings, crude language, and grammatical mistakes you'll need to circumvent.

Step 2: Text Preprocessing

Next, you’ll need to clean and standardize the input text before your system analyzes it. Your pre-process will likely include some or all of these steps:

• Tokenizing text into single words or subwords

• Removing punctuation marks from the text

• Lower casing all the text

• Removing stop words like "the", "and," and "is."

For example, you can accomplish such a simple form of NLP using Python, but note that you need to import specific libraries (we will discuss them later):

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# Download required NLTK data
nltk.download('punkt')
nltk.download('stopwords')

# Raw text input
text = "The quick brown fox jumps over the lazy dog!"

# Tokenization
tokens = word_tokenize(text.lower())

# Remove punctuation and stop words
stop_words = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word.isalnum() and word not in stop_words]

print(filtered_tokens)
# Output: ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

Step 3: Syntactic Parsing and Analysis

After cleaning, you’ll analyze the text’s grammatical structure by constructing parse trees. While parse trees can vary in complexity, they map the relationships between words, phrases, and clauses. You can leverage part-of-speech tagging information to assign grammatical roles (noun, verb, adjective, and so on) to words, and dependency parsing to learn how related words are linked syntactically.

For example, the code below illustrates how to perform part-of-speech tagging with spaCy, which determines the grammatical function of each word within a sentence.

import spacy

# Load English language model
nlp = spacy.load("en_core_web_sm")

# Process text
doc = nlp("The cat sat on the mat")

# Part-of-speech tagging
for token in doc:
    print(f"{token.text}: {token.pos_}")

# Output:
# The: DET
# cat: NOUN
# sat: VERB
# on: ADP
# the: DET
# mat: NOUN

Step 4: Feature Engineering and Text Representation

Here, you convert words into numerical vectors that computers can parse using embedding or transformer-based techniques to capture similarities and semantic relationships between terms. For instance, this allows your system to understand that the words "kid" and "child" are similar in meaning.

from sentence_transformers import SentenceTransformer

# Load pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Convert sentences to embeddings
sentences = ["The cat sits on the mat", "The feline rests on the rug"]
embeddings = model.encode(sentences)

print(f"Embedding shape: {embeddings.shape}")
# Output: Embedding shape: (2, 384)

Step 5: Modeling and Pattern Recognition

In this part of the process, you’ll use machine learning algorithms to identify patterns from vectorized text. You may use either a traditional machine learning representation or one of the deep learning methods, such as transformers. Your models will learn about patterns in the language, classify the content presented, or extract entities in the text.

To understand this method, let’s see a straightforward example of a text classification model that uses transformers to identify sentiments.

from transformers import pipeline

# Load a pre-trained sentiment analysis model
classifier = pipeline("sentiment-analysis")

# Classify text sentiment
texts = ["I love this product!", "This is terrible and disappointing"]
results = classifier(texts)

for text, result in zip(texts, results):
    print(f"Text: {text}")
    print(f"Sentiment: {result['label']}, Confidence: {result['score']:.2f}\n")

# Output:
# Text: I love this product!
# Sentiment: POSITIVE, Confidence: 0.99
#
# Text: This is terrible and disappointing
# Sentiment: NEGATIVE, Confidence: 0.99

This illustrates how the model detects linguistic patterns for sentiment categorization, which is a typical task in NLP. In subsequent sections, we’ll delve into more specialized modeling techniques tailored for various NLP applications.

Step 6: Evaluation and Deployment

Next, you will evaluate your model from metrics such as precision, recall, and F1 scores. After evaluation, you will deploy your model to production, and the model will continue to learn from data produced from real-world text. Here’s an example of how it’s done:

from sklearn.metrics import classification_report

# Example predictions vs actual labels
y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 0, 0, 1]

# Generate evaluation metrics
print(classification_report(y_true, y_pred))

Typical NLP Tasks

Natural Language Understanding (NLU) tasks

Natural Language Understanding (NLU) tasks deal with actually understanding what people are communicating about. There are several elements involved in this process.

Sentiment analysis and text classification

Here, you recognize and categorize documents according to emotion. Your engine identifies whether the text conveys a positive, negative, or neutral sentiment. Then, it autonomously filters content across digital platforms. Consider this example:

from transformers import pipeline

# Load sentiment analysis pipeline
classifier = pipeline("sentiment-analysis")

# Analyze sentiment
result = classifier("I love this product! It works great.")
print(result)
# Output: [{'label': 'POSITIVE', 'score': 0.9998}]

Named Entity Recognition (NER)

NER is a pipeline that involves automatically identifying and classifying distinct pieces of information within a body of text. This includes names of people, locations, organizations, dates, and monetary figures.

Your NER system analyzes unstructured text to accurately label these entities, converting raw data into a structured format that can be easily analyzed. Your algorithm can also uncover relationships among these entities, allowing you to gain valuable insights from extensive amounts of text.

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple Inc. was founded by Steve Jobs in Cupertino, California.")

for ent in doc.ents:
     print(f"{ent.text}: {ent.label_}")

# Output:
# Apple Inc.: ORG
# Steve Jobs: PERSON
# Cupertino: GPE
# California: GPE

Question answering

You can create systems that consume natural language questions and retrieve appropriate answers. Your system can also use entailment and contradiction detection to analyze the logical relationships between text blocks.

Intent recognition

You can recognize user intentions in conversational domains. Your dialog systems are conscious of the user’s goals, allowing buttons or voices to respond in kind.

Now, let’s move on to some general natural language-related tasks.

General Natural Language Tasks

This class of tasks pulls together some aspects of understanding while dealing with generation as well.

Machine translation

You can translate text across multiple languages while preserving context and meaning. Neural networks use encoder-decoder architectures to create linguistic outputs in the target language.

Let’s see how it’s done with the MarianMTModel and MarianTokenizer models:

from transformers import MarianMTModel, MarianTokenizer

# Load translation model
model_name = 'Helsinki-NLP/opus-mt-en-es'
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)

# Translate English to German
text = "Hello, how are you?"
translated = model.generate(**tokenizer(text, return_tensors="pt", padding=True))
print(tokenizer.decode(translated[0], skip_special_tokens=True))
# Output: Hallo, wie geht's dir?

Text summarization

Often you’ll need to shorten a long document into a more accessible summary – this is text summarization, and it’s a common NLP task. Your system retains key details and coherence while reducing the length of a document.

Speech recognition and text-to-speech

Using these techniques, you can turn speech into text (speech recognition) or text into natural audio (text-to-speech). These tasks close the gap between text and audio modalities.

Syntactic parsing

Here, you examine the grammatical construction to determine the syntactic relationships between words in the sentence. This critical task gives a structural analysis of the text to support more complex understanding tasks.

These tasks, when combined, create powerful applications for different industries and use cases in Natural Language Processing.

Conventional Machine Learning Methods for NLP

Instead of relying on manually created linguistic rules (where programmers specify patterns like "if a word ends with '-ing', it is likely a verb" or "sentences containing 'not' followed by positive words suggest negative sentiment"), ML approaches apply statistical methods to discover patterns automatically within the data.

These methods learn through examples and don’t require human experts to define every potential language structure explicitly. As a result, they are more scalable and adaptable across different languages and fields. Let’s look at some of them now.

Logistic Regression

For tasks involving binary classification, you can use logistic regression. Based on input features, it predicts event probability by learning linear decision boundaries. Consider the following example:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Sample data
texts = ["This is spam", "Normal email", "Buy now!", "Meeting tomorrow"]
labels = [1, 0, 1, 0]  # 1 = spam, 0 = not spam

# Convert text to features
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(texts)

# Train model
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.25)
model = LogisticRegression()
model.fit(X_train, y_train)

# Predict
new_text = vectorizer.transform(["Free money now"])
prediction = model.predict(new_text)
print(f"Prediction: {'Spam' if prediction[0] == 1 else 'Not Spam'}")

Typical uses include toxicity classification, sentiment analysis, and spam detection.

Naive Bayes

Using the premise that words are independent, Naive Bayes applies Bayes' Theorem.

To classify documents, it computes:

$$P(label|text) = P(label) × P(text|label) / P(text)$$

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

# Training data
texts = ["I love this product", "Terrible service", "Amazing quality", "Waste of money"]
labels = [1, 0, 1, 0]  # 1 = positive, 0 = negative

# Vectorize text
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Train Naive Bayes
clf = MultinomialNB()
clf.fit(X, labels)

# Predict sentiment
new_review = vectorizer.transform(["Great purchase"])
print(f"Sentiment: {'Positive' if clf.predict(new_review)[0] == 1 else 'Negative'}")

Common uses for this algorithm that you can try are spam detection and bug detection in software.

Decision Trees

Decision trees partition data sets recursively by choosing the feature that maximizes information gain at each split in a way that builds interpretable, tree-like models. Each internal node is a decision (on a feature), each branch is an outcome of the decision, and each leaf node is a classification.

Decision trees are especially useful for text classification and feature selection because the decision tree allows you to trace exactly how the model made the predicted classification.

Let’s see a code example that shows how the decision tree learns which words, converted to TF-IDF features, predict whether the sentiment of the text in question is positive or negative:

from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split

# Sample text data with labels
texts = [
    "I love this movie, it's fantastic",
    "Terrible film, waste of time",
    "Amazing performance and great story",
    "Boring and disappointing",
    "Excellent cinematography and acting",
    "Awful, would not recommend"
]
labels = [1, 0, 1, 0, 1, 0]  # 1 = positive, 0 = negative

# Convert text to TF-IDF features
vectorizer = TfidfVectorizer(max_features=20)
X = vectorizer.fit_transform(texts)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.3, random_state=42)

# Train decision tree
clf = DecisionTreeClassifier(max_depth=3, random_state=42)
clf.fit(X_train, y_train)

# Make predictions
test_text = ["This movie is wonderful"]
test_vector = vectorizer.transform(test_text)
prediction = clf.predict(test_vector)

print(f"Text: {test_text[0]}")
print(f"Predicted sentiment: {'Positive' if prediction[0] == 1 else 'Negative'}")
print(f"Model accuracy: {clf.score(X_test, y_test):.2f}")

At each node, the decision tree asks a question: "Does the text have a high TF-IDF score for 'wonderful'?" Then the tree will branch accordingly based on the answer to the question until reaching a classification.

One key parameter in the above code is max_depth=3 – without it, the tree may become too complex and overfit. The parameter limits the complexity of the tree.

Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) automatically determines thematic structures in large collections of texts by treating documents as probabilistic mixtures of topics, and topics as distributions over words. This discovery approach uses unsupervised learning, which means that no labeled training data are needed to discover structured but hidden themes. LDA is suited for exploratory text analysis and organization of data in significant amounts of text.

Let’s see some code that generates a word frequency matrix from documents. In this code, LDA identifies two underlying topics based on patterns of word co-occurrence, a process that is a type of clustering analysis for text documents.

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

# Document collection
documents = [
    "Machine learning algorithms process data",
    "Deep learning uses neural networks",
    "Python is great for data science",
    "Neural networks learn from examples"
]

# Create document-term matrix
vectorizer = CountVectorizer(max_features=50)
doc_term_matrix = vectorizer.fit_transform(documents)

# Train LDA model
lda = LatentDirichletAllocation(n_components=2, random_state=42)
lda.fit(doc_term_matrix)

# Display topics
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words_idx = topic.argsort()[-5:]
    top_words = [feature_names[i] for i in top_words_idx]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

In this illustration, we could interpret Topic 0 as "data science and algorithms," and Topic 1 as "neural networks and deep learning." The LDA model will assign, in a mixed model fashion, a probability distribution of each document falling under the two topics. For instance, a document titled "neural networks for data processing" could be considered 60% Topic 1 and 40% Topic 0.

Deep Learning Models

Deep learning models automatically extract hierarchical representations from raw text without manual feature engineering. Applying deep learning to language processing is important because language understanding requires modeling not just individual words, but also phrases, sentences, and the context as a whole.

A neural architecture achieves this modeling by learning multiple layers of abstraction and can interpret the sentences in more complex ways, such as sentiment, intent, or topic.

Let’s illustrate how it works with an example showing a simplified deep learning model that can be used for text classification using TensorFlow/Keras. This specific example uses an embedding layer to map words to dense vectors that capture their semantic meaning, as well as a Bidirectional LSTM layer which is able to capture information from the past and future of a sequence and outputs to a Dense layer for binary classification.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Example sentences and labels
texts = ["I like this movie", "I hate this movie"]
labels = [1, 0]  # 1 = positive, 0 = negative

# Tokenize text and pad sequences
tokenizer = Tokenizer(num_words=50)
tokenizer.fit_on_texts(texts)
X = pad_sequences(tokenizer.texts_to_sequences(texts), maxlen=5)

# Simple model: embedding + LSTM + output
model = Sequential([
    Embedding(input_dim=50, output_dim=8, input_length=5),
    LSTM(4),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(X, labels, epochs=5, verbose=0)

# Predict sentiment for new sentence
test_text = ["I love this"]
test_seq = pad_sequences(tokenizer.texts_to_sequences(test_text), maxlen=5)
pred = model.predict(test_seq)[0][0]

print(f"Sentiment score: {pred:.2f} (1=positive, 0=negative)")

The model learns these patterns from example sentences that have been labeled as positive or negative and then uses those learned patterns to predict the sentiment of new text input. This is an example of how deep learning models learn to automatically represent the text that is processed and then use that representation to interpret sequences of text for classification purposes, without any feature engineering.

Convolutional Neural Networks (CNNs)

CNNs apply the same pattern-detecting framework to the text as they do to image recognition. CNNs see documents as sequences, and when a convolutional filter is applied across the text, it detects patterns for various types of features, such as n-grams (sequences of symbols that are adjacent to one another), and meaningful phrases.

CNNs encompass multi-filter layers to detect different features. Each filter layer detects features that are continuously more abstract, going from simple combinations of words to capturing combinations of words that are consistently used in semantic patterns, creating an effective use for the text classification task. (Source: Yoon Kim)

Here is an example of the convolutional layer scanning through the text using filters. It detects meaningful patterns established through previous learning, such as the word "excellent" or "terrible waste," learning to treat each combination of words as expressing a positive or negative sentiment during a final classification step.

import tensorflow as tf
from tensorflow.keras.layers import Embedding, Conv1D, GlobalMaxPooling1D, Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Sample training data
texts = [
    "This movie is excellent and entertaining",
    "Terrible film, complete waste",
    "Amazing story and great acting",
    "Boring and poorly made"
]
labels = [1, 0, 1, 0]  # 1 = positive, 0 = negative

# Tokenize and pad sequences
tokenizer = Tokenizer(num_words=100)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
X = pad_sequences(sequences, maxlen=10)

# Build CNN model
model = Sequential([
    Embedding(input_dim=100, output_dim=32, input_length=10),  # Convert words to dense vectors
    Conv1D(filters=64, kernel_size=3, activation='relu'),  # Detect 3-word patterns
    GlobalMaxPooling1D(),  # Extract most important features
    Dense(1, activation='sigmoid')  # Binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X, labels, epochs=10, verbose=0)

# Test prediction
test_text = ["wonderful movie with great plot"]
test_seq = tokenizer.texts_to_sequences(test_text)
test_pad = pad_sequences(test_seq, maxlen=10)
prediction = model.predict(test_pad)

print(f"Sentiment probability: {prediction[0][0]:.2f}")
print(f"Classification: {'Positive' if prediction[0][0] > 0.5 else 'Negative'}")

The pooling layer analyzes this filtered text and brings forth the most substantial signals for measuring positive versus negative sentiments from the convolutional text features of the previous steps.

Recurrent Neural Networks (RNNs)

RNNs handle sequential data by tracking hidden states that reflect dependencies over time. At each time step, the RNN receives the current word and the previous hidden state as input and changes the hidden state, which reflects the accumulated context.

Here's a concrete example where, as the RNN reads the next word from left to right, it updates its hidden state to maintain the context.

import tensorflow as tf
from tensorflow.keras.layers import Embedding, SimpleRNN, Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Training data
texts = [
    "I really enjoyed this book",
    "The plot was confusing and dull",
    "Fantastic read, highly recommend",
    "Disappointing and poorly written"
]
labels = [1, 0, 1, 0]

# Prepare data
tokenizer = Tokenizer(num_words=100)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
X = pad_sequences(sequences, maxlen=10)

# Build RNN model
model = Sequential([
    Embedding(input_dim=100, output_dim=32, input_length=10),
    SimpleRNN(units=64, return_sequences=False),  # Process sequence and maintain hidden state
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X, labels, epochs=20, verbose=0)

# Test
test_text = ["amazing story highly engaging"]
test_seq = tokenizer.texts_to_sequences(test_text)
test_pad = pad_sequences(test_seq, maxlen=10)
prediction = model.predict(test_pad)

print(f"Sentiment probability: {prediction[0][0]:.2f}")

Longer sentences are more complex because the information contained in the hidden state is lost over an ever-increasing number of time steps. That's the motivation for the more sophisticated architectures of long short-term memory (LSTM) and gated recurrent unit (GRU).

Encoder-Decoder Architectures

These architectures have two neural networks which work together. The first encoder neural network takes the input text and reduces it to a dense, fixed-size representation but encodes the essential meaning. Then a second decoder network generates an output text based on the meaning representation.

These architectures learn a compressed representation of the input data, and they are often used for:

• Dimensionality reductions.

• Feature learning.

• Document clustering.

• Sequence-to-sequence tasks (for example, translations or summarizations).

The following example illustrates how to use a Text-to-Text Transfer Transformer (T5) encoder-decoder model to translate English into German. The encoder takes the input English sentence and builds its internal representation of the text, while the decoder generates the German translation based on the representation:

from transformers import T5Tokenizer, T5ForConditionalGeneration

# Load T5 model for text generation
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

# Translate text
input_text = "translate English to German: Hello, how are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

# Generate translation
outputs = model.generate(input_ids)
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Translation: {translation}")

This architecture solves the issue of variable-length input and output in a very elegant way. The encoding neural network reduces the sentence to a fixed-size representation regardless of the input length. Subsequently, the decoder generates an output for whatever length it determines is appropriate based on the input length, whether it’s one sentence or six sentences.

Transformer Models

Unlike RNNs, in which text is processed sequentially (one word at a time), transformers use a processing mechanism that evaluates the sequence in parallel. This means that the transformer can simultaneously consider all of the words in a sentence and directly compute relationships between any two words, even apart in distance.

In the example below, "The girl didn't go to school because she was ill," the model directly connects "she" with "girl" despite other words between these two. This brings a faster ability to train on information and helps avoid the degradation of information through time steps. (Source: Vaswani and others).

In the example, BERT, one of the most well-known transformer models, performs sentiment classification on a text. Here’s how the transformer justifies text classification by understanding pre-trained language and only using minimal additional training:

from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Load pre-trained BERT
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# Prepare input
text = "This movie was fantastic!"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

# Get predictions
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

print(f"Prediction scores: {predictions}")

In the above code, the tokenizer converts the sequence of text into numerical tokens (which BERT understands) and special tokens, such as [CLS] (for classification), at the beginning of the list of tokens. BERT then models the entire length of the sentence using multiple layers, where each layer is able to learn abstract representations of meaning in each layer.

How to Use NLP in Various Industries

You can use NLP to solve issues in almost any sector, and there are many sector-specific implementations. You can choose to try the snippets below depending on the area you’re most interested in.

Tourism and Hospitality

You can use NLP techniques to build intelligent booking systems that comprehend natural language requests from clients. Important uses you can apply:

• Sentiment analysis monitors consumer feedback to spot patterns in satisfaction and problems with customer service.

• NER-enabled chatbots retrieve dates and locations from consumer inquiries such as "I need a flight to Paris next Tuesday."

Here’s an example:

from transformers import pipeline

# Load NER model
ner = pipeline("ner", grouped_entities=True)

# Extract booking information
query = "I need a hotel in London from December 15 to December 20"
entities = ner(query)

for entity in entities:
    print(f"{entity['entity_group']}: {entity['word']}")
# Output: LOC: London

Through machine translation, you can provide multilingual support to your customers in various languages. And an intent classification model based on BERT automatically identifies how to route your customers for service or makes bookings automatically for them.

Logistics and Supply Chain

You can automate document processing via NLP and optimize delivery routing using predictive algorithms. Let’s see the common areas of application:

• You can use OCR to process documents to automatically extract shipping information from invoices and customs forms. Here’s an example:

import pytesseract
from PIL import Image

# Extract text from shipping document
image = Image.open('invoice.png')
text = pytesseract.image_to_string(image)

# Parse extracted information
# (Add parsing logic based on document structure)

• Text classification can place shipments into categories based on descriptions, allowing for recursive sorting of shipments for transport.

• Predictive routing models can use historical delivery data and weather reports to create delivery schedules.

• Natural Language Generation takes technical data across logistics to create user-friendly tracking updates.

Retail and eCommerce

Within the eCommerse operations, you can personalize your customers’ shopping experience and optimize pricing with NLP techniques.

Some key applications that you can benefit from:

• Recommendation engines utilize word embeddings to learn product descriptions and corresponding user reviews to suggest relevant items. Here’s how, for instance:

from sentence_transformers import SentenceTransformer, util

# Load embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Product descriptions
products = [
    "Wireless Bluetooth headphones with noise cancellation",
    "USB-C charging cable for smartphones",
    "Noise-cancelling earbuds with long battery life"
]

# User query
query = "I need headphones that block outside noise"

# Calculate similarities
query_embedding = model.encode(query)
product_embeddings = model.encode(products)
similarities = util.cos_sim(query_embedding, product_embeddings)

# Find best match
best_match_idx = similarities.argmax()
print(f"Recommended product: {products[best_match_idx]}")

• Chatbots that include dialogue management can respond to inquiries from customers about products, orders, and returns.

• Sentiment analysis on social media tracks brand health and customer sentiment in real-time.

• Price optimization algorithms analyze competitors' pricing and market signals to change prices in real-time.

• Demand forecasting analyzes news and social sentiment to predict inventory needs.

Healthcare

Healthcare, with the great amount of data from patient records, is a natural area for NLP to optimize. You can support clinical decision-making and process medical records using specialized NLP systems.

Here are a few of the possible uses and an example:

• Clinical NER identifies conditions, medications, and treatments mentioned in the clinicians' notes within electronic health records. For instance:

import spacy

# Load medical NER model (requires installation of scispacy)
# pip install scispacy
# pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/en_core_sci_sm-0.5.1.tar.gz

nlp = spacy.load("en_core_sci_sm")

# Process clinical note
text = "Patient presents with hypertension and type 2 diabetes. Prescribed metformin 500mg."
doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text}: {ent.label_}")

• Clinical decision support systems scan descriptions of symptoms and provide suggestions for potential diagnoses to help a physician's decision-making.

• Literature mining scans clinical studies and identifies new treatment patterns or potential drug discovery targets.

Of course, NLP can also be used in patient assistance chatbots, as they can comprehend natural language and its nuances.

Financial Services

In the finance sector, there are unique bottlenecks you might face. Financial data security gaps and the risks of fraud are among the most threatening ones, as well as the regulatory fines that come with these issues.

With NLP, you can improve security mechanisms and create systems for detecting fraud.

You can also detect phishing attacks with high accuracy with ML classifiers and NLP using CNNs and RNNs combined. (Source: Saidat and others, ResearchGate).

Some other use cases include:

• Document analysis processes loans applications/contract to assess credit risk by automated analysis of documents.

• Fraud detection systems analyze transaction data and communication dat to identify suspicious activity. For example:

from transformers import pipeline

# Load zero-shot classification model
classifier = pipeline("zero-shot-classification")

# Analyze transaction description
description = "Wire transfer to offshore account for investment opportunity"
candidate_labels = ["legitimate transaction", "potential fraud", "suspicious activity"]

result = classifier(description, candidate_labels)
print(f"Classification: {result['labels'][0]} (Score: {result['scores'][0]:.4f})")

• Automated compliance monitoring scans messages for adherence with regulations.

• Robo-advisors leverage natural language interfaces to engage with clients while providing investment advice.

Apart from these uses, conventional chatbots also provide assistance by using NLP techniques. OCR algorithms are widely used for document analysis – but we’ve mentioned those other use cases, so we won’t discuss them further here.

Legal Industry and Compliance Regulations

Even more than financial services, the legal sector depends on strict requirements, laws, and regulations. NLP techniques can help you improve safety, security, and efficiency in processing legal documents.

Key examples of how it can be applied:

• Multimodal authentication is a secure identity verification process consisting of a combination of facial recognition, voice recognition, and natural language processing.

• Speaker recognition uses automatic speech-to-text encoding and intent recognition to process a verbal response to security questions.

• Contract analysis scans legal documents to identify key terms, deliverables, and dates, extracting that information automatically. As an example, you can try the following snippet with the spaCy library installed:

import spacy

nlp = spacy.load("en_core_web_sm")

# Extract dates and obligations from contract
contract_text = "The agreement shall commence on January 1, 2026 and continue for a period of 12 months."
doc = nlp(contract_text)

for ent in doc.ents:
    if ent.label_ in ["DATE", "CARDINAL"]:
        print(f"{ent.label_}: {ent.text}")

• Compliance monitoring looks for possible regulatory infractions in legal communications.

These real-world examples show how NLP can be used to solve practical business issues and boost operational effectiveness. You can modify the samples to your case or discover other potential uses, but these are the most widespread ones for you to try.

How to Choose the Most Effective NLP Tools and Libraries

There is a great variety of tools and libraries that can help you learn how to use NLP or that you can use to implement NLP into a project. You should select the appropriate tools considering your project needs and background in the associated technologies.

Below are some popular tools you can choose to learn or check out, along with tips about when they’re most useful.

Hugging Face Transformers

Hugging Face Transformers has thousands of pre-trained models for text generation, classification, and question answering. It gives you more than 100 languages supported and is compatible with PyTorch and TensorFlow.

It also provides model hosting and datasets, and allows collaboration with community members. It will assist you with deep learning NLP applications that require top-notch software to implement the algorithms.

NLTK (Natural Language Toolkit)

NLTK is the primary package in Python for education and research concerning NLP. Developed at the University of Pennsylvania, it provides extensive packages for your tokenizers, stemmers, parsers, and semantic reasoning. It’s a great choice if you need to get learning concepts in NLP or conduct research projects.

spaCy

spaCy is a Python library developed to be production-ready, and has the fastest syntactic parser. It’s constructed using Cython for optimal performance and offers excellent named entity recognition. It will fit well if you need strong dependency parsing and a developer-friendly API. It’s easy for you to use spaCy for quick prototyping.

Google Cloud NLP

Google Cloud NLP also offers enterprise-level API services. It will fit your project if you need sentiment analysis, entity recognition, syntax analysis, automatic language identification, and simple, trouble-free scaling. And if you’re already in the Google Cloud ecosystem working with big volumes of customer feedback, it’s just what you need.

Amazon Comprehend

Comprehend is a fully-managed service from AWS for text analysis in the cloud. It supports the major functions you might want to cover: sentiment analysis, entity recognition, topic modeling, built-in protection of personally identifiable information (PII), and auto-scaling. And it’s perfect if you need a built-in integration with the AWS suite.

IBM Watson

Watson has NLP models specific to regulated industries (healthcare, finance, and so on). Its library offers pre-trained models in 20 programming languages. Its top features you can use are strong data controls, reliable REST API access, and truly compliance-ready outputs. These makes this tool a great choice if you’re in healthcare, finance, or legal industries.

TextBlob

TextBlob is a simplified library that’s a great option if you’re a beginner. It’s a user-friendly Python library for common NLP tasks. For your convenience, it offers a simplified API design, but still provides decent sentiment analysis, translation, spelling correction, and noun phrase extraction. Apart from beginner projects, it will fit your quick prototypes creation needs.

How to Prepare and Train NLP Systems

As you get ready to train your NLP model, you’ll need to prepare your data accurately to ensure its quality doesn’t hinder the outputs. Remember that poor quality data results in a poor performing model, so you’ll want to make sure you have solid data.

Understanding Data Quality and Preprocessing

Raw text data is messy and unstructured. It contains typos, slang, and irrelevant information that degrades the performance of your model.

Preprocessing is the operation that takes messy data and converts it into clean, structured text that models can accept as input.

Research shows that 85.4% of NLP research studies utilized some sort of restructuring/preprocessing to allow NLP models to process raw text. The key data quality components that were essential included accuracy (68.3%), relevance (34.1%) and comparability (31.7%). (Source: Nesca and others, NCBI).

Preprocessing comes down to a specific list of tasks you’ll need to perform. Let’s break them down.

Text Cleaning

Text cleaning is the process of standardizing the text format by removing anything that may affect model training. Raw text often contains extra elements (HTML tags, URLs, special characters, inconsistent use of capitalization, and excess whitespace) that add noise to your data.

The following example shows a cleaning pipeline that removes the above-mentioned elements. This function performs multiple steps of text cleaning:

import re

def clean_text(text):
    # Convert to lowercase
    text = text.lower()

    # Remove HTML tags
    text = re.sub(r'<[^>]+>', '', text)    

    # Remove URLs
    text = re.sub(r'http\S+|www.\S+', '', text)    

    # Remove special characters and numbers
    text = re.sub(r'[^a-zA-Z\s]', '', text)    

    # Remove extra whitespace
    text = ' '.join(text.split())    

    return text

# Example
raw_text = "Check out https://example.com! It's <b>AMAZING</b> :-)"
cleaned = clean_text(raw_text)
print(cleaned)
# Output: check out its amazing

The first step the model made was to convert everything to lower case for uniformity. Then it used regular expressions to parse the text to remove HTML tags, URLs, special characters, and numbers. Finally, it normalized whitespace by splitting and joining the text back together. The output is clean text, in a standardized format you can use for tokenization.

Tokenization

The next step is to divide the text into smaller digestible chunks that are easier for ML models to understand. These chunks are known as tokens.

Tokenization comes in three varieties:

• Word tokenization separates text according to punctuation and whitespace.

• Sentence tokenization uses punctuation cues to divide text into sentences.

• Subword tokenization breaks words up into more manageable chunks.

The example below addresses examples of word and sentence tokenizing by using NLTK (Natural Language Toolkit).

from nltk.tokenize import word_tokenize, sent_tokenize

text = "Natural language processing is exciting! It helps computers understand text."

# Word tokenization
words = word_tokenize(text)
print(f"Words: {words}")

# Sentence tokenization
sentences = sent_tokenize(text)
print(f"Sentences: {sentences}")

Notice that after word tokenization was performed, the punctuation marks '!' or '.' were considered individual tokens, as punctuation conveys meaning. Sentence tokenization correctly identified the boundaries of the two sentences, and despite the presence of an exclamation mark, it indicated that it had more complex rules beyond just splitting based on periods.

Stop Word Removal

Here, you reduce the text to the meaning without any extra details. You can do this by removing the commonly used words that have little semantic value – the “stop words”.

Common stop words include articles, prepositions, pronouns, auxiliary verbs and conjunctions. Here’s how to do it:

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download('stopwords')

text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text.lower())

stop_words = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word not in stop_words]

print(f"Original: {tokens}")
print(f"Filtered: {filtered_tokens}")
# Output: ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

Stemming and Lemmatization

In this next step, you’ll process the text further by reducing words to their root form. This will treat words with similar variations as a single token.

• To be precise, stemming is simply using heuristic rules that remove the endings of words. For example (running|runs|run) → run.

• Lemmatization uses morphological analysis and vocabulary. For example, (children|mice) → child|mouse.

Lemmatization typically gives better, more accurate outcomes (but calls for more computation at the same time).

Here’s how you can apply both:

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

words = ["running", "runs", "ran", "children", "better"]

print("Stemming:")
for word in words:
    print(f"{word} -> {stemmer.stem(word)}")

print("\nLemmatization:")
for word in words:
    print(f"{word} -> {lemmatizer.lemmatize(word, pos='v')}")

Expanding Contractions

AI systems need standardization. And “don’ts” and “you’res” are unacceptable for them. This is why, typically, you would want to expand the contraction to the real word for standardization. Here’s how you can do that:

import contractions

text = "I can't believe it's already 2025. We'll see what happens."
expanded = contractions.fix(text)
print(expanded)
# Output: I cannot believe it is already 2025. We will see what happens.

Correcting Spelling Errors

Orthographic, spelling, or grammar errors shouldn’t be in the data that you feed to your ML models. You can make corrections to these errors using statistical language models, which can predict the most likely intended word, edit distance algorithms that can find the closest valid word, or neural approaches that can learn patterns of common errors.

For example, let’s see how TextBlob, a library that uses a mix of a dictionary-based approach and contextual probability, detects and corrects misspellings.

from textblob import TextBlob

text = "Natral languag procesing is powrful"
corrected = TextBlob(text).correct()
print(f"Original: {text}")
print(f"Corrected: {corrected}")

TextBlob analyzes each word, identifies which ones are not in its dictionary, calculates edit distances, finds the most similar valid words, and selects corrections based on the frequency of word use in context.

Parts-of-Speech Tagging

Parts-of-speech tagging (POS) refers to assigning grammatical classification to words based on their role in a sentence. This is important because the same word can function as a different part of speech depending on the context. For example, "walk" can be a noun (like "an evening walk") or a verb (like "I walk to the office").

POS taggers rely on statistical models trained to predict the most likely grammatical role for a word given the context. The following code shows POS tagging using NLTK, which applies a pre-trained model that will tag grammatical structure.

import nltk

text = "The cat sat on the mat"
tokens = nltk.word_tokenize(text)
pos_tags = nltk.pos_tag(tokens)

for word, tag in pos_tags:
    print(f"{word}: {tag}")
# Output:
# The: DT (Determiner)
# cat: NN (Noun)
# sat: VBD (Verb, past tense)

The function pos_tag() assesses each token and assigns it a standardized notation. For example, DT indicates determiners (such as "the"), NN indicates singular nouns, VBD indicates past tense verbs, and IN indicates prepositions. The tagger can also use context to make these decisions: it can determine that "sat" is VBD and not NN because it appears after a noun and before a preposition, all of which are typical patterns of the English sentence.

Establishing and Labeling Datasets

For supervised learning tasks such as sentiment analysis, NER, or classification, unlabeled data is useless.

This is why, to create training datasets, you must annotate raw data with relevant labels. Models can learn patterns and make predictions thanks to this "ground truth." Let’s define the most common methods for labeling.

Automated Labeling Based on Libraries

You don’t always have to create labels from scratch. Libraries like TextBlob have built-in sentiment analysis models trained on large datasets that label text. They run a polarity score (a number that represents sentiment) and assign a categorical label.

For example, TextBlob sees a word choice, modifiers in particular (words like "very" or "not"), and grammatical patterns to run a polarity score from -1 (most negative) to +1 (most positive), with zero meaning a neutral sentiment.

In this example, we’re automatically labeling sentiments based on a pre-trained TextBlob sentiment analyzer:

from textblob import TextBlob

def label_sentiment(text):
    blob = TextBlob(text)
    polarity = blob.sentiment.polarity

    if polarity < 0:
        return "negative"
    elif polarity == 0:
        return "neutral"
    else:
        return "positive"

# Example
texts = [
    "I love this product!",
    "It's okay, nothing special",
    "Terrible experience, very disappointed"
]

for text in texts:
    label = label_sentiment(text)
    print(f"{text} -> {label}")

Manual Labeling

In many cases, automated library-based labeling isn’t an option. For domain-specific standards, you should annotate your data by hand to ensure accuracy and relevance.

For projects involving manual labeling:

• Establish precise labeling standards. Provide an annotation guideline that defines each label with clear criteria and edge cases. As an example, if annotating for customer support tickets, an example of potential criteria should be that "I need help resetting my password" is "Technical support," and another example is "When will my order arrive?" which is an example of "Order inquiry."

• For quality control, use several annotators. Have 2-3 people label the same data samples independently. For example, if annotating for medical symptoms, having multiple annotators better reduces the chance of bias from one person's labeling and may protect against clerical errors.

• Determine the inter-annotator agreement. Calculate Cohen's kappa or Fleiss' kappa scores to measure the consistency of agreement among annotators. A score of above 0.80 would signify very good agreement, while a score below 0.60 would indicate that the labeling guidelines were not clear enough to the annotators.

• Give instructions and illustrations. Create a reference document with 20-30 examples you pre-labeled, showing examples of typical use cases and edge cases. For example, in a sentiment analysis, case you can provide examples of when a sentiment would be neutral, with an example such as "This product is fine, I guess," even though it may have seemed slightly negative. A sentiment is also classified as a good positive example even though it contains two negatives: "Not bad at all.”

This is the gold standard for high-quality datasets, but it’s also time-consuming and expensive – labeling 10,000 customer reviews might take a week if done manually.

Approaches with Semi-Supervision

There are instances where you should combine the previous approaches. This semi-supervised method uses a small, manually labeled data set (high-quality data) and a large pool of unlabeled data (cheap, large amounts of data).

The method operates through iterative self-training, where you first train the model on your small dataset of signed data, then predict the labels on the unlabeled data using this model, then add the most confident predictions to your training data during training, and retrain. The self-training process is then repeated, improving and expanding your labeled data set gradually.

Here is an example of self-training in practice: this code demonstrates the semi-supervised workflow.

from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.svm import SVC

# Small labeled dataset + large unlabeled dataset
X_labeled = [[1, 2], [3, 4], [5, 6]]
y_labeled = [0, 1, 0]
X_unlabeled = [[2, 3], [4, 5], [6, 7]]

# Combine datasets (-1 represents unlabeled)
X_train = X_labeled + X_unlabeled
y_train = y_labeled + [-1, -1, -1]

# Self-training classifier
base_classifier = SVC(probability=True, gamma='auto')
self_training = SelfTrainingClassifier(base_classifier)
self_training.fit(X_train, y_train)

The code shows a version of the SelfTrainingClassifier that first trains on the three labeled examples, then uses the model to predict inputs and labels for the unlabeled data. The classifier then selects predictions where it has high confidence (for example, predictions that are >90% probability) while using them as newly signed data. The classifier then re-trains itself, and the process continues.

So how do you decide which approach will fit your needs? In most cases, the optimal one will depend on the following aspects:

• Available budget and time.

• Desired accuracy.

• Size of the dataset.

• Complexity of the domain.

As you see, approaches vary and can be mixed now and then. The key thing is to make sure the inputs for final pre-generation processing are cleaned, standardized, and labeled. Remember the key principle: “garbage in, garbage out”. Send gold instead, and good luck!

Conclusion

At this point, you should know the basics of working with NLP projects.

Throughout this article, you've learned:

• How to set up your NLP development environment using a set of tools and libraries.

• The five parts of NLP systems and how they are used to process language.

• How to conduct common tasks like NER, sentiment analysis, and text classification.

• How to choose the library to use to accommodate your project needs.

• How to prepare and label datasets for training.

• How to find the key practical applications of NLP tailored to your industry and use case.

Next steps

• Try to start with a simple project, like sentiment analysis, that uses pre-trained models.

• Practice preprocessing methods with your own text data.

• Use and try different libraries to see how to get the best output for your project.

• Build a full pipeline from preparing text data to deploying models.

• Continue to practice and see advanced applications like transformer models and fine-tuning.

Most importantly, keep in mind that NLP is an iterative process. Start small, test appropriately to get it to work, and then build in complexity when you are more comfortable and sure of your abilities and familiarity with the practices.

About the author

Hope you enjoyed the article and found it helpful. I’ve been a contributor to freeCodeCamp for more than 8 years, and to make this piece more precise and detailed, I used some expert help.

I’m grateful for the technical ideas of my co-workers at COAX Software who wished to stay anonymous. The company is a well-regarded AI/ML development company.

To find out more about me and read more content on tech and digital, you can visit my LinkedIn page.

Over time, this academic interest evolved into a working prototype that combines on-device AI and cloud AI to describe images and translate them into English meanings. The idea was simple: I wanted to build a lightweight web app that recognized Makaton gestures or symbols and instantly provided an English interpretation.

In this article, I’ll walk you through how I built my Makaton AI Companion, a single-page web app powered by Gemini Nano (on-device) and the Gemini API (cloud). You’ll see how it works, how I solved common issues like CORS and API model errors, and how this small project became part of my journey toward AI for accessibility.

By the end of this article, you will be able to:

• Understand the core concept behind Makaton and why it’s important in accessibility and inclusive education.

• Learn how to combine on-device AI (Gemini Nano) and cloud-based AI (Gemini API) in a single web project.

• Build a functional AI-powered web app that can describe images and map them to predefined English meanings.

• Discover how to handle common errors such as model endpoint issues, missing API keys, and CORS restrictions when working with generative AI APIs.

• Learn how to store API keys locally for user privacy using localStorage.

• Use browser speech synthesis to convert the AI-generated English meanings into spoken output.

Table of Contents

• Tools and Tech Stack

• Building the App Step by Step

• How to Fix the Common Issues

• Demo: The Makaton AI Companion in Action

• Broader Reflections

• Conclusion

Tools and Tech Stack

To build the Makaton AI Companion, I wanted something lightweight, fast to prototype, and easy for anyone to run without complicated dependencies. I chose a plain web stack with a focus on accessibility and transparency.

Here’s what I used:

Frontend

• HTML + CSS + JavaScript (Vanilla): No frameworks, just clean and understandable code that any beginner can follow.

• A single index.html page handles the upload interface, output display, and AI logic.

AI Components

• Gemini Nano runs locally in Chrome Canary. This on-device model lets users generate short text without calling the cloud API.

• Gemini API (Cloud) used as a fallback when on-device AI isn’t available or when image analysis is required.

  • Model tested: gemini-1.5-flash and gemini-pro-vision.

  • Fallback logic ensures the app checks multiple model endpoints if one returns a 404 error.

Local Storage

• The Gemini API key is stored safely in the browser’s localStorage, so it never leaves the user’s computer.

Browser SpeechSynthesis API

• Converts the translated English meaning into spoken audio with one click.

Mapping Logic

• A small custom dictionary (mapping.js) links AI-generated descriptions to likely Makaton meanings. For example: { keywords: ["open hand", "raised hand", "wave"], meaning: "Hello / Stop" }

Local Server

• The app is served locally using Python’s built-in HTTP server to avoid CORS issues:

  python -m http.server 8080

Then open http://localhost:8080 in Chrome Canary.

Building the App Step by Step

Now let’s dive into how the Makaton AI Companion works under the hood. This project follows a simple but effective flow: Upload an image → Describe (AI) → Map to Meaning → Speak or Copy the result

We’ll go through each part step by step.

1. Setting Up the Project Folder

You don’t need any complex setup. Just create a new folder and add these files:

makaton-ai-companion/
│
├── index.html
├── styles.css
├── app.js
└── lib/
    ├── mapping.js
    └── ai.js

If you prefer a ready-to-run version, you can serve everything from one zip (I’ll share a GitHub link at the end).

2. Creating the Basic HTML Structure

Your index.html file defines the interface where users upload an image, click Describe, and view the results.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
  <title>Makaton AI Companion</title>
  <link rel="stylesheet" href="styles.css"/>
</head>
<body>
  <header class="app-header">
    <h1>🧩 Makaton AI Companion</h1>
    <button id="btnSettings" class="btn secondary">Settings</button>
  </header>

  <main class="container">
    <section class="card">
      <h2>1) Upload an image (Makaton sign/symbol)</h2>
      <label for="file">
        Choose an image file
        <input id="file" type="file" accept="image/*" title="Select an image file"/>
      </label>
      <div id="preview" class="preview hidden"></div>
      <p id="status" class="status"></p>
      <div class="actions">
        <button id="btnDescribe" class="btn">Describe (Cloud or Nano)</button>
        <button id="btnType" class="btn ghost">Type a description instead</button>
      </div>
      <div id="typedBox" class="typed hidden">
        <textarea id="typed" rows="3" placeholder="Describe what you see..."></textarea>
        <button id="btnUseTyped" class="btn">Use this description</button>
      </div>
    </section>

    <section class="card">
      <h2>2) AI Output</h2>
      <div class="grid">
        <div>
          <h3>Image Description</h3>
          <div id="output" class="output"></div>
        </div>
        <div>
          <h3>English Meaning (Mapped)</h3>
          <div id="meaning" class="meaning"></div>
          <div class="actions">
            <button id="btnSpeak" class="btn ghost" disabled>🔊 Speak</button>
            <button id="btnCopy" class="btn ghost" disabled>📋 Copy</button>
          </div>
        </div>
      </div>
    </section>
  </main>

  <dialog id="settings">
    <form method="dialog" class="settings-form">
      <h2>Settings</h2>
      <label>Gemini API key (optional)<input id="apiKey" type="password" placeholder="AIza..."/></label>
      <div class="settings-actions">
        <button id="btnSaveKey" type="submit" class="btn">Save</button>
        <button id="btnCloseSettings" type="button" class="btn secondary">Close</button>
      </div>
      <div id="apiStatus" class="api-status"></div>
    </form>
  </dialog>

  <script type="module" src="lib/mapping.js"></script>
  <script type="module" src="lib/ai.js"></script>
  <script type="module" src="app.js"></script>
</body>
</html>

This interface is intentionally minimal: no frameworks, no build tools, just clear HTML.

3. Mapping Descriptions to Makaton Meanings

The mapping.js file holds a simple keyword-based dictionary. When the AI describes an image (like “a raised open hand”), the app searches for keywords that match known Makaton signs.

// lib/mapping.js

export const MAKATON_GLOSSES = [
  { keywords: ["open hand", "raised hand", "wave", "hand up"], meaning: "Hello / Stop" },
  { keywords: ["eat", "food", "spoon", "hand to mouth"], meaning: "Eat" },
  { keywords: ["drink", "cup", "glass", "bottle"], meaning: "Drink" },
  { keywords: ["home", "house", "roof"], meaning: "Home" },
  { keywords: ["sleep", "bed", "eyes closed"], meaning: "Sleep" },
  { keywords: ["book", "reading", "pages"], meaning: "Book / Read" },
  // Added so your current screenshot maps correctly:
  { keywords: ["help", "assist", "thumb on palm", "hand over hand", "assisting"], meaning: "Help" },
];

export function mapDescriptionToMeaning(desc) {
  if (!desc) return "";
  const d = desc.toLowerCase();
  for (const entry of MAKATON_GLOSSES) {
    if (entry.keywords.some(k => d.includes(k))) return entry.meaning;
  }
  if (d.includes("hand")) return "Gesture / Hand sign (clarify)";
  return "No direct mapping found.";
}

It’s simple but effective enough to simulate real symbol-to-language translation for demo purposes.

4. Adding Gemini AI Logic

The ai.js file connects to Gemini Nano (on-device) or the Gemini API (cloud). If Nano isn’t available, the app falls back to the cloud model. And if that fails, it lets users type a description manually.

// lib/ai.js — dynamic model discovery (try-all version)

// --- On-device availability (Gemini Nano) ---
export async function checkAvailability() {
  const res = { nanoTextPossible: false };
  try {
    const canCreate = self.ai?.canCreateTextSession || self.ai?.languageModel?.canCreate;
    if (typeof canCreate === "function") {
      const ok = await (self.ai.canCreateTextSession?.() || self.ai.languageModel.canCreate?.());
      res.nanoTextPossible = ok === "readily" || ok === "after-download" || ok === true;
    }
  } catch {}
  return res;
}

export async function createNanoTextSession() {
  if (self.ai?.createTextSession) return await self.ai.createTextSession();
  if (self.ai?.languageModel?.create) return await self.ai.languageModel.create();
  throw new Error("Gemini Nano text session not available");
}

// --- Cloud: dynamically discover models for this key ---
async function listModels(key) {
  const url = "https://generativelanguage.googleapis.com/v1/models?key=" + encodeURIComponent(key);
  const r = await fetch(url);
  if (!r.ok) throw new Error("ListModels failed: " + (await r.text()));
  const j = await r.json();
  return (j.models || []).map(m => m.name).filter(Boolean);
}

function rankModels(names) {
  // Prefer Gemini 1.5 (multimodal), then flash variants, then anything with vision/pro.
  return names
    .filter(n => n.startsWith("models/"))              // ignore tunedModels, etc.
    .filter(n => !n.includes("experimental"))          // skip experimental
    .sort((a, b) => score(b) - score(a));

  function score(n) {
    let s = 0;
    if (n.includes("1.5")) s += 10;
    if (n.includes("flash")) s += 8;
    if (n.includes("pro-vision")) s += 7;
    if (n.includes("pro")) s += 6;
    if (n.includes("vision")) s += 5;
    if (n.includes("latest")) s += 2;
    return s;
  }
}

async function tryGenerateForModels(imageDataUrl, key, models, mimeType) {
  const base64 = imageDataUrl.split(",")[1];
  const body = {
    contents: [{
      parts: [
        { text: "Describe this image briefly in one sentence focusing on the main gesture or symbol." },
        { inline_data: { mime_type: mimeType || "image/png", data: base64 } }
      ]
    }]
  };
  let lastErr = "";
  for (const model of models) {
    const endpoint = "https://generativelanguage.googleapis.com/v1/" + model + ":generateContent?key=" + encodeURIComponent(key);
    try {
      const r = await fetch(endpoint, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body)});
      if (!r.ok) { lastErr = await r.text().catch(()=>String(r.status)); continue; }
      const j = await r.json();
      const text = j?.candidates?.[0]?.content?.parts?.map(p=>p.text).join(" ").trim();
      if (text) return text;
      lastErr = "Empty response from " + model;
    } catch (e) {
      lastErr = String(e?.message || e);
    }
  }
  throw new Error("All discovered models failed. Last error: " + lastErr);
}

export async function describeImageWithGemini(imageDataUrl, apiKey, mimeType = "image/png") {
  if (!apiKey) throw new Error("No API key provided");

  const models = await listModels(apiKey);
  if (!models.length) throw new Error("No models returned for this key. Ensure Generative Language API is enabled and T&Cs accepted in AI Studio.");

  const ranked = rankModels(models);
  if (!ranked.length) throw new Error("No usable model names returned (models/*).");

  return await tryGenerateForModels(imageDataUrl, apiKey, ranked, mimeType);
}

// --- Key storage (local only) ---
const KEY = "makaton_demo_gemini_key";
export function saveApiKey(k) { localStorage.setItem(KEY, k || ""); }
export function loadApiKey() { return localStorage.getItem(KEY) || ""; }

Note: This retry system is essential because many users encounter 404 model errors due to the unavailability of certain Gemini versions in every account.

5. The Main Logic (app.js)

This script ties everything together: file upload, AI call, meaning mapping, and output display.

import { mapDescriptionToMeaning } from './lib/mapping.js';
import { checkAvailability, createNanoTextSession, describeImageWithGemini, saveApiKey, loadApiKey } from './lib/ai.js';

document.addEventListener('DOMContentLoaded', () => {
  console.log('[Makaton] DOM ready');

  const $ = (s) => document.querySelector(s);

  // Elements
  const fileInput   = $('#file');
  const preview     = $('#preview');
  const meaningEl   = $('#meaning');
  const outputEl    = $('#output');
  const btnDescribe = $('#btnDescribe');
  const btnType     = $('#btnType');
  const typedBox    = $('#typedBox');
  const typed       = $('#typed');
  const btnUseTyped = $('#btnUseTyped');
  const btnSpeak    = $('#btnSpeak');
  const btnCopy     = $('#btnCopy');
  const statusEl    = $('#status');

  const settings        = $('#settings');
  const btnSettings     = $('#btnSettings');
  const btnCloseSettings= $('#btnCloseSettings');
  const btnSaveKey      = $('#btnSaveKey');
  const apiKeyInput     = $('#apiKey');
  const apiStatus       = $('#apiStatus');

  let currentImageDataUrl = null;
  let currentImageMime    = "image/png";

  // Sanity logs
  console.log('[Makaton] Elements:', {
    fileInput: !!fileInput, preview: !!preview, outputEl: !!outputEl,
    meaningEl: !!meaningEl, btnDescribe: !!btnDescribe, statusEl: !!statusEl
  });

  // Init API key
  if (apiKeyInput) apiKeyInput.value = loadApiKey() || "";

  // --- Helpers ---
  function setStatus(text) {
    if (statusEl) statusEl.textContent = text || '';
    console.log('[Makaton][Status]', text);
  }
  function clearOutputs() {
    if (outputEl) outputEl.textContent = '';
    if (meaningEl) meaningEl.textContent = '';
    if (btnSpeak) btnSpeak.disabled = true;
    if (btnCopy)  btnCopy.disabled  = true;
  }
  function setOutput(desc) {
    if (outputEl) outputEl.textContent = desc || '';
    const meaning = mapDescriptionToMeaning(desc || '');
    if (meaningEl) meaningEl.textContent = meaning;
    if (btnSpeak) btnSpeak.disabled = !meaning || meaning.includes('No direct mapping');
    if (btnCopy)  btnCopy.disabled  = !meaning;
    setStatus('Done.');
  }
  function fileToDataURL(file) {
    return new Promise((resolve, reject) => {
      const reader = new FileReader();
      reader.onload  = () => resolve(reader.result);
      reader.onerror = (e) => reject(e);
      reader.readAsDataURL(file);
    });
  }
  function handleFiles(files) {
    const file = files?.[0];
    if (!file) { setStatus('No file selected.'); return; }
    currentImageMime = file.type || "image/png";
    fileToDataURL(file)
      .then((dataUrl) => {
        currentImageDataUrl = dataUrl;
        if (preview) {
          preview.innerHTML = `<img alt="preview" src="${dataUrl}" />`;
          preview.classList.remove('hidden');
        }
        setStatus('Image loaded. Click "Describe" to continue.');
      })
      .catch((err) => {
        console.error('[Makaton] fileToDataURL error', err);
        setStatus('Could not read the image.');
      });
  }

  // --- File input change ---
  if (fileInput) {
    fileInput.addEventListener('change', (e) => {
      console.log('[Makaton] file input change');
      handleFiles(e.target.files);
    });
  } else {
    console.warn('[Makaton] #file input not found in DOM.');
  }

  // --- Drag & drop support on preview area ---
  if (preview) {
    preview.addEventListener('dragover', (e) => { e.preventDefault(); preview.classList.add('drag'); });
    preview.addEventListener('dragleave', () => preview.classList.remove('drag'));
    preview.addEventListener('drop', (e) => {
      e.preventDefault();
      preview.classList.remove('drag');
      console.log('[Makaton] drop');
      handleFiles(e.dataTransfer?.files);
    });
  }

  // --- Describe click ---
  if (btnDescribe) {
    btnDescribe.addEventListener('click', async () => {
      console.log('[Makaton] Describe clicked');
      if (!currentImageDataUrl) { setStatus('Please upload an image first.'); return; }
      clearOutputs();
      setStatus('Checking on-device AI availability…');

      const avail = await checkAvailability().catch(() => ({ nanoTextPossible: false }));
      try {
        const apiKey = loadApiKey();
        if (apiKey) {
          setStatus('Using Gemini cloud for image description…');
          const desc = await describeImageWithGemini(currentImageDataUrl, apiKey, currentImageMime);
          setOutput(desc);
          return;
        }
        if (avail.nanoTextPossible) {
          setStatus('No API key found. Using on-device AI (text) for best guess…');
          const session = await createNanoTextSession();
          const desc = await session.prompt('Given an image is uploaded by the user (not directly visible to you), infer a likely one-sentence description of a common Makaton sign or symbol a teacher might upload. Keep it generic and safe.');
          setOutput(desc);
          return;
        }
        setStatus('No AI available. Please type a brief description.');
        if (typedBox) typedBox.classList.remove('hidden');
      } catch (err) {
        console.error('[Makaton] Describe error', err);
        setStatus('Description failed: ' + (err?.message || err));
        if (typedBox) typedBox.classList.remove('hidden');
      }
    });
  } else {
    console.warn('[Makaton] Describe button not found.');
  }

  // --- Manual typing flow ---
  if (btnType) {
    btnType.addEventListener('click', () => {
      if (typedBox) typedBox.classList.remove('hidden');
      if (typed) typed.focus();
    });
  }
  if (btnUseTyped) {
    btnUseTyped.addEventListener('click', () => {
      const text = (typed?.value || '').trim();
      if (!text) { setStatus('Type a description first.'); return; }
      setOutput(text);
    });
  }

  // --- Utilities ---
  if (btnSpeak) {
    btnSpeak.addEventListener('click', () => {
      const text = meaningEl?.textContent?.trim();
      if (!text) return;
      const u = new SpeechSynthesisUtterance(text);
      speechSynthesis.cancel();
      speechSynthesis.speak(u);
    });
  }
  if (btnCopy) {
    btnCopy.addEventListener('click', async () => {
      const text = meaningEl?.textContent?.trim();
      if (!text) return;
      try {
        await navigator.clipboard.writeText(text);
        setStatus('Copied meaning to clipboard.');
      } catch {
        setStatus('Copy failed.');
      }
    });
  }

  // --- Settings modal ---
  if (btnSettings && settings) btnSettings.addEventListener('click', () => settings.showModal());
  if (btnCloseSettings && settings) btnCloseSettings.addEventListener('click', () => settings.close());
  if (btnSaveKey) {
    btnSaveKey.addEventListener('click', (e) => {
      e.preventDefault();
      const k = apiKeyInput?.value?.trim() || "";
      saveApiKey(k);
      if (apiStatus) apiStatus.textContent = k ? "API key saved locally. Try Describe again." : "Cleared API key. You can still use on-device or typed mode.";
    });
  }

  // First status
  setStatus('Ready. Upload an image to begin.');
});

Let's break down the main sections of the app.js script for the Makaton AI Companion, as there’s a lot going on here:

1. Imports and Initial Setup:

   • The script imports functions from mapping.js and ai.js to handle mapping descriptions to meanings and AI interactions.

   • It sets up event listeners for when the DOM content is fully loaded, ensuring all elements are ready for interaction.

2. Element Selection:

   • It uses a helper function $ to select DOM elements by their CSS selectors. This includes file inputs, buttons, and display areas for image previews and outputs.

3. Sanity Logs:

   • It logs the presence of key elements to the console for debugging purposes, ensuring that all necessary elements are found in the DOM.

4. API Key Initialization:

   • It loads any saved API key from local storage and sets it in the input field for user convenience.

5. Helper Functions:

   • setStatus: Updates the status message displayed to the user.

   • clearOutputs: Clears the output and meaning display areas and disables buttons for speaking and copying.

   • setOutput: Displays the AI-generated description and maps it to a Makaton meaning, enabling buttons if a valid meaning is found.

   • fileToDataURL: Converts an uploaded file to a data URL for image preview and processing.

   • handleFiles: Handles file selection, updating the preview and setting the current image data URL.

6. File Input Change Handling:

   • It listens for changes in the file input, processes the selected file, and updates the preview area.

7. Drag & Drop Support:

   • It adds drag-and-drop functionality to the preview area, allowing users to drag files directly onto the app for processing.

8. Describe Button Click:

   • It handles the "Describe" button click event, checking for an uploaded image and attempting to describe it using either the Gemini API or on-device AI.

   • If no AI is available, it prompts the user to type a description manually.

9. Manual Typing Flow:

   • It allows users to manually type a description if AI processing is unavailable or fails, updating the output with the typed text.

10. Utilities:

    • btnSpeak: Uses the browser's SpeechSynthesis API to read aloud the mapped meaning.

    • btnCopy: Copies the mapped meaning to the clipboard for easy sharing.

11. Settings Modal:

    • It manages the settings modal for entering and saving the API key, providing feedback on the key's status.

12. Initial Status:

    • It sets the initial status message to guide the user to upload an image to begin the process.

This script effectively ties together the user interface, file handling, AI processing, and output display, providing a seamless experience for translating Makaton signs into English meanings.

How Vision and Language Work Together Here

While working on this project, I started appreciating how computer vision and language understanding complement each other in multimodal systems like this one.

• The vision model (Gemini or Nano) interprets what it sees like hand shapes, gestures, or layout and turns that visual context into descriptive language.

• The language mapping logic then interprets those words, infers intent, and finds the closest semantic match (e.g., “help,” “friend,” “eat”).

• It’s a collaboration between two forms of understanding (perceptual and semantic) that together allow the AI to bridge the gap between gesture and meaning.

This realization reshaped how I think about accessibility: the best assistive technologies often emerge not from smarter models alone, but from the interaction between modalities like seeing, describing, and reasoning in context.

6. Optional — Speak and Copy

To make the app more accessible, I added speech output and a quick copy button:

btnSpeak.addEventListener('click', () => {
  const text = meaningEl.textContent.trim();
  if (text) speechSynthesis.speak(new SpeechSynthesisUtterance(text));
});

btnCopy.addEventListener('click', async () => {
  const text = meaningEl.textContent.trim();
  if (text) await navigator.clipboard.writeText(text);
});

This gives users both visual and auditory feedback, especially helpful for learners or educators.

How to Fix the Common Issues

No AI or web integration project runs smoothly the first time – and that’s okay. Here’s a breakdown of the main issues I faced while building the Makaton AI Companion, how I diagnosed them, and how I fixed each one.

These lessons will help anyone trying to integrate Gemini APIs, on-device AI, or local web apps without a full backend.

1. The “CORS” Error When Running With file://

When I first opened my index.html directly from my file explorer, Chrome threw several CORS policy errors:

Access to script at 'file:///lib/ai.js' from origin 'null' has been blocked by CORS policy.

At first this looked confusing, but the reason is simple: modern browsers block JavaScript modules (import/export) when running from file:// paths for security reasons.

✅ Fix: I realized I needed to serve the files over HTTP, not from the file system. So I ran a quick local web server using Python:

python -m http.server 8080

Then opened:

http://localhost:8080/index.html

That single step fixed all the CORS errors and allowed my modules to load correctly.

2. “Model Not Found” (404) From the Gemini API

The next big challenge came from the Gemini API. Even though I had a valid API key, my console showed this error:

"models/gemini-1.5-flash" is not found for API version v1beta, or is not supported for generateContent.

It turns out Google’s API endpoints can vary slightly depending on your project setup and key permissions.

✅ Fix: I rewrote my lib/ai.js script to automatically try multiple Gemini model endpoints until it found one that worked. Something like this:

const GEMINI_IMAGE_ENDPOINTS = [
  "https://generativelanguage.googleapis.com/v1/models/gemini-1.5-flash:generateContent",
  "https://generativelanguage.googleapis.com/v1/models/gemini-1.5-pro:generateContent",
  "https://generativelanguage.googleapis.com/v1/models/gemini-1.5-flash-latest:generateContent",
];

And I wrapped it in a loop that stopped once one endpoint succeeded.

Later, I improved it further by listing available models dynamically using
https://generativelanguage.googleapis.com/v1/models?key=YOUR_KEY and automatically trying whichever ones supported image generation.

That dynamic discovery approach fixed the 404 errors permanently.

3. Packaging a Local Single-File Version

Once I got everything working, I wanted a version that others could test easily without installing Node.js or running build tools.

✅ Fix: I bundled the project into a simple zip file containing:

index.html
app.js
lib/ai.js
lib/mapping.js
styles.css

That way, anyone can just unzip and run:

python -m http.server 8080

and open localhost:8080.

Everything runs locally in the browser, no server-side code required. This also makes it perfect for demos, classrooms, and so on.

4. Debugging Script Import Errors in the Console

Another subtle issue appeared when I noticed this red message:

The requested module './lib/mapping.js' does not provide an export named 'mapDescriptionToMeaning'

That line told me exactly what was wrong: my import and export function names didn’t match. The fix was straightforward:

// app.js
import { mapDescriptionToMeaning } from './lib/mapping.js';

And then ensuring the mapping file exported it:

// mapping.js
export function mapDescriptionToMeaning(desc) { ... }

After that, all the pieces connected smoothly.

Using the browser console as my debugging dashboard turned out to be the most powerful tool of all. Every fix started by reading and reasoning about those red error lines.

Demo: The Makaton AI Companion in Action

Let’s see the Makaton AI Companion in action and understand what’s happening under the hood.

Step 1: Run the app locally

Once you’ve downloaded or cloned the project folder, open your terminal in that directory and start a local development server: python -m http.server 8080. Then open your browser and visit: http://localhost:8080/index.html

You should see the Makaton AI Companion interface:

Step 2: Get Your Gemini API Key

To enable cloud-based image description, you’ll need a Gemini API key from Google AI Studio.

Here’s how to generate one:

1. Visit: https://aistudio.google.com/welcome

2. Click “Create API key” and link it to your Google Cloud project (or create a new one).

3. Copy the key it will look like this: AIzaSyA...XXXXXXXXXXXX

4. Open the Makaton AI Companion in your browser and click the Settings button (top left).

5. Paste your key in the input box and click Save.

You’ll see a confirmation message like this:

“API key saved locally. Try Describe again.”

This means your key is stored safely in localStorage and is only accessible from your browser.

Step 3: Enable Gemini Nano for On-Device AI

If you’re using Chrome Canary, you can run Gemini Nano locally without internet access. This allows the Makaton AI Companion to generate text even when the API key isn’t set.

Download and Install Chrome Canary:

Visit the official Chrome Canary download page and install it on your Windows or macOS system. Chrome Canary is a special version of Chrome designed for developers and early adopters, offering the latest features and updates.

Enable Gemini Nano:

Open Chrome Canary and type chrome://flags/#prompt-api-for-gemini-nano in the address bar.

Locate the "Prompt API for Gemini Nano" flag in the list. Set this flag to Enabled. This action allows Chrome Canary to support the Gemini Nano model for on-device AI processing.

After enabling the flag, relaunch Chrome Canary to apply the changes.

Download the Gemini Nano Model:

Open a new tab in Chrome Canary and enter chrome://components in the address bar.

Scroll down to find the “Optimization Guide” component. Click on Check for update. This action will initiate the download of the Gemini Nano model, which is necessary for running AI tasks locally without an internet connection.

Verify Installation:

Once the Gemini Nano model is installed, the Makaton AI Companion app will automatically detect it. You should see a message indicating that the app is using on-device AI: “No API key found. Using on-device AI (text) for best guess…”

This confirmation means that the app can now generate text descriptions using the Gemini Nano model without needing an API key or internet access.

By following these detailed steps, you ensure that the Gemini Nano model is correctly set up and ready to use for on-device AI processing in the Makaton AI Companion.

Step 4: Upload a Makaton sign or symbol

Click Choose File to upload any Makaton image (for example, the “help” sign), then press Describe (Cloud or Nano). You’ll immediately see console logs confirming that the app is running correctly and connecting to the Gemini API:

Step 5: AI Description and Mapping

Here’s what happens next:

1. The image is read and encoded as Base64.

2. The Gemini API (cloud or on-device) generates a short visual description.

3. The description is passed to the mapDescriptionToMeaning() function.

4. If keywords match an entry in the MAKATON_GLOSSES dictionary, the app displays the corresponding English meaning.

5. Finally, users can click Speak or Copy to hear or reuse the translation.

Example outputs:

When no mapping is found:
The AI description is accurate but doesn’t yet match a known Makaton keyword.

After updating the mapping list:
Adding new keywords like "help", "assist", or "hand over hand" enables correct translation.

Why this matters

This demonstrates how accessible, AI-assisted tools can support communication for people who rely on Makaton. Even when a gesture isn’t recognized, the system provides a structured output and allows users or educators to expand the mapping list making the tool smarter over time.

Broader Reflections

Building this project turned out to be much more than a coding exercise for me.
It was a meaningful experiment in combining accessibility, natural language processing, and computer vision. These three fields, when brought together, can create real social impact.

While working on it, I began to understand how computer vision and language understanding complement each other in practice. The vision model perceives the world by identifying shapes, gestures, and spatial patterns, while the language model interprets what those visuals mean in human terms.
In this project, the artificial intelligence system first sees the Makaton sign, then describes it, and finally maps it to an English word that carries intent and meaning.

This interaction between perception and semantics is what makes multimodal artificial intelligence so powerful. It is not only about recognizing an image or generating text; it is about building systems that connect understanding across different forms of information to make technology more inclusive and human centered.

This realization changed how I think about accessibility technology. True innovation happens not only through smarter models but through the harmony between seeing and understanding, between what an artificial intelligence system observes and how it communicates that observation to help people.

Accessibility Meets AI

Working on this project reminded me that accessibility isn’t just about compliance or assistive devices. It’s also about inclusion. A simple AI system that can describe a hand gesture or symbol in real time can empower teachers, parents, and students who communicate using Makaton or similar systems.

By mapping AI-generated descriptions to meaningful phrases, the app demonstrates how AI can support inclusive education, even at small scales. It bridges the communication gap between verbal and nonverbal learners, which is something that traditional translation systems often overlook.

Integrating NLP and Computer Vision

On the technical side, this project showed me how naturally computer vision and language understanding complement each other. The Gemini API’s multimodal models were able to analyze an image and produce coherent natural-language sentences, something that older APIs couldn’t do without chaining multiple tools.

By feeding that output into a lightweight NLP mapping function, I was able to simulate a very early-stage symbol-to-language translator the core of my broader research interest in automatic Makaton-to-English translation.

Why Local AI (Gemini Nano) Matters

While the cloud models are powerful, experimenting with Gemini Nano revealed something exciting:
on-device AI can make accessibility tools faster, safer, and more private.

In classrooms or therapy sessions, you often can’t rely on stable internet connections or share sensitive student data. Running inference locally means learners’ gestures or symbol images never leave the device, a crucial step toward privacy-preserving accessibility AI.

And since Nano runs directly inside Chrome Canary, it shows how AI is becoming embedded at the browser level, lowering barriers for teachers and developers to build inclusive solutions without needing large infrastructure.

Looking Forward

This prototype is just a starting point. Future iterations could integrate gesture recognition directly from camera input, support multiple symbol sets, or even learn from user feedback to expand the dictionary automatically.

Most importantly, it reinforces a central belief in my research and teaching journey:

Accessibility innovation doesn’t require massive systems. It starts with curiosity, empathy, and a few lines of purposeful code.

Conclusion

Building the Makaton AI Companion has been one of the most rewarding projects in my AI journey – not just because it worked, but because it proved how accessible innovation can be.

With just a browser, a few lines of JavaScript, and the right API, I was able to combine computer vision, language understanding, and accessibility design into a working system that translates symbols into meaning. It’s a small step toward a future where anyone, regardless of speech or language ability, can be understood through technology.

The project also reinforced something deeply personal to me as a researcher and educator: that AI for accessibility doesn’t need to be complex, expensive, or centralized. It can be lightweight, open, and built with empathy by anyone who’s willing to learn and experiment.

Join the Conversation

If this project inspires you, I’d love to see your own experiments and improvements. Can you make it support live webcam gestures? Could you adapt it for other symbol systems, like PECS or BSL?

Share your ideas in the comments or tag me if you publish your own version. Together, we can grow a small prototype into a community-driven accessibility tool and continue exploring how AI can give more people a voice.

Full source code on GitHub: Makaton-ai-companion

Whether you build chatbots, recommendation systems, or search engines, understanding how close two sentences are in meaning can improve user experience – and this is what sentence similarity allows you to do.

Sentence Transformers make this process simple and efficient. In this guide, you will learn what sentence similarity is, how Sentence Transformers work, and how to write code to measure similarity between two sets of sentences.

Table of Contents

• What Is Sentence Similarity?

• Why Use Sentence Transformers

• Loading a Pre-trained Model

• Defining Sentences to Compare

• Converting Sentences into Embeddings

• Calculating Cosine Similarity

• Printing the Results

• Sample Output

• How to Interpret the Scores

• Real-World Applications of Sentence Similarity

  • Semantic Search

  • Duplicate Detection

  • Recommendation Systems

  • Chatbots and Virtual Assistants

  • Improving Performance with Larger Models

• Conclusion

What Is Sentence Similarity?

Sentence similarity is the process of comparing two sentences to see how close they are in meaning. It does not look at the exact words but focuses on the meaning behind them.

For example:

• “The cat is sitting outside”

• “The dog is playing in the garden”

Both sentences talk about animals outdoors, so they share some similarity even though they use different words.

This kind of understanding is essential for tasks like document clustering, duplicate detection, or semantic search.

Why Use Sentence Transformers

Traditional methods like Bag of Words relied on simple word matching or frequency counts. But these fail when words differ but the meaning stays the same.

Sentence Transformers solve this by using transformer-based language models like BERT or RoBERTa to create embeddings.

An embedding is a list of numbers that represents the meaning of a sentence. When two embeddings are close together in this high-dimensional space, their sentences are similar in meaning.

The Sentence Transformers library in Python makes this easy by providing pre-trained models that can generate embeddings for sentences.

Installing the Required Libraries

Before you start coding, make sure you install the required packages. Run this command to do so:

pip install -U sentence-transformers

This will install the Sentence Transformers library along with its dependencies.

Loading a Pre-trained Model

Sentence Transformers offers several pre-trained models. For this example, you will use the all-MiniLM-L6-v2 model. It’s lightweight, fast, and works well for most applications.

Here is how to load it in Python:

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("all-MiniLM-L6-v2")

Once loaded, this model can convert any sentence into its corresponding embedding.

Defining Sentences to Compare

You need two lists of sentences for comparison. Here is an example:

sentences1 = [
    'The cat sits outside',
    'A man is playing guitar',
    'The movies are awesome'
]

sentences2 = [
    'The dog plays in the garden',
    'A woman watches TV',
    'The new movie is so great'
]

Each sentence in sentences1 will be compared with the sentence at the same position in sentences2.

Converting Sentences into Embeddings

Now that you have sentences, you must convert them into embeddings using the model.

Add this code:

# Convert sentences to embeddings
embeddings1 = model.encode(sentences1, convert_to_tensor=True)
embeddings2 = model.encode(sentences2, convert_to_tensor=True)

The convert_to_tensor=True argument tells the model to return PyTorch tensors, which work well with similarity calculations.

Calculating Cosine Similarity

Once you have embeddings, you need a way to measure similarity. The cosine similarity metric is commonly used for this.

Cosine similarity looks at the angle between two vectors in a high-dimensional space. If the angle is small, the vectors are similar.

Add this code to compute similarity:

from sentence_transformers import util
# Compute cosine similarity
cosine_scores = util.cos_sim(embeddings1, embeddings2)

Now cosine_scores contains the similarity score for each sentence pair.

Printing the Results

To see the results clearly, format them like this:

# Print formatted results
for i in range(len(sentences1)):
    print(f"{sentences1[i]} \t\t {sentences2[i]} \t\t Score: {cosine_scores[i][i]:.4f}")

This will print each sentence pair along with its similarity score.

Sample Output

If you run this code, you will see a result similar to the below.

The third pair has the highest score because both sentences talk about movies in a positive way.

How to Interpret the Scores

The cosine similarity score ranges between -1 and 1.

• A score close to 1 means the sentences are very similar.

• A score near 0 means they are unrelated.

• Negative values mean the sentences are not related or even opposite in meaning.

In most real-world cases, you focus on values between 0 and 1. The higher the value, the closer the meanings.

Real-World Applications of Sentence Similarity

Sentence similarity has become a core part of many modern applications because it helps systems understand meaning rather than relying on exact words. This shift makes search, analysis, and recommendations far more accurate and useful.

Semantic Search

Traditional search engines depend on keyword matches. If the exact words are missing, results often become irrelevant. Semantic search solves this problem by looking at the meaning behind a query.

For example, if someone searches for “best ways to learn guitar,” the system can return results for “top tips to play the guitar” even though the keywords differ. This makes search experiences smoother and more intelligent.

Duplicate Detection

Large datasets often contain repeated or near-duplicate content. Manual checking is impossible when dealing with millions of records.

Sentence similarity automates this by detecting texts that carry the same meaning even if the wording changes slightly. This is especially useful in data cleaning, web scraping pipelines, or managing user-generated content.

Recommendation Systems

Recommendation engines work best when they understand context. For instance, if a user likes articles about “healthy cooking,” the system can recommend content on “nutritious recipes” or “quick healthy meals” using similarity scores. This approach goes beyond surface-level keywords and finds deeper connections in the text.

Chatbots and Virtual Assistants

Chatbots store a large set of possible user questions and answers. When someone types a new question, the system must find the most relevant response. By using sentence similarity, chatbots match user input with the closest existing query in meaning, not just words, leading to more accurate and natural conversations.

Improving Performance with Larger Models

The all-MiniLM-L6-v2 model is fast and accurate for small to medium tasks.

For more accuracy, you can try larger models like all-mpnet-base-v2, though they may require more memory and time to run.

Replace the model name in your code to use a different pre-trained model:

model = SentenceTransformer("all-mpnet-base-v2")

Conclusion

Sentence Transformers make it easy to measure sentence similarity using pre-trained models. By converting sentences into embeddings and comparing them with cosine similarity, you can build systems that understand meaning rather than relying on simple word matching.

With just a few lines of code, you can integrate this into chatbots, search engines, or recommendation systems and create more intelligent applications.

Hope you enjoyed this article. Signup for my free newsletter TuringTalks.ai for more hands-on tutorials on AI. You can also visit my website.

Sometimes, even though you might not see the exact term you’re looking for, the document might contain similar words or phrases that have the same meaning or context but don’t have the exact same form (such as differences in spelling).

Traditional NLP search approaches have relied on using exact forms to search for words or phrases in a particular document. But this fails at finding words based on semantic or contextual meaning.

To solve this, semantic matching comes into play. It’s an advanced way of searching that takes advantage of traditional search methods while also focusing more on locating or matching words or phrases based on their meaning or context (rather than solely on their exact form).

In this article, you will learn how to perform semantic matching using NLP. Without further ado, let’s get started.

Requirements

To make sure that you can reproduce the experiment in this tutorial, you’ll need to have a few things.

First, you’ll need to have Python 3.x (preferably Python 3.10) installed on your PC. You’ll also need some libraries, which you can install using the Pip package manager.

You should also have basic knowledge of NLP such as text preprocessing and text representation techniques. You can learn more here.

You can also fork the repo which contains all the code in this article so you can follow along.

To install everything using Pip, type the following command:

// to install with pip
pip install pypdf2 keybert sentence-transformers

Problem Definition

Suppose you’re a data scientist who’s part of a curriculum development team and want to know if a particular concept (word or phrase), say birth control, is being taught in a curriculum that’s in a pdf document.

One way you could do this is to open the pdf using a pdf tool and then use the ctrl + f (find) method to check if the phrase birth control is in the pdf.

You could also do it programmatically, as shown below:

# import library
import PyPDF2

# use PDFreader from PyPDF2 to read pdf content.
pdf_reader = PyPDF2.PdfReader("Relationships_Education_RSE_and_Health_Education.pdf")

# join all the content in the pdf pages together and lowercase the letters
pdf_document = " ".join([page.extract_text().lower() for page in pdf_reader.pages])

# check if the string 'birth control' is in the document [Returns False]
"birth control" in pdf_document

Below is the output of the above code:

False

As shown above, you can see that both the programmatic way of searching and the pdf tool say that the phrase “birth control” doesn't exist in the pdf document.

Well, this might be true, but because this is a traditional way of NLP searching (that matches word for word in exact form) let’s not fully trust it. As I explained earlier, some words might be in different forms or have a different spelling, but they might mean the same thing contextually or semantically.

So how do we solve this issue? This is where semantic matching comes into play.

What is Semantic Matching?

Semantic Matching is a technique used to determine if two elements have the same meaning. An element can be a word, phrase, sentence, document, or even a corpus. It refers to matching elements based on meaning or context and not just matching based on exact form.

In order to perform semantic matching in NLP, there are certain things you need to know and do. Let’s go through them now:

What is Word Embedding?

Word embedding is an advanced text representation technique used to represent words in a lower-dimensional vector representation. This vector representation captures inter-word semantic and syntactic information. This means that words that have similar meanings – even though they might be spelled differently – will have close to similar vector representations.

What does Lower-Dimensional Vector representation mean?

In NLP, traditional ways of representing text in a way machines can understand (that is, numerical vector representations) are Bag of Words, Term-Frequency and Inverse Document Frequency (TF-IDF), and One-hot encoding. But these techniques usually generate high dimensions (usually the size of the vocabulary) for a particular word representation and are sparse (meaning there will be lots of zeros).

So, for example, if a word is to be represented as a numerical vector and the document or corpus the word belongs to has 10,000 vocabularies, the size of the dimension of that word would be 10,000 (making it high).

The disadvantages of these techniques are high dimensions, sparsity, and their non-capability in capturing semantic information. So, advancements in NLP led to the development of word embedding techniques that simply create lower (also known as more dense) vector representations of words and can capture inter-word semantic information.

Word embedding is the holy grail in NLP and language technology, serving as the foundation for advanced language representation models such as GPT (Generative Pre-trained Transformer).

There is also sentence embedding that represents sentences in a lower-dimension vector representation.

How do we measure if two vectors are similar?

This is where cosine similarity comes into play. Cosine similarity is a mathematical technique that we use to know how similar two vectors are to each other.

In NLP, it usually outputs a value between 0 to 1. A value close to 1 means that the two vectors are highly similar.

For example, to understand how cosine similarity works, let’s create a word embedding vector representation for three words: Man, Woman, and Cat. Then we’ll use cosine similarity to figure out which vectors are similar.

Based on our own instincts, we know that Man should be closer to Woman than Cat. So, let’s use NLP to help us validate this.

Thanks to advancements in NLP, there are numerous models we can use to create word embeddings, which you can find on the Hugging Face repository. In this article, we are going to use the ⁣all-mpnet-base-v2 model from the ⁣SentenceTransformer library. According to ⁣SentenceTransformer, it provides the best quality performance in terms of sentence embedding, and you can also use it to create word embeddings.

The below code allows us to validate our claim using NLP. So, firstly, we initialize the SentenceTransformer with all-mpnet-base-v2 and then use the encode method to get the embedding of each word. Then, finally, we’ll use the cos_sim class, also from SentenceTransformer, to determine which vectors are similar.

# import library
from sentence_transformers import SentenceTransformer # sentence transformer
from sentence_transformers.util import cos_sim # cosine similarity

# initialize sentence transformer with the 'all-mpnet-base-v2' model
model = SentenceTransformer("all-mpnet-base-v2")

# get the embedding vector of the man, woman, and cat words.
man_vector = model.encode("man")
woman_vector = model.encode("woman")
cat_vector = model.encode("cat")

# get the similarity between man and woman
similarity = cos_sim(man_vector, woman_vector)

# get the similarity between man and cat
cat_similarity = cos_sim(man_vector, cat_vector)

print("The Similarity between Man vector and Woman Vector:", similarity, "\n")

print("The Similarity between Man vector and Cat Vector:", cat_similarity)

// Result

The Similarity between Man vector and Woman Vector: tensor([[0.3501]]) 

The Similarity between Man vector and Cat Vector: tensor([[0.2553]])

As you can see, the similarity score between man and woman (0.35) is higher than that of man and cat (0.26). This shows the beauty of word embedding and cosine similarity together.

Now let’s get back to our business.

How to Perform Semantic Matching on a PDF Document

Now we are going to use semantic matching to look for a word or phrase in the document that matches the birth control phrase.

How to Get Words from the PDF using KeyBERT

Word embedding generates embeddings for individual words. Our PDF document contains a large volume of textual components, including digits, special characters, symbols, stopwords, and the actual words we want to match. So, to save time on preprocessing, we are going to utilize KeyBERT. This is a library that allows us to get meaningful keywords (words or phrases) from a particular document in a minimal way.

Keep in mind that by default, KeyBERT extracts single keywords – but we can also tell it to extract phrases with two or more words. We’ll use it here to extract single-word and 2-word phrases. Below is the implementation of using KeyBERT to extract keywords from our document:

from keybert import KeyBERT
# initialize model
keybert_model =  KeyBERT()

# extract all keywords (single word and 2 word phrase) from the pdf
all_keywords = keybert_model.extract_keywords(docs=pdf_document, top_n=-1, keyphrase_ngram_range=(1, 2))
# print length of keywords extracted                                             
print(len(all_keywords))
# show the first 5 keywords
print(all_keywords[:5])

The above code imports KeyBERT from the keybert library. It then initializes KeyBERT, and extracts all keywords (that is, single word and 2-word phrases) from the document. Then the next line prints the number of keywords extracted. Lastly, the code prints the first five 5 keywords out of all the keywords extracted from the PDF.

Below is the output of the above code:

8669
[('education guidance', 0.5954),
 ('schools guidance', 0.5542),
 ('education policies', 0.5405),
 ('sex education', 0.5228),
 ('education safeguarding', 0.5001)]

As you can see above, KeyBERT extracted 8,669 keywords from the PDF. Also, the KeyBERT model usually returns the keywords extracted along with a score of each word. We don’t need the score, so we will only extract each keyword from the tuple it is enclosed in.

# remove score from each keyword

all_keywords = [keyword[0] for keyword in all_keywords]
all_keywords[:5]

Below is the output of the above code:

['education guidance',
 'schools guidance',
 'education policies',
 'sex education',
 'education safeguarding']

Embedding of the Birth Control Phrase and the Keywords Extracted from the PDF

Now that we’ve extracted these keywords from the document, the next step is to get the embedding of our phrase and the keywords from the document.

The below code lets us do this:

# initialize sentence transformer with the 'all-mpnet-base-v2' model
model = SentenceTransformer("all-mpnet-base-v2")

# get the embedding of the 'birth control' phrase
birth_control_embedding = model.encode("birth control")

# get the embedding of all the keywords in the document
keywords_embedding =  model.encode(all_keywords)

Cosine Similarity of Birth Control Phrase and Keywords in PDF

After getting the embedding of the phrase and the keywords, the next step is to get the similarity score of the phrase and the keywords. This will help us know which keyword in the document is highly similar to the phrase.

The below code allows us to get the cosine similarity of the phrase and the keywords’ embedding vector.

# calculate the cosine similarity of the birth control word and each word in the document
cosine_similarity_result = cos_sim(birth_control_embedding, keywords_embedding)
# print the shape (equal to the number of keywords)
print(cosine_similarity_result.shape)
# show the top 5 similarities
print(cosine_similarity_result[:5])

Below is the output of the above code:

torch.Size([1, 2034])
tensor([[0.2166, 0.1977, 0.0998,  ..., 0.1634, 0.1082, 0.2194]])

Now that we have the similarity score of the phrase and the keywords, the total size of the resulting tensor will be the number of keywords, as shown above. Then we can use the argmax() method to get the index of the element of the tensor with the highest score. This index will help us filter out the particular keyword in the all_keywords list variable. The below code achieves this:

# return the index number of the high similarity score
index = cosine_similarity_result.argmax()
print(index)

Below is the output of the above code. It tells us that the keyword with the highest similarity to the Birth Control phrase is at index 1490.

tensor(1490)

Now, let’s look at the keyword at index 1490 in the all_keywords variable.

# print the keyword at index 1490 
print(all_keywords[index])

Below is the output of the above code:

contraceptive

After examining it, we found that "contraceptive" was the word with the highest similarity, which makes sense because "birth control" and "contraceptive" mean the same thing. This demonstrates the elegance of semantic matching in finding similar words.

Let’s Also Explore Top 5 Keywords in the PDF that Match with the Phrase “Birth Control”

Let’s explore the 5 top keywords with the highest similarity score to “birth control” to see what the result would look like.

To do that, we can use the topk() method to get the top 5 indices. Then we can then loop through these indices to get the actual keywords:

# extract the top 5 indices
top_5_indices = cosine_similarity_result.topk(5)[1].tolist()[0]

print(top_5_indices)

Below is the result of the above code:

[1490, 1972, 871, 1199, 1944]

# get top 5 keywords
top_5_keywords = [all_keywords[index] for index in top_5_indices]
print(top_5_keywords)

Below is the output of the above code:

['contraceptive', 'contraception', 'contraceptive choices', 'range contraceptive', 'cover contraception']

There, we can see that the top five results relate to contraception and contraceptives. This demonstrates that semantic matching is an effective way to find related elements in a document.

Wrapping Up

In this article, you learned what semantic matching is and its advantages compared to traditional NLP search methods. You also encountered concepts such as word embeddings and cosine similarity and learned how they help us perform semantic matching. Then we implemented semantic matching by finding a phrase in a document.

Thank you for reading this article, and I will see you in the next one.

References

1. https://sbert.net/

2. https://maartengr.github.io/KeyBERT/guides/quickstart.html

3. https://huggingface.co/spaces/mteb/leaderboard

Or how your favorite voice assistant can comprehend complex questions?

The secret behind much of this smart technology is a powerful tool called BERT.

In this article, I’ll break down what BERT is, why it’s a game-changer in the world of natural language processing (NLP), and how you can get started with a simple code example.

What is BERT?

BERT stands for Bidirectional Encoder Representations from Transformers. It is an advanced method developed by Google for natural language processing (NLP).

It represents a shift in how computers understand human language.

Imagine you’re trying to understand a sentence with a word that has multiple meanings. For example, the word “bank” could refer to the side of a river or a financial institution. This is where BERT shines.

Instead of just looking at the words before it, like we usually read, BERT looks at the words before and after that word at the same time.

This way, the model gets a fuller picture of what the word means based on the entire sentence, not just part of it. It’s like having a conversation with someone who listens to everything you say before and after a question before answering it.

This bidirectional approach allows BERT to grasp the nuanced meanings of words within their specific context, leading to more accurate interpretations of text.

BERT supports many of the recent improvements in search engines, language translation services, and conversational AI.

Why BERT Matters

BERT excels at understanding the context, helping computers grasp the meaning of ambiguous language.

This has huge implications for improving search engines, translation services, and even generating text that feels more natural to humans.

• Understands context: BERT’s ability to understand the context of words in a sentence from both directions leads to more accurate interpretations of the meaning of text, which is crucial for understanding human language.
• Improves search engines: BERT has been used to enhance search engine algorithms, allowing them to better understand the intent behind users’ queries. This means that search results are more relevant and useful to what people are looking for.
• Enhances language-based applications: Applications like language translation, question-answering systems, and virtual assistants benefit significantly from BERT. They become more accurate and conversational, improving user experience and making technology more accessible.
• Handles ambiguity in language: BERT’s deep understanding of context helps it deal with ambiguity in language, distinguishing between different meanings of the same word based on context. This is crucial for accurate language interpretation and translation.
• Advances AI research: BERT represents a significant step forward in machine learning and AI research, pushing the boundaries of what’s possible in understanding and generating human-like text. It opens up new possibilities for AI applications and has set a new standard in the field of NLP.

Overall, BERT matters because it represents a leap forward in how machines understand and interact with human language, making technology more intuitive and effective in processing and generating text.

How BERT Works

Bert Architecture

BERT makes use of a transformer, an attention mechanism that learns contextual relations between words (or sub-words) in a text.

In its base form, a transformer includes two separate mechanisms — an encoder that reads the text input and a decoder that produces a prediction for the task. However, BERT only uses the encoder mechanism.

By adopting this approach, BERT models can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial modifications to the underlying model.

How to Work with BERT

Let’s build a simple sentiment analyzer using BERT. We will be using the Huggingface Transformer’s library to use a pre-trained model of BERT and use it to build a sentiment analyzer:

 from transformers import pipeline

# Load BERT model for text classification
classifier = pipeline("sentiment-analysis", model="bert-base-uncased")

# Define input text
text = "It was a fantastic movie and I loved it!"

# Perform sentiment analysis
result = classifier(text)

# Map output label to human-readable sentiment
if result[0]['label'] == 'LABEL_1':
    sentiment_label = 'Positive'
else:
    sentiment_label = 'Negative'

# Print result
print("Sentiment:", sentiment_label)
print("Score:", result[0]['score'])

Let’s look at what this code does:

• First, we import the required modules from the transformers library.
• Next, we load the pre-trained model and tokenizer. We specify a model name (bert-base-uncased) that represents a BERT model. The tokenizer is loaded to preprocess text in the way BERT expects (for example, converting text to lowercase).
• Next, we create a sentiment analysis pipeline from the transformers library. The pipeline function from Hugging Face Transformers abstracts away much of the manual work of preprocessing and applying the model. We specify sentiment analysisas the task to automatically handle tokenization, model inference, and output interpretation.
• We then give it an input, which in our case is a sentence to analyze the sentiment of. You can replace this with any text you want to analyze.
• Next, the example text is sent to the pipeline to get the sentiment analysis results.
• Finally, we print the sentiment of along with the confidence score (how confident the model is about the result).

Here is the output for the above code:

Sentiment: Negative
Score: 0.5871706604957581

The pipeline function makes it very straightforward to apply pre-trained models to specific tasks, including sentiment analysis. The label and score give you a quick understanding of the model's sentiment prediction and its confidence in that prediction, respectively.

This example provides a basic understanding of how you can use BERT for a sentiment analysis task. The model takes in a sentence, processes it to understand its context, and predicts its sentiment as either positive or negative.

Conclusion

BERT represents a significant leap forward in the ability of machines to understand and interact with human language. Its bidirectional training and context-aware capabilities enable a wide range of applications, from enhancing search engine results to creating more powerful chatbots.

By experimenting with BERT and other NLP models, you can begin to explore the vast potential of language understanding technologies. Whether you’re a seasoned developer or just starting, the world of NLP offers endless opportunities for innovation and improvement.

Remember, this example is just the beginning. As you dive deeper into BERT and NLP, you’ll discover more complex and powerful ways to use these tools. Happy coding!

Hope you enjoyed this article. Visit turingtalks.ai for daily byte-sized AI tutorials.

That's where a text summarizer comes in. Imagine it as a filter that separates the essential bits from the overwhelming flood of words.

In this article, I'll walk you through what a summarizer is, its use cases, what Hugging Face Transformers are, and how you can build your own text summarizer using Hugging Face Transformers. Let's dive in.

What is a Summarizer?

A summarizer does exactly what its name suggests. It takes a large block of text and condenses it into a shorter version.

This shorter version keeps only the key points. Think of it as the difference between reading a whole novel and glancing at its back cover. The aim is to save time while still getting the essence of the content.

Use Cases for a Summarizer

Summarizers are not just cool tech tricks. They serve real-world needs.

Journalists use them to quickly sift through reports and studies. Students use them to summarize lengthy readings. Businesses use them to condense market analyses or lengthy reports.

In essence, anyone who needs to process large amounts of text quickly can benefit from a summarizer.

What are Hugging Face Transformers?

Hugging Face is a company that has created a state-of-the-art platform for natural language processing (NLP).

Their Transformers library is like a treasure trove for NLP tasks. It includes pre-trained models that can do everything from translation and sentiment analysis, to yes, summarization.

These models have learned from vast amounts of text and can understand and generate language in a surprisingly human-like way.

How to Build a Summarizer with Hugging Face Transformers

Now, let's roll up our sleeves and start building. We will use the Huggingface pipeline to implement our summarization model using Facebook’s Bart model.

The BART model is pre-trained in the English language. It is a sequence-to-sequence model and is great for text generation (such as summarization and translation). It also works well for comprehension tasks (for example, text classification and question answering).

Hugging Face Pipelines offers a simpler approach to implementing various tasks. Instead of preparing a dataset, training it with the model and then using it, pipeline simplifies the code because it hides away the need for manual tokenization and model customisation.

How to set up your environment

First, you need to set up your coding environment. I prefer to use a Google Collab notebook instead of installing it on your local machine. Here is the notebook for this tutorial.

Let’s start by installing the transformer library. Use a ! before the command if you are running it in a collab notebook:

pip install transformers

Now let’s initialize a text summarization pipeline using the Hugging Face transformers library:

summarizer = pipeline("summarization", model="facebook/bart-large-cnn")

Let's break down what each part does:

• pipeline: This is a function provided by the Hugging Face transformers library to make it easy to apply different types of Natural Language Processing (NLP) tasks, such as text classification, translation, summarization, and so on. The function returns a ready-to-use pipeline object for the specified task.
• "summarization": This is the first argument to the pipeline function and specifies the type of task you want the pipeline to perform. In this case, "summarization" means that the pipeline will be configured to summarize text.
• model="facebook/bart-large-cnn": This argument specifies the pre-trained model to be used for the summarization task. Here, "facebook/bart-large-cnn" refers to a specific model that has been trained on a large dataset to perform text summarization. This model is provided by Facebook and is based on the BART (Bidirectional and Auto-Regressive Transformers) architecture, which is effective for tasks that require understanding and generating natural language. The large-cnn part indicates that this particular model variant is optimized for summarization tasks similar to those tackled by traditional CNN news-style summaries.

When this line of code is executed, it creates a summarizer object. This object can then be used to perform text summarization by passing text data to it. The model will generate a shorter version of the input text, capturing the most important or relevant information, according to its training on the summarization task.

Now we are ready to use the model to summarize our text (yeah, really!). Let's use an annual report from FDA and use it as input to get our summary.

text = """In 2023 generic drugs continued to play a critical role in the U.S. health care system allowing patients greater access to needed medicines. Generic drugs are generally lower cost than their brand-name equivalent and the approval of generic drugs often means multiple manufacturers for generic medicines, which can help stabilize the supply chain and reduce drug shortage risks.

The mission of the Office of Generic Drugs is to ensure high-quality, safe, and effective generic medicines are available to the American public. Our 2023 Annual Report provides highlights of activities and accomplishments including generic drug approvals, first generic approvals, science and research innovations for generic medicines – including complex generics, and international collaboration, as well as how we are doing on agreements made under the third iteration of the Generic Drug User Fee Amendments."""

Now let’s use this text as input and call our summarizer:

summary = summarizer(text, max_length=150, min_length=40, do_sample=False)

This line of code is using the summarizer object created from a Hugging Face pipeline to generate a summary of the input text. Here's a breakdown of the function call and its parameters:

• summarizer: This is the object initialized previously with the pipeline function.
• text: This is the input text that you want to summarize.
• max_length=150: This parameter specifies the maximum length of the summary in terms of the number of tokens (words and punctuation marks).
• min_length=40: Similarly, this parameter sets the minimum length of the summary.

Finally, we will print our summary:

print(summary[0]['summary_text'])

And here is the response:

In 2023 generic drugs continued to play a critical role in the U.S. health care system allowing patients greater access to needed medicines. Generic drugs are generally lower cost than their brand-name equivalent and the approval of generic drugs often means multiple manufacturers for generic medicines.

In short, this code:

• loads a summarization pipeline that is pre-configured to use the facebook/bart-large-cnn model.
• feeds the text to the summarizer.
• outputs a summary with specified minimum and maximum lengths.

Conclusion

Building a text summarizer with Hugging Face Transformers is not just about playing with cool technology. It's about harnessing the power of AI to make our lives a bit easier. Whether you're a student, a professional, or just someone curious about NLP, the ability to quickly condense information is invaluable.

With Hugging Face Transformers, you're standing on the shoulders of giants, leveraging some of the most advanced NLP models available today. So, give it a try. Who knows? It might just change the way you deal with text forever.

Thanks for reading this article. Find more AI tutorials at TuringTalks.ai.

Have you ever wondered how modern AI achieves such remarkable feats, like understanding human language or generating text that sounds like it was written by a person?

A significant part of this magic stems from a groundbreaking model called the Transformer. Many frameworks released into the Natural Language Processing (NLP) space are based on the Transformer model, and an important one is the Hugging Face Transformer Library.

In this article, I’ll walk you through why this library is not just another piece of software, but a powerful tool for engineers and researchers alike. Then you'll see a practical example of how to use it.

What is the Hugging Face Transformer Library?

The Hugging Face Transformer Library is an open-source library that provides a vast array of pre-trained models primarily focused on NLP. It’s built on PyTorch and TensorFlow, making it incredibly versatile and powerful.

One of the first reasons the Hugging Face library stands out is its remarkable user-friendliness. Even if you’re not a deep learning expert, you can use this library with relative ease.

It offers straightforward interfaces that allow you to implement complex models with just a few lines of code. This simplicity opens the doors of advanced AI to a broader range of developers and researchers.

Pre-Trained and Ready to Go

The beauty of today’s deep learning models is that you don't have to train a model from scratch. Most models are pre-trained and your job as an AI engineer will be to train a model using custom data.

So imagine having access to a toolbox where each tool is tailored for a specific job. That’s what Hugging Face offers with its wide range of pre-trained models.

Whether you’re working on text classification, question answering, or language generation, there’s a model ready for you to use. This saves an enormous amount of time and resources as you don’t have to start from scratch.

While pre-trained models are fantastic, they might not fit every specific need. This is where Hugging Face truly shines. The library allows you to fine-tune models on your dataset, making it possible to customize the models to your specific requirements.

Community Support

What sets Hugging Face apart is not just its technical capabilities but also its vibrant community. By engaging with this community, you gain access to a wealth of knowledge and support.

Users continuously contribute to the library, adding new models and features, making it a living, evolving ecosystem. This collaborative spirit ensures that the library stays at the cutting edge of AI research and application.

Performance and Scalability

In the world of AI, performance is key, and the Hugging Face library doesn’t disappoint. It’s designed to handle large-scale models efficiently, which means you can work with some of the most advanced AI models without needing a supercomputer at your disposal.

Hugging Face is also not just about English. It supports multiple languages, which is essential for organizations and developers aiming to create AI applications for a diverse user base.

Popular Hugging Face Models

1. BERT (Bidirectional Encoder Representations from Transformers): BERT excels in understanding the context of a word in a sentence, making it effective for tasks like sentiment analysis, question-answering, and language understanding. It’s widely used in chatbots, search engines, and to enhance user interaction with AI systems.
2. GPT (Generative Pretrained Transformer): Known for its ability to generate human-like text, GPT is used for creative writing, generating conversational responses, and even writing code. It’s particularly popular in chatbots, automated content creation tools, and customer service applications.
3. DistilBERT: A streamlined version of BERT, DistilBERT offers similar capabilities but is faster and requires less computational power. It’s ideal for environments where resources are limited, like mobile applications, and is used in tasks like text classification and information extraction.
4. RoBERTa (Robustly Optimized BERT Approach): An optimized version of BERT, RoBERTa is trained on a larger dataset and for a longer time, leading to improved performance. It’s used in more complex NLP tasks like sentiment analysis, language inference, and text classification.
5. T5 (Text-To-Text Transfer Transformer): T5 converts all NLP problems into a text-to-text format, providing a versatile approach to tasks like translation, summarization, and question answering. Its adaptability makes it valuable in diverse applications, from automated translation services to information summarization tools.

Each of these models has its unique strengths, and you should choose them based on the specific requirements of your tasks. Make sure to balance factors like computational resources, complexity of the task, and the desired level of performance.

How to Use the Hugging Face Transformers Library

Let me show you how easy it is to work with the Hugging Face Transformers library. We will implement a simple summarization script that takes in a large text and returns a short summary.

We will first import pipeline from the transformers library. In Hugging Face, a “pipeline” is like a tool that helps you perform a series of steps to change data into the form you want.

from transformers import pipeline

The pipeline makes it simple to use these tools for different jobs, without needing to know all the complex details about how these tools work on the inside. For this example, we will use the "summarization" pipeline.

summarizer = pipeline("summarization")

And we are now ready to start using the summarization pipeline. Let's pass in a long chunk of text and see what the response is.

text = """
The development of the internet has been one of the most transformative events in human history, altering virtually every aspect of modern life. Initially conceived as a military and academic network in the late 1960s, the internet evolved rapidly through the 1970s and 1980s, expanding its reach and capabilities with each passing year. The introduction of the World Wide Web in the early 1990s was a critical moment, making the internet much more accessible and user-friendly, sparking a global revolution in communication, business, and entertainment. As a tool for information dissemination, the internet has been unparalleled, allowing instant access to vast amounts of data from all over the world. It has democratized information, breaking down barriers that once existed due to geography or social status. The internet has also had a profound impact on commerce, giving rise to e-commerce and transforming traditional business models. The ease of online shopping and the rise of digital marketplaces have reshaped consumer habits and expectations. Socially and culturally, the internet has connected people across the globe, facilitating the exchange of ideas and cultures in a way that was previously unimaginable. However, it has also raised concerns about privacy, data security, and the digital divide. The rapid dissemination of information has sometimes led to the spread of misinformation, posing challenges for societies in discerning truth from falsehood. As the internet continues to evolve, it poses new challenges and opportunities, shaping the future of human interaction, governance, and technology.
"""

summary = summarizer(text)
print(summary[0]['summary_text'])

Here is a sample response:

 The introduction of the internet in the 1970s and 1980s was a major event for the world's first time . As a result, the internet has been able to connect people across the globe . The internet has also raised concerns about privacy and security in the digital age of 21.

That's how easy it is to work with the Hugging Face Transformers library.

Ethical AI and Transparency: A Step Towards Responsible AI

Since AI ethics are increasingly under the spotlight, Hugging Face commits to transparency and responsible AI development. The open-source nature of the library promotes a level of transparency that’s essential for ethical AI development. Users can see exactly how models are built and make informed decisions about their use.

AI is a field that never stands still, and neither does the Hugging Face Transformer Library. It’s continuously updated with the latest breakthroughs in AI research. This means that when you use Hugging Face, you’re always at the forefront of AI technology.

Finally, the real test of any tool is its applications in the real world, and here, Hugging Face excels. It’s used by academics for cutting-edge research and by companies for practical applications like sentiment analysis, content generation, and language translation.

Conclusion

In summary, the Hugging Face Transformer Library is more than just a collection of AI models. It’s a gateway to advanced AI for people of all skill levels. Its ease of use and the availability of a comprehensive range of models make it a standout library in the world of AI.

Whether you’re a seasoned AI expert or just starting, the Hugging Face library is a useful resource that can help you achieve your AI goals.

Hope you enjoyed this article. Find more beginner-friendly tutorials on AI at turingtalks.ai.

Imagine you’re running a business and you want to know what your customers think about your product. Or maybe you’re a movie director wanting to gauge the public reaction to your latest release.

This is where sentiment analysis comes into play.

Sentiment analysis is a technique used in text analysis that helps in identifying and categorizing opinions expressed in a piece of text.

Sentiment analysis determines whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral.

In a world where data is king, sentiment analysis is a crown jewel. It’s like having a superpower to understand the emotional tone behind words at scale.

Companies use it to understand customer feedback on products and services. Governments and organizations use it to get a sense of public opinion.

In social media management, sentiment analysis is used for brand monitoring, customer service, and market research.

It’s not just about understanding how many people are talking about your brand or product, but how they feel about it.

What is Hugging Face?

Now, let’s talk about Hugging Face. No, it’s not what you think. You don’t go around hugging faces.

In the world of AI, Hugging Face is quite the star. It’s an AI community and platform that provides state-of-the-art tools and models for Natural Language Processing (NLP).

Think of it as a toolbox that gives you the power to understand and generate human language. It’s like having a linguistic wizard by your side.

Hugging Face’s most popular offering is the ‘Transformers’ library. The Transformers library comes packed with APIs and tools that let you easily grab and train top-notch pre-trained models.

When you pick these pre-trained models, you’re cutting down on compute costs and carbon footprint. Plus, you save loads of time and resources that you’d otherwise spend training a model from scratch.

These models solve common tasks across various domains, like:

• Natural Language Processing (NLP): Here, you can do a bunch of cool stuff like text classification, spotting names or entities in text, answering questions, language modelling, summarizing, translating, handling multiple-choice questions, and even generating text.
• Computer Vision: This involves image classification, spotting and outlining objects in images, and more.
• Audio: You can work on recognizing speech automatically and classifying different types of sounds.
• Multimodal Tasks: These are tasks that mix it up, like answering questions based on tables, recognizing text in images (like scanned documents), pulling out information from these documents, classifying videos, and answering questions based on images.

The neat thing about Transformers is that they’re flexible with different frameworks. Whether you’re into PyTorch, TensorFlow, or JAX, Transformers has got you covered.

Its ease of use and comprehensive nature make it a go-to for researchers, developers, and businesses alike.

Code for Sentiment Analysis

Now that you know what sentiment analysis and Hugging Face are, let’s write some code. We’ll use Python and the Hugging Face transformers library to build a simple sentiment analyzer.

You can either use your terminal, install Python and run the code, or use a Google Colab notebook. I would recommend the latter since it comes pre-installed with Python.

Install the transformerslibrary with this command:

pip install transformers

If you are using a Colab notebook, use a ! symbol before the command for the notebook to treat it as a shell command (Colab executes code as Python by default).

!pip install transfomers

Once the installation is complete, you can start using the library. First, let's import pipeline from the transformers library.

from transformers import pipeline

In Hugging Face, a “pipeline” is like a tool that helps you perform a series of steps to change data into the form you want. The pipeline makes it simple to use these tools for different jobs, without needing to know all the complex details about how these tools work on the inside.

Now let’s load the sentiment-analysis pipeline.

sentiment_pipeline = pipeline("sentiment-analysis")

Now would you believe me if I said we are pretty much done? Our sentiment analysis model is ready and we can pass text to the pipeline and get the label as well as a sentiment score.

# Run sentiment analysis
result = sentiment_pipeline("Every new day brings a chance to create joyful memories and embrace new opportunities.")

# Print the result
print(result)

This is the output of the above code:

[{'label': 'POSITIVE', 'score': 0.9998821020126343}]

If you want to pass multiple sentences, pass an array of inputs to the pipeline.

result = sentiment_pipeline(["Every new day brings a chance to create joyful memories and embrace new opportunities.","Despite the effort, the project failed to meet expectations, leading to disappointment and frustration among the team."])
print(result)

Following will be the output of the above code:

[{'label': 'POSITIVE', 'score': 0.9998821020126343}, {'label': 'NEGATIVE', 'score': 0.9997937083244324}]

I hope you understand how powerful the Hugging Face Transformer library is. This is just a sample of the many pre-trained models that Hugging Face provides. Unless you are working on a unique problem, you should find a pre-trained model in Hugging Face available for you to work with.

Summary

In this article, we’ve learned about sentiment analysis and Hugging Face, a powerful tool in the world of NLP. Most importantly, you’ve taken your first steps in performing sentiment analysis by using the Hugging Face Transformers library.

Remember, what we’ve covered is just the tip of the iceberg. The field of NLP is vast and constantly evolving. The Hugging Face Transformers library is a powerful ally in your journey through AI. It simplifies complex tasks and gives you access to pre-trained models, saving you time and resources.

Hope you enjoyed this article. Find more beginner-friendly articles on AI at turingtalks.ai

For example, you can analyze a large volume of textual content to extract a common theme. Companies and businesses utilize this technique to understand public opinion about their brand. This lets them make informed decisions and improve their services.

The ability to extract themes from a large amount of textual data is referred to as topic identification.

In this article, you will learn how to utilize NLP techniques for topic identification, enhancing your skillset as a data scientist. So sit back, because it's gonna be an interesting journey.

What is Topic Identification?

Topic identification, simply put, is a sub-field under natural language processing. It involves the process of automatically discovering and organizing the main themes or topics present in a collection of textual data.

There are several Natural Language Processing (NLP) techniques you can use to identify themes in text, from simple ones to more algorithm based techniques. In this article we will look at the common NLP techniques used for topic identification. We'll discuss these in more detail below.

I recently tweeted about the essence of NLP. It really is purely statistics, because there are different manipulations you can do to ensure that numbers serve as representations for text (since computers don't understand text).

Requirements for this Project

In order for you to be able to follow along and get hands-on practical experience while learning, you should have Python 3.x installed on your machine.

We'll also use the following libraries: Gensim, Scikit-Learn, and NLTK. You can install them using the Pip package installer with the following command:

pip install gensim nltk scikit-learn

Techniques Used in NLP for Topic Identification

There are various techniques you can use for topic identification. In this article, you will learn about some common NLP techniques that work quite well, from simple and effective methods to more advanced ones.

Bag of Words

Bag of Words (BoW) is a common representation used in NLP for textual data. You can use it to count the frequency at which each word occurs in a document.

BoW, in the context of topic identification, is based on the assumption that the more frequently a word occurs in a document, the more important it is. Then you can use those more common words to infer what the document is all about.

Bag of words is the simplest technique used to identify topics in NLP. While Bag of Words is simple and efficient, it is highly affected by stop words, which are common words in text data (like "the," "and," "is," and so on).

But once you eliminate the issue of stop words from the text, allowing you to perform effective text processing (using techniques like normalization), BoW can still prove effective in identifying some main topics.

Let's look at how you can use BoW to identify the topic below.

How to implement of Bag of Words in Python

A bit of background about the example article we'll use here: I got it from the BBC, and it's titled "US lifts ban on imports of latest Apple watch." The article discusses the lifted ban on Apple's latest watches, Ultra 2 and Series 9.

Now let's go over how to implement the bag of words in Python. I'll break this code block up into sections and explain each part as I go to make it a bit more easy to digest.

#import necessary libraries
from collections import Counter
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

article = "Apple's latest smart watches can resume being sold in the US after the tech company filed an emergency appeal with authorities.\
Sales of the Series 9 and Ultra 2 watches had been halted in the US over a patent row.\
The US's trade body had barred imports and sales of Apple watches with technology for reading blood-oxygen level.\
Device maker Masimo had accused Apple of poaching its staff and technology. \
It comes after the White House declined to overturn a ban on sales and imports of the Series 9 and Ultra 2 watches which came into effect this week.\
Apple had said it strongly disagrees with the ruling.\
The iPhone maker made an emergency request to the US Court of Appeals, which proved successful in getting the ban lifted."

In the above code, we're importing the necessary libraries that we'll use to implement the BoW.

We'll use the Counter library to count the frequency of each word, and the word_tokenize library to tokenize the document into individual word tokens so they can be counted. Lastly, the stopwords library will remove stop words from the document.

# Initialize english stopwords
english_stopwords = stopwords.words("english")

#convert article to tokens
tokens = word_tokenize(article)

#extract alpha words and convert to lowercase
alpha_lower_tokens = [word.lower() for word in tokens if word.isalpha()]

#remove stopwords
alpha_no_stopwords = [word for word in alpha_lower_tokens if word not in english_stopwords]

#Count word
BoW = Counter(alpha_no_stopwords)

#3 Most common words
BoW.most_common(3)

In the above code, we use the first line of code to extract all stop words in the English language. Then, the second line tokenizes the article string into individual words. The third line of code normalizes each word into lowercase and only extracts alphabetic words from the article. The last two lines of code are used to count the frequency of each word and select the most common three words.

Below is the output of the BoW model:

[('watches', 4), ('us', 4), ('apple', 3), ('emergency', 2)]

From this, we can infer that the article is all about "Apple's watches in the US". As you can see, with the simplicity in reasoning behind the bag of words, it is still possible to infer a bit of knowledge about the article.

Latent Dirichlet Allocation

Latent Dirichlet Allocation, or LDA for short, is a popular probabilistic model used in NLP and machine learning for topic modeling (using algorithms to identify topics). It is based on the assumption that documents are mixtures of topics, and topics are mixtures of words.

Simply put, LDA is an NLP technique used to identify the topic to which a document belongs based on the words contained in the document.

LDA operates on the bag-of-words representation of documents, where each document is represented as a vector of word frequencies. You can implement LDA using the Gensim library in Python (which is an open source library used for topic modelling and document similarity analysis).

Steps for implementing LDA include:

• Import Libraries: First step is to import the necessary libraries you will be utilizing.

• Data Preparation: Convert raw data to a document format then tokenize, remove stop words, and optionally perform stemming or lemmatization.

• Create Dictionary and Corpus: Build a dictionary with unique word IDs. Then form a bag of words corpus representing document-word frequency.

• Train LDA Model: Use the document-word frequency and dictionary to train the LDA model, setting the desired number of topics.

• Print Topics: Explore and print the discovered topics.

# Import the necessary libraries
from gensim.corpora.dictionary import Dictionary
from gensim.models import LdaModel
from nltk import sent_tokenize, word_tokenize
from nltk.corpus import stopwords

article = "Apple's latest smart watches can resume being sold in the US after the tech company filed an emergency appeal with authorities. \
Sales of the Series 9 and Ultra 2 watches had been halted in the US over a patent row. \
The US's trade body had barred imports and sales of Apple watches with technology for reading blood-oxygen level. \
Device maker Masimo had accused Apple of poaching its staff and technology. \
It comes after the White House declined to overturn a ban on sales and imports of the Series 9 and Ultra 2 watches which came into effect this week. \
Apple had said it strongly disagrees with the ruling. \
The iPhone maker made an emergency request to the US Court of Appeals, which proved successful in getting the ban lifted."

The above lines of code include the necessary libraries that we'll use to implement the LDA.

The first line of code contains the Dictionary object. Then, the second line imports the LDA model, and the third line of code contains the sent_tokenize, which we'll use to convert the article into document. After that, word_tokenize will tokenize the document into individual words. Lastly, we have the stop_words library.

# convert article to documents
documents = sent_tokenize(article)

#toeknize and normalize the document
tokenized_words = [word_tokenize(doc.lower()) for doc in documents]

# remove stops words and onl extract alphabets
cleaned_token = [[word for word in sentence if word not in english_stopwords and word.isalpha()]
                 for sentence in tokenize_words]

# create a dictionary
dictionary = Dictionary(cleaned_token)

# Create a corpus from the document
corpus = [dictionary.doc2bow(text) for text in cleaned_token]

The above lines of code include the preprocessing steps that will be performed on the article, including converting the article to a document, normalizing, and tokenizing the document into individual words.

The next part removes stopwords from the text and then extracts words and numbers from the document. After that, we create a dictionary, which is a map between each word and its numerical identifier. The last line of code then creates a corpus of the document.

# Build the LDA model
model = LdaModel(corpus=corpus, id2word=dictionary, num_topics=3)

# Print the topics
print("Identified Topics:")
for idx, topic in lda_model.print_topics():
    print(f"Topic {idx + 1}: {topic}")

The above code is used to train the model on the corpus and then prints the top 3 topics from the article.

Below is the output of the LDA Model:

Identified Topics:
Topic 1: 0.045*"9" + 0.045*"ultra" + 0.044*"sales" + 0.044*"2" + 0.043*"series" + 0.043*"watches" + 0.029*"apple" + 0.028*"ruling" + 0.028*"disagrees" + 0.028*"said"
Topic 2: 0.051*"maker" + 0.035*"ban" + 0.035*"us" + 0.031*"emergency" + 0.031*"made" + 0.031*"successful" + 0.031*"court" + 0.031*"lifted" + 0.031*"request" + 0.031*"proved"
Topic 3: 0.055*"apple" + 0.054*"us" + 0.054*"watches" + 0.031*"sales" + 0.031*"technology" + 0.031*"imports" + 0.031*"authorities" + 0.031*"barred" + 0.031*"appeal" + 0.031*"filed"

The LDA technique shows some improvement as compared to BoW method. We can still obtain a more information that the article is all about a ban related to Apple ultra series watches in the US.

Non-Negative Matrix Factorization

Non-Negative Matrix Factorization (NMF), just like LDA, is another topic modeling technique that uncovers latent topics in a collection of documents.

But instead of relying on BoW, it relies on the Term Frequency-Inverse Document Frequency (TF-IDF) representation to capture and retrieve hidden themes or topics from the documents.

By incorporating TF-IDF information, NMF is able to weigh the importance of terms, thereby identifying more hidden patterns. You can perform NMF using the Scikit-learn library.

Steps for performing NMF

• Import necessary libraries

• Data Preparation: Convert text into document, then perform necessary data preparation like removing stop words. The TF-IDF function in Scikit-Learn has as an argument that does that.

• Convert the document to a TF-IDF matrix using the TF-IDF vectorizer in Scikit-learn

• Apply the NMF function on the TF-IDF matrix and specify the numbers of topic you want and the number of words in each topic

• Lastly, interpret your result.

# import the necessary libraries
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF

article = "Apple's latest smart watches can resume being sold in the US after the tech company filed an emergency appeal with authorities. \
Sales of the Series 9 and Ultra 2 watches had been halted in the US over a patent row. \
The US's trade body had barred imports and sales of Apple watches with technology for reading blood-oxygen level. \
Device maker Masimo had accused Apple of poaching its staff and technology. \
It comes after the White House declined to overturn a ban on sales and imports of the Series 9 and Ultra 2 watches which came into effect this week. \
Apple had said it strongly disagrees with the ruling. \
The iPhone maker made an emergency request to the US Court of Appeals, which proved successful in getting the ban lifted."

The above code contains the libaries that we'll use to implement NMF and the article itself.

# convert article to documents
documents = sent_tokenize(article)

# Create a TF-IDF vectorizer
tfidf_vectorizer = TfidfVectorizer(stop_words='english').fit_transform(document)

# Apply NMF
num_topics = 5  # Set the number of topics you want to identify
nmf_model = NMF(n_components=num_topics, init='random', random_state=42)
nmf_matrix = nmf_model.fit_transform(tfidf)

The above code converts the article into documents. Then it creates a Term-Frequency Inverse Document Frequency matrix of the article document. The last three lines of code then define the number of topics and create the topics from the document matrix using the NMF.

Below is the output of the NMF Model:

Topic #1: ultra, series, sales, watches, row, halted, patent, white, house, effect
Topic #2: lifted, court, iphone, getting, request, successful, proved, appeals, ban, maker
Topic #3: disagrees, strongly, ruling, said, apple, body, blood, level, trade, oxygen
Topic #4: filed, resume, appeal, latest, tech, authorities, sold, smart, company, emergency
Topic #5: technology, apple, accused, masimo, device, staff, poaching, maker, trade, level

You can see that NMF reveals more insights concerning the themes of the document. For example, you can tell that another company called Masimo is accusing Apple of a patent infringement in their Ultra series watches.

How to Choose Which Technique to Use?

I recommend experimenting with all the approaches in order to gain different perspectives concerning the contents of your document.

Bag of Words and LDA are based on how frequently words occur, making these techniques useful for inferring the biggest/most general themes about the document.

On the other hand, when using NMF, which is based on TF-IDF, less frequent words can be used to infer additional topics and provide a different perspective on the document.

For example, NMF was able to identify key terms like "Masimo" and "accused," whereas LDA was not able to do this. So depending on your needs, go ahead and experiment with all the approaches to see which one is able to yield better results.

Conclusion

In this article, you've learned about topic identification and how you can use it to extract themes or topics from a large document.

We covered some different techniques you can use to identify topic including simple ones like BoW and more advanced ones like LDA and NMF.

Happy learning, and see you in the next one.

What is Hugging Face 🤗?

If you are interested in Artificial Intelligence and Natural Language Processing, you have probably heard of Hugging Face – the company named after a cute emoji.

Hugging Face is not only a company, but also a platform that is transforming the fields of AI and NLP through open source and open science.

Hugging Face offers a platform called the Hugging Face Hub, where you can find and share thousands of AI models, datasets, and demo apps. The Hub is like the GitHub of AI, where you can collaborate with other machine learning enthusiasts and experts, and learn from their work and experience.

Hugging Face’s mission is to democratize good machine learning, one commit at a time. Whether you are a beginner or a professional, you can benefit from the amazing resources and tools that Hugging Face provides.

In this post, I'll guide you through the basics of Hugging Face. You'll learn how to create your Hugging Face account, set up your development environment, and use some of the pre-trained models that are available on the Hub. Let’s get started! 🚀

Here's what we'll cover:

1. What can you do on the Hugging Face Platform?
   • Download and fine-tune existing Open Source models
   • Run models directly from Hugging Face
   • Add/create your own model
   • Use existing datasets
   • Create/browse demo apps (also known as Spaces)
   • Join or create an organization
   • Create a portfolio
   • Learn AI skills
2. Hugging Face terminology
3. How to get started with Hugging Face
   • Create a Hugging Face account
   • Set up your environment
4. How to use pre-trained models in Hugging Face
5. How to find the right pre-trained model
6. What's next?

What Can You Do on the Hugging Face Platform?

Here are some of the awesome things you can do on Hugging Face:

Download and fine-tune existing Open Source models:

Why start from scratch when you can leverage the power of over 450k models that are already available on the Hugging Face model library?

You can easily download these models and fine-tune them on your own custom dataset with just a few lines of code. This way, you can save time and resources, and still get a model that suits your specific needs.

You can use these models to perform various tasks, such as:

1. Natural language processing (for example, translation, summarization, and text generation)
2. Audio-related functions (for example, automatic speech recognition, voice activity detection, and text-to-speech)
3. Computer vision tasks (for example, depth estimation, image classification, and image-to-image processing),
4. Multimodal models capable of handling diverse data types (text, images, audio) and producing multiple types of output.

Run Models directly from Hugging Face:

If you don’t want to set up these models on your own machines, you can simply use Hugging Face’s Transformer library to connect to these models, send requests, and receive outputs.

Add/create your own model:

If you have a brilliant idea for a new model, or you want to improve an existing one, you can also add/create your own model on Hugging Face.

The platform will host your model, and allow you to provide additional information, upload essential files, and manage different versions. You can also choose whether your models are public or private, so you can decide when or if you want to share them with the world.

Once your model is ready, you can access it directly from Hugging Face, send requests, and retrieve the outputs for integration into any applications you are developing.

Use existing datasets:

A good model needs a good dataset. Hugging Face provides a repository of over 90,000 datasets that you can use and feed into your models.

You can take an in-depth look inside the dataset using the dataset viewer. You can also contribute your own datasets to the repository, and help the machine learning community grow.

Screenshot of dataset viewer

Create/browse demo apps (also known as Spaces):

Hugging Face’s Spaces are Git repositories that allow you to showcase your machine learning applications. You can also browse and try out the Spaces created by other users, and find inspiration for your next AI app.

With thousands of ML apps to choose from, you will never run out of fun and interesting things to do.

Here are a few cool Spaces you can check out:

• OpenAI's Whisper: Transcribe long-form microphone or audio inputs with the click of a button.
• AI Comic Factory: Create your own comic books.
• QR Code AI Art Generator: Generate beautiful QR codes using AI.
• Stable Video Diffusion (Img2Vid - XT): Generate 4s video from a single image.
• Video-LLaMA: Audio-Visual Language Model for Video Understanding.

Join or create an organization:

You can join or create your own organization on Hugging Face. This allows you to showcase your work and collaborate with other members from your university, lab, or company. You can also work on private datasets, models, and spaces with your organization.

Create a portfolio:

You can create a professional portfolio on Hugging Face to showcase your work and start building your reputation. This can help you land jobs related to AI model training, integration, and development.

Hugging Face provides the basic computing resources for running the demo app, including 16 GB of RAM, 2 CPU cores, and 50 GB of disk space for free. You can also upgrade your hardware for improved and faster performance with paid options.

Learn AI Skills:

Hugging Face is an excellent platform for learning AI skills. It offers a comprehensive set of tools and resources for training and using models. This includes demos, use cases, documentation, and tutorials that guide you through the entire process of using these tools and training models.

You can also learn from the experts and the community on Hugging Face, and improve your AI knowledge and skills.

Hugging Face Terminology

There are some terms you'll need to know to get the most out of working with Hugging Face.

Pretrained model: A model that has been trained on a large dataset for a specific task before being made available for use.

Inference: Inference is the process of using a trained model to make predictions or draw conclusions about new, unseen data based on the learned patterns from the training data.

Transformers: Transformers are models that can handle text-based tasks, such as translation, summarization, and text generation. They use a special architecture that relies on attention mechanisms to capture the relationships between words and sentences.

Tokenizer: A tokenizer is a process that breaks down text into smaller units called tokens. Tokens are usually words or subwords that can be used for natural language processing (NLP) tasks.

How to Get Started with Hugging Face

To get started with HuggingFace, you will need to set up an account and install the necessary libraries and dependencies. Don’t worry, it’s easy and fun!

Here are the steps you need to follow:

Create a Hugging Face Account

Signing up as a Community individual contributor is free of charge. You can also opt for a ‘Pro’ plan or a customized plan for Organizations if you need more features and resources.

Go to the Hugging Face website and click on “Sign Up” to create a free account.

Then enter your email address and a password. Click next and complete your profile and security check.

Setting up a Hugging Face account

Congratulations, you are now a Hugging Face member! 🎉 You will be directed to the Hugging Face ‘Welcome’ page, where you can find more information and tips on how to use the platform.

As a bonus, you also get a Git-based hosted repository where you can create your Models, Datasets and Spaces. You can do this directly using the website or using the CLI. If you prefer the latter, you can check the detailed instructions on the ‘Welcome’ page under the ‘Programmatic access’ section.

Hugging Face welcome screen showing options to create a new model, browse the docs, and set up programmatic access

Set Up Your Environment

Before you start using the Hugging Face hub programmatically, you will need to set up your environment.

Step 1: Install Python and Pip:

Make sure you have Python 3.8 or higher installed on your system. You will also need Pip, the package manager for Python, to install the Hugging Face libraries. If you don’t have Python, you can install it by following the instructions here.

Step 2: Install HuggingFace libraries:

Open a terminal or command prompt and run the following command to install the HuggingFace libraries:

pip install transformers

This will install the core Hugging Face library along with its dependencies. To have the full capability, you should also install the datasets and the tokenizers library.

pip install tokenizers, datasets

Step 3: Set up a development environment:

Choose a code editor or IDE of your choice, such as Jupyter Notebook, PyCharm, or Visual Studio Code. Create a new project directory and set up a virtual environment to isolate your project dependencies. You can find more information on how to do this here.

With these steps completed, you have successfully set up Hugging Face on your system and are ready to start exploring its features and capabilities. Let’s go! 🚀

How to Use Pre-Trained Models in Hugging Face

One of the best things about Hugging Face is that it gives you access to thousands of pre-trained models that can perform various tasks on different types of data. Whether you are working with text, vision, audio, or a combination of them, you can find a model that suits your needs.

Hugging Face has two main libraries that provide access to pre-trained models: Transformers and Diffusers. The Transformers library handles text-based tasks, such as translation, summarization, and text generation. Diffusers can handle image-based tasks, such as image synthesis, image editing, and image captioning.

You have already installed the transformers library during the environment setup. Let’s see how you can use it to work with pre-trained models.

Step 1: Visit the PyPI page

To learn more about the transformers library, you can visit its page on PyPI, the Python Package Index.

Go to PyPi and search for ‘transformers’. Click on the latest version of the transformers library displayed in the search result. You will see a brief introduction of the library, as well as some useful links and information.

Step 2: Download and use pre-trained models

The transformers library provides APIs to quickly download and use pre-trained models on a given text, fine-tune them on your own datasets, and then share them with the community on Hugging Face’s model hub.

Step 3: Use the pipeline() method

To use a pre-trained model on a given input, Hugging Face provides a pipeline() method, an easy-to-use API for performing a wide variety of tasks.

The pipeline() method makes it simple to use any model from the Hub for inference on any language, computer vision, speech, and multimodal tasks.

Let’s try to perform a task using the pipeline() method.

Task: Sentiment analysis:

Let’s use the pipeline() method to classify positive versus negative texts provided by the user:

from transformers import pipeline

# Load the pre-trained sentiment analysis model
sentiment_analysis = pipeline(
"sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

input_text = [
"It’s a great app, my biggest problem is the card readers regularly do not connect. Which is very poor customer service for us because we have to manually enter our customers debit cards, which takes time. This slows down our efficiency."
]

# Perform sentiment analysis on the input text
result = sentiment_analysis(input_text)

# Print the result
print(result)

The pipeline statement downloads and caches the pretrained model used by the pipeline, while the statement result = sentiment_analysis(input_text) evaluates it on the given text.

Output:

[{'label': 'NEGATIVE', 'score': 0.9996176958084106}]

Here, the answer is "NEGATIVE" with a confidence of 99.96%.

Task: Automatic speech recognition

Let’s try another task that involves speech recognition.

from transformers import pipeline

transcriber = pipeline(task="automatic-speech-recognition",
                       model="openai/whisper-small")
result = transcriber(
    "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")

print(result)

Output:

{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}

You can see how easy it is to get a pre-trained model up and running using Hugging Face's libraries.

How to find the right pre-trained model

But how can you find the right pre-trained model if you want to perform a specific task?

This is actually quite easy. You can browse the models on the Hugging Face website, and filter them by task, language, framework, and more. You can also search for models and datasets by keyword and sort them by trending, most likes, most downloads, or by recent updates.

Searching for models

Each model has a model card that contains important information, such as model details, inference example, training procedure, community interaction features, and link to the files. You can also try the model on the model card page by using the Inference API section.

Inference API

You can also check the list of spaces that are using that particular model and further explore the spaces by clicking on the space link.

Spaces

What's Next?

In this guide, you have learned the basics of Hugging Face, and how to use its libraries, models, datasets, and spaces. But there is so much more to discover and enjoy!

Here are some tips on how to make the most of Hugging Face:

• Dive into Hugging Face’s Spaces: Spaces are where the magic happens. You can find and try out thousands of machine learning applications created by the community, and see what’s trending and popular. You can also create your own spaces and showcase your work to the world.
• Explore the Hugging Face documentation and tutorials: If you want to learn more about the Hugging Face platform and its features, you can check out the documentation and tutorials. They provide detailed information and guidance on how to use the tools and resources that Hugging Face offers. You can also find information about common ML/AI tasks, such as text classification, image generation, and speech recognition, on the tasks page.
• Visit the learn section: If you are interested in acquiring new skills and knowledge in AI and NLP, you can visit the ‘learn’ page that displays courses from Hugging Face. Here, you can learn from the experts and the best practices in the field, and apply them to your own projects.
• Join the Hugging Face community: Machine learning is more fun when collaborating! You can join the Hugging Face community on platforms like GitHub, Discord, and Twitter to connect with other users and stay updated on the latest developments. You can also share your feedback, questions, and ideas with the community, and help Hugging Face grow and improve.

Hugging Face is not just a platform for AI and NLP – it's also a playground for your curiosity and creativity. You can experiment with new models, expand your AI knowledge, and enrich your AI toolkit with various tools and resources. So, keep learning, keep exploring. There is always something new and exciting to discover with Hugging Face. 😊

This article takes you through one of the most basic steps in NLP which is text-pre-processing. This is a must-know topic for anyone interested in language models and NLP in general which is a core part of the Artificial Intelligence (AI) and ML field.

What is text pre-processing?

Text pre-processing is the process of transforming unstructured text to structured text to prepare it for analysis.

When you pre-process text before feeding it to algorithms, you increase the accuracy and efficiency of said algorithms by removing noise and other inconsistencies in the text that can make it hard for the computer to understand.

Making the text easier to understand also helps to reduce the time and resources required for the computer to pre-process data.

Processes involved in text pre-processing

To properly pre-process your text and get it in the right state to perform further analysis and actions with it, there are quite a few operations that need to be done on the text and a couple of steps to be followed to get a well structured text.

Let's go over these processes in the following sub-sections.

Tokenization

Tokenization is the first stage of the process.

Here your text is analysed and then broken down into chunks called ‘tokens’ which can either be words or phrases. This allows the computer to work on your text token by token rather than working on the entire text in the following stages.

The two main types of tokenisation are word and sentence tokenisation.

Word tokenisation is the most common kind of tokenisation.

Here, each token is a word, meaning the algorithm breaks down the entire text into individual words:

text = 'Wisdoms daughter walks alone. The mark of Athena burns through rome'

words = text.split()
print(words)

#the output of this is given below
>>>> ['Wisdoms', 'daughter', 'walks', 'alone.', 'The', 'mark', 'of', 'Athena', 'burns', 'through', 'rome']

On the other hand, sentence tokenisation breaks down text into sentences instead of words. It is a less common type of tokenisation only used in few Natural Language Processing (NLP) tasks.

There are various tokenisation algorithms such as the whitespace tokenisation, Regular expression tokenisation (also called Regex), and the statistical tokenisation.

The type of algorithm you use will depend on the particular task you are working on and what you aim to achieve with it.

Normalisation

In normalisation your text is converted to standard form.

An example of this is converting all text to lowercase, removing numbers, or removing punctuations. Normalization helps to make the text more consistent.

There are a couple of different normalisation techniques, but I’ll give you an explanation of some of the most commonly employed normalisation techniques below.

Case normalisation

This technique converts all the letters in your text to a single case, either uppercase or lowercase.

Case normalisation ensures that your data is stored in a consistent format and makes it easier to work with the data.

An example would be looking for all the instances of a word and searching for it in your text. Without case normalisation, the result of searching for the word ‘Boy’ would be different from the result of searching for ‘boy’.

You can use the following code to perform case normalisation:

text = "'To Sleep Or NOT to SLEep, THAT is THe Question'"

def lower_case(text):
    text = text.lower()
    return text

lower_case = lower_case(text)#converts everthing to lowercase
print(lower_case)

#the output of this is given below
>>>> to sleep or not to sleep, that is the question

Stemming

Stemming words like coding, coder, and coded all have the same base word which is code.

ML models most-often-than-not understand that these words are all derived from one base word. They can work with your text without the tenses, prefixes, and suffixes that we as humans would normally need to make sense of it.

Stemming your texts not only helps to reduce the number of words the model has to work with, and by extension improves the efficiency of the model.

Although the efficiency of a model is increased with this technique, it also removes important information from your text and could cause some words to be wrongly categorised by the model.

An example of this would be the difference between writing and write in the sentences below:

💡 Writing makes me happy.

💡 He writes regularly.

In the first sentence the word writing represents a noun, while writes in the second sentence represents a verb.

If your ML models stems both writing and writes to the base write the difference in their respective parts of speech is overlooked causing some information to be lost in the process of analysing the text.

Lemmatisation

This method is very similar to stemming in that it is also used to identify the base of words. It is however a more complex and accurate technique than stemming.

Unlike stemming, lemmatisation takes in the structure of words before identifying a base word.

Due to the complexity of this technique it has high computational requirements and is therefore more expensive than stemming.

Punctuation removal

During human conversations, punctuation marks like ‘’, ! , [, }, *, #, /, ?, and ‘’ are incredibly relevant and necessary to have a proper conversation. Thelp to fully convey the message of the writer.

ML models on the other hand find punctuations distracting.

Their presence could interfere with text analysis and the natural language processing (NLP) process.

By removing punctuation marks from our text we allow the model to focus on the text alone rather than distracting it with symbols. This makes it easier for the text to be analysed.

To perform punctuation removal on text the following code can be used:

import re

text = ' (to love is to destroy, and to be loved, is to be "the" one <destroyed>} '

def remove_punctuations(text):
    punctuation = re.compile(r'[{};():,."/<>-]')
    text = punctuation.sub(' ', text)
    return text

clean_text = remove_punctuations(text)
print(clean_text)

#the output of this is given below
>>>> to love is to destroy  and to be loved  is to be  the  one  destroyed

Accent removal

This process is about removing language specific character symbols from text.

Some characters are written with specific accents or symbols to either imply a different pronunciation or to signify that words containing such accented texts have a different meaning.

An example of this would be the difference in both meaning and pronunciation between the words résumé and resume.

The former refers to a document that highlights your professional skills and achievements, whereas the latter means ‘to take on something again, or to continue a previous task or action’.

You can use the code below to perform accent removal on your text:

import re

text = "her fiancé's résumé is beautiful"

def remove_accents(text):
    accents = re.compile(u"[\u0300-\u036F]|é|è")
    text = accents.sub(u"e", text)
    return text

cleaned_text = remove_accents(text)
print(cleaned_text)

#the output of this is given below
>>>> her fiance's resume is beautiful

Stop-word removal

Stop-words are words with no meaning. They don't add any additional value to data.

Words like A, the, and, of and so on are called stop-words.

Like all the previous processes, stop-word removal also helps to increase the efficiency of your model.

Since it reduces the size of our dataset, it makes it more manageable and increases the accuracy of NLP tasks.

Conclusion

In this article you learned the basics of NLP.

You are now familiar with the proper procedure to follow when pre-processing your text for NLP tasks. Feel free to go ahead and practice this on your own and work on a few NLP projects.

Note that choosing the right pre-processing technique / techniques to use on your text will depend largely on the type of text you’re working with, the source of your data, and whatever goal you aim to achieve with it.

To learn more about NLP, you can check out FreeCodeCamp for more articles and courses on NLP and ML in general.

Connect with me on twitter @Iqma and follow my hashnode blog to read more content like this and to learn more about all things AI and Machine Learning.

Happy learning !

Topic Modeling answers the question: "Given a text corpus of many documents, can we find the abstract topics that the text is talking about?"

In this tutorial, you’ll:

• Learn about two powerful matrix factorization techniques - Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF)
• Use them to find topics in a collection of documents

By the end of this tutorial, you'll be able to build your own topic models to find topics in any piece of text.📚📑

Let's get started.

Table of Contents

1. What is Topic Modeling?
2. TF-IDF Score Equation
3. Topic Modeling Using Singular Value Decomposition (SVD)
4. What is Truncated SVD or k-SVD?
5. Topic Modeling Using Non-Negative Matrix Factorization (NMF)
6. 7 Steps to Use SVD for Topic Modeling
7. How to Visualize Topics as Word Clouds
8. How to Use NMF for Topic Modeling
9. SVD vs NMF – An Overview of the Differences

What is Topic Modeling?

Let's start by understanding what topic modeling is.

Suppose you're given a large text corpus containing several documents. You'd like to know the key topics that reside in the given collection of documents without reading through each document.

Topic Modeling helps you distill the information in the large text corpus into a certain number of topics. Topics are groups of words that are similar in context and are indicative of the information in the collection of documents.

The general structure of the Document-Term Matrix for a text corpus containing M documents, and N terms in all, is shown below:

Structure of the Document-Term Matrix

Let's parse the matrix representation:

• D1, D2, ..., DM are the M documents.
• T1, T2, ..., TN are the N terms

To populate the Document-Term Matrix, let’s use the widely-used metric—the TF-IDF Score.

TF-IDF Score Equation

The TF-IDF score is given by the following equation:

where,

• TF_ij is the number of times the term Tj occurs in the document Di.
• dfj is the number of documents containing the term Tj

A term that occurs frequently in a particular document, and rarely across the entire corpus has a higher IDF score.

I hope you’ve now gained a cursory understanding of the DTM and the TF-IDF score. Let’s now go over the matrix factorization techniques.

Topic Modeling Using Singular Value Decomposition (SVD)

The use of Singular Value Decomposition (SVD) for topic modeling is explained in the figure below:

Singular Value Decomposition on the the Document-Term Matrix D gives the following three matrices:

• The left singular vector matrix U. This matrix is obtained by the eigen decomposition of the Gram matrix D.D_T—also called the document similarity matrix. The i,j-th entry of the document similarity matrix signifies how similar document i is to document j.
• The matrix of singular values S, which (values) signify the relative importance of topics.
• The right singular vector matrix V_T, which is also called the term topic matrix. The topics in the text reside along the rows of this matrix.

If you'd like to refresh the concept of eigen decomposition, here's an excellent tutorial by Grant Sanderson from 3Blue1Brown. It explains eigenvectors and eigenvalues visually.

Embedded content

It's totally fine if you find the working of SVD a bit difficult to understand. 🙂 For now, you may think of SVD as a black box that operates on your Document-Term Matrix (DTM) and yields 3 matrices, U, S, and V_T. And the topics reside along the rows of the matrix V_T.

Note: SVD is also called Latent Semantic Indexing (LSI).

What is Truncated SVD or k-SVD?

Suppose you have a text corpus of 150 documents. Would you prefer skimming through 150 different topics that describe the corpus, or would you be happy reading through 10 topics that can convey the content of the corpus?

Well, it's often helpful to fix a small number of topics that best convey the content of the text. And this is what motivates k-SVD.

As matrix multiplication requires a lot of computation, it's preferred to choose the k largest singular values, and the topics corresponding to them. The working of k-SVD is illustrated below:

Topic Modeling Using Non-Negative Matrix Factorization (NMF)

Non-negative Matrix Factorization (NMF) works as shown below:

Non-negative Matrix Factorization acts on the Document-Term Matrix and yields the following:

• The matrix W which is called the document-topic matrix. This matrix shows the distribution of the topics across the documents in the corpus.
• The matrix H which is also called the term-topic matrix. This matrix captures the significance of terms across the topics.

NMF is easier to interpret as all the elements of the matrices W and H are now non-negative. So a higher score corresponds to greater relevance.

But how do we get matrices W and H?

NMF is a non-exact matrix factorization technique. This means that you cannot multiply W and H to get back the original document-term matrix V.

The matrices W and H are initialized randomly. And the algorithm is run iteratively until we find a W and H that minimize the cost function.

The cost function is the Frobenius norm of the matrix V - W.H, as shown below:

The Frobenius norm of a matrix A with m rows and n columns is given by the following equation:

7 Steps to Use SVD for Topic Modeling

1️⃣ To use SVD to get topics, let's first get a text corpus. The following code cell contains a piece of text on computer programming.

text=["Computer programming is the process of designing and building an executable computer program to accomplish a specific computing result or to perform a specific task.",

      "Programming involves tasks such as: analysis, generating algorithms, profiling algorithms' accuracy and resource consumption, and the implementation of algorithms in a chosen programming language (commonly referred to as coding).",

      "The source program is written in one or more languages that are intelligible to programmers, rather than machine code, which is directly executed by the central processing unit.",

      "The purpose of programming is to find a sequence of instructions that will automate the performance of a task (which can be as complex as an operating system) on a computer, often for solving a given problem.",

      "Proficient programming thus often requires expertise in several different subjects, including knowledge of the application domain, specialized algorithms, and formal logic.",

      "Tasks accompanying and related to programming include: testing, debugging, source code maintenance, implementation of build systems, and management of derived artifacts, such as the machine code of computer programs.",

      "These might be considered part of the programming process, but often the term software development is used for this larger process with the term programming, implementation, or coding reserved for the actual writing of code.",

      "Software engineering combines engineering techniques with software development practices.",

    "Reverse engineering is a related process used by designers, analysts and programmers to understand and re-create/re-implement"]

The text for which you need to find topics is now ready.

2️⃣ The next step is to import the TfidfVectorizer class from scikit-learn's feature extraction module for text data:

from sklearn.feature_extraction.text import TfidfVectorizer

You'll use the TfidfVectorizer class to get the DTM populated with the TF-IDF scores for the text corpus.

3️⃣ To use Truncated SVD (k-SVD) discussed earlier, you need to import the TruncatedSVD class from scikit-learn's decomposition module:

from sklearn.decomposition import TruncatedSVD

▶ Now that you've imported all the necessary modules, it's time to start your quest for topics in the text.

4️⃣ In this step, you'll instantiate a Tfidfvectorizer object. Let's call it vectorizer.

vectorizer = TfidfVectorizer(stop_words='english',smooth_idf=True) 
# under the hood - lowercasing,removing special chars,removing stop words
input_matrix = vectorizer.fit_transform(text).todense()

So far, you've:

☑ collected the text,
☑ imported the necessary modules, and
☑ obtained the input DTM.

Now you'll proceed with using SVD to obtain topics.

5️⃣ You'll now use the TruncatedSVD class that you imported in step 3️⃣.

svd_modeling= TruncatedSVD(n_components=4, algorithm='randomized', n_iter=100, random_state=122)
svd_modeling.fit(input_matrix)
components=svd_modeling.components_
vocab = vectorizer.get_feature_names()

6️⃣ Let’s write a function that gets the topics for us.

topic_word_list = []
def get_topics(components): 
  for i, comp in enumerate(components):
    terms_comp = zip(vocab,comp)
  sorted_terms = sorted(terms_comp, key= lambda x:x[1], reverse=True)[:7]
     topic=" "
     for t in sorted_terms:
      topic= topic + ' ' + t[0]
     topic_word_list.append(topic)
     print(topic_word_list)
  return topic_word_list
get_topics(components)

7️⃣ And it's time to view the topics, and see if they make sense. When you call the get_topics() function with the components obtained from SVD as the argument, you'll get a list of topics, and the top words in each of those topics.

Topic 1: 
  code programming process software term computer engineering

Topic 2: 
  engineering software development combines practices techniques used

Topic 3: 
  code machine source central directly executed intelligible

Topic 4: 
  computer specific task automate complex given instructions

And you have your topics in just 7 steps. Do the topics look good?

How to Visualize Topics as Word Clouds

In the previous section, you printed out the topics, and made sense of the topics using the top words in each topic.

Another popular visualization method for topics is the word cloud. In a word cloud, the terms in a particular topic are displayed in terms of their relative significance. The most important word has the largest font size, and so on.

!pip install wordcloud
from wordcloud import WordCloud
import matplotlib.pyplot as plt
for i in range(4):
  wc = WordCloud(width=1000, height=600, margin=3,  prefer_horizontal=0.7,scale=1,background_color='black', relative_scaling=0).generate(topic_word_list[i])
  plt.imshow(wc)
  plt.title(f"Topic{i+1}")
  plt.axis("off")
  plt.show()

The word clouds for topics 1 through 4 are shown in the image grid below:

Topic Clouds from SVD

As you can see, the font-size of words indicate their relative importance in a topic. These word clouds are also called topic clouds.

How to Use NMF for Topic Modeling

In this section, you'll run through the same steps as in SVD. You need to first import the NMF class from scikit-learn's decomposition module.

from sklearn.decomposition import NMF
NMF_model = NMF(n_components=4, random_state=1)
W = NMF_model.fit_transform(input_matrix)
H = NMF_model.components_

And then you may call the get_topics() function on the matrix H to get the topics.

Topic 1: 
  code machine source central directly executed intelligible

Topic 2: 
  engineering software process development used term combines

Topic 3: 
  algorithms programming application different domain expertise formal

Topic 4: 
  computer specific task programming automate complex given

Topic Clouds from NMF

For the given piece of text, you can see that both SVD and NMF give similar topic clouds.

SVD vs NMF – An Overview of the Differences

Now, let's put together the differences between these two matrix factorization techniques for topic modeling.

• SVD is an exact matrix factorization technique – you can reconstruct the input DTM from the resultant matrices.
• If you choose to use k-SVD, it's the best possible k-rank approximation to the input DTM.
• Though NMF is a non-exact approximation to the input DTM, it's known to capture more diverse topics than SVD.

Wrapping Up

I hope you enjoyed this tutorial. As a next step, you may spin up your own Colab notebook using the code cells from this tutorial. You only have to plug in the piece of text that you'd like to find topics for, and you'd have your topics and word clouds ready!

Thank you for reading, and happy coding!

References and Further Reading on Topic Modeling

• A Code-First Approach to Natural Language Processing by fast.ai
• Computational Linear Algebra by fast.ai

Cover Image: Photo by Brett Jordan on Unsplash