This guide will teach you how to create NLP projects from scratch. It includes details on how to organize your workflow, utilize the appropriate tools, and perform typical NLP tasks.

After reading this article, you will understand how to:

• Configure your environment for NLP development.

• Select the appropriate frameworks and libraries for your project.

• Execute fundamental NLP tasks such as sentiment analysis and text classification.

• Create and implement a functional NLP application.

• Diagnose and fix common problems in NLP projects.

Before beginning, you should have some basics at hand already. They include a solid understanding of Python programming and knowledge of the general ideas of machine learning. You should also know how to build algorithms and data structures. Finally, your system should have Python 3.8 or higher installed so you can try running the example snippets.

Table of Contents

• What is Natural Language Processing?

• How NLP Systems Interpret Speech

• Typical NLP tasks

• Conventional Machine Learning Methods for NLP

• How to Use NLP in Various Industries

• How to Choose the Most Effective NLP Tools and Libraries

• How to Prepare and Train NLP systems

• Establishing and Labeling Datasets

• Conclusion

What is Natural Language Processing?

NLP (natural language processing) is a set of methodologies that allow computers to learn to comprehend human language and produce relevant outputs.

NLP manages the intricacy of human communication. In contrast to conventional machine learning, which operates with structured data only, NLP handles unstructured text data.

Specifically, to more accurately comprehend language, NLP systems simultaneously analyze the syntax (which is the arrangement of words and grammar), the semantics (the meanings of specific words and phrases), and interpret context (how adjacent information affects meaning). This allows them to differentiate between various interpretations of identical words, grasp implied messages, and produce responses as relevant as possible.

The ability of machines to process language was demonstrated by early experiments such as the Georgetown-IBM translation in 1954 and the ELIZA chatbot in 1966 (Sources: Szmurlo and Akhtar, MDPI; Hutchins, ResearchGate). With today's tools, any developer can access and use the capabilities of NLP tools.

So why is this important for you? In 2025, the market for NLP, which currently powers chatbots, translation software, and content creation platforms, has reached $42.47 billion. (Source: Precedence Research)

The growth is only accelerating. By 2030, the global NLP market is expected to grow to $439.85 billion. (Source: GrandviewResearch).

Important NLP Concepts to Know

Five interconnected layers generally make up NLP systems. Every layer addresses a distinct language processing problem. (Source: Khatri and others, ResearchGate).

• Analysis of morphology is where you break down words into their most meaningful components by this layer. Words will be broken down into prefixes, roots, and suffixes. For instance, "working" becomes "work" plus "ing." This makes it easier for your system to comprehend word relationships even when they change form.

• Analysis of syntactic structure is where you use grammar rules to determine sentence structure. Here, you construct parse trees that map the grammatical relationships between words. Individual words are represented as leaves, phrases as intermediate nodes, and sentences as roots in the tree.

• Analysis of semantics is where, from the parsed structure, you derive the true meaning.

• You deal with synonyms, antonyms, and homophones as well as word ambiguity. This transforms grammatical structure into meaning.

• Analysis of discourse is where you connect sentences within longer text structures. You'll observe how ideas flow from one paragraph to the next and spot recurring themes. This connects meaning at the sentence level to meaning at the document level.

• Analysis of pragmatics is where you decipher intent and context. You will be able to resolve references, comprehend dialogue structure, and decipher implied meanings. You can process sarcasm, cultural background, and other aspects of everyday communication at this layer.

Understanding these layers gives you the ability to build NLP systems that can manage challenging language tasks in a variety of contexts.

How NLP Systems Interpret Speech

NLP systems use a pipeline to convert raw text into computational meaning. Each step builds on its predecessor, allowing for better analysis of unstructured language data. In this section, I’ll provide real snippets of code you can insert into an editor for training.

Step 1: Text Input

To start, your system will take in raw text that can come in various forms. Potential sources for raw input include emails, social media posts, articles, documents, or transcripts of speeches. The raw data will contain misspellings, crude language, and grammatical mistakes you'll need to circumvent.

Step 2: Text Preprocessing

Next, you’ll need to clean and standardize the input text before your system analyzes it. Your pre-process will likely include some or all of these steps:

• Tokenizing text into single words or subwords

• Removing punctuation marks from the text

• Lower casing all the text

• Removing stop words like "the", "and," and "is."

For example, you can accomplish such a simple form of NLP using Python, but note that you need to import specific libraries (we will discuss them later):

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

# Download required NLTK data
nltk.download('punkt')
nltk.download('stopwords')

# Raw text input
text = "The quick brown fox jumps over the lazy dog!"

# Tokenization
tokens = word_tokenize(text.lower())

# Remove punctuation and stop words
stop_words = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word.isalnum() and word not in stop_words]

print(filtered_tokens)
# Output: ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

Step 3: Syntactic Parsing and Analysis

After cleaning, you’ll analyze the text’s grammatical structure by constructing parse trees. While parse trees can vary in complexity, they map the relationships between words, phrases, and clauses. You can leverage part-of-speech tagging information to assign grammatical roles (noun, verb, adjective, and so on) to words, and dependency parsing to learn how related words are linked syntactically.

For example, the code below illustrates how to perform part-of-speech tagging with spaCy, which determines the grammatical function of each word within a sentence.

import spacy

# Load English language model
nlp = spacy.load("en_core_web_sm")

# Process text
doc = nlp("The cat sat on the mat")

# Part-of-speech tagging
for token in doc:
    print(f"{token.text}: {token.pos_}")

# Output:
# The: DET
# cat: NOUN
# sat: VERB
# on: ADP
# the: DET
# mat: NOUN

Step 4: Feature Engineering and Text Representation

Here, you convert words into numerical vectors that computers can parse using embedding or transformer-based techniques to capture similarities and semantic relationships between terms. For instance, this allows your system to understand that the words "kid" and "child" are similar in meaning.

from sentence_transformers import SentenceTransformer

# Load pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Convert sentences to embeddings
sentences = ["The cat sits on the mat", "The feline rests on the rug"]
embeddings = model.encode(sentences)

print(f"Embedding shape: {embeddings.shape}")
# Output: Embedding shape: (2, 384)

Step 5: Modeling and Pattern Recognition

In this part of the process, you’ll use machine learning algorithms to identify patterns from vectorized text. You may use either a traditional machine learning representation or one of the deep learning methods, such as transformers. Your models will learn about patterns in the language, classify the content presented, or extract entities in the text.

To understand this method, let’s see a straightforward example of a text classification model that uses transformers to identify sentiments.

from transformers import pipeline

# Load a pre-trained sentiment analysis model
classifier = pipeline("sentiment-analysis")

# Classify text sentiment
texts = ["I love this product!", "This is terrible and disappointing"]
results = classifier(texts)

for text, result in zip(texts, results):
    print(f"Text: {text}")
    print(f"Sentiment: {result['label']}, Confidence: {result['score']:.2f}\n")

# Output:
# Text: I love this product!
# Sentiment: POSITIVE, Confidence: 0.99
#
# Text: This is terrible and disappointing
# Sentiment: NEGATIVE, Confidence: 0.99

This illustrates how the model detects linguistic patterns for sentiment categorization, which is a typical task in NLP. In subsequent sections, we’ll delve into more specialized modeling techniques tailored for various NLP applications.

Step 6: Evaluation and Deployment

Next, you will evaluate your model from metrics such as precision, recall, and F1 scores. After evaluation, you will deploy your model to production, and the model will continue to learn from data produced from real-world text. Here’s an example of how it’s done:

from sklearn.metrics import classification_report

# Example predictions vs actual labels
y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 0, 0, 1]

# Generate evaluation metrics
print(classification_report(y_true, y_pred))

Typical NLP Tasks

Natural Language Understanding (NLU) tasks

Natural Language Understanding (NLU) tasks deal with actually understanding what people are communicating about. There are several elements involved in this process.

Sentiment analysis and text classification

Here, you recognize and categorize documents according to emotion. Your engine identifies whether the text conveys a positive, negative, or neutral sentiment. Then, it autonomously filters content across digital platforms. Consider this example:

from transformers import pipeline

# Load sentiment analysis pipeline
classifier = pipeline("sentiment-analysis")

# Analyze sentiment
result = classifier("I love this product! It works great.")
print(result)
# Output: [{'label': 'POSITIVE', 'score': 0.9998}]

Named Entity Recognition (NER)

NER is a pipeline that involves automatically identifying and classifying distinct pieces of information within a body of text. This includes names of people, locations, organizations, dates, and monetary figures.

Your NER system analyzes unstructured text to accurately label these entities, converting raw data into a structured format that can be easily analyzed. Your algorithm can also uncover relationships among these entities, allowing you to gain valuable insights from extensive amounts of text.

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple Inc. was founded by Steve Jobs in Cupertino, California.")

for ent in doc.ents:
     print(f"{ent.text}: {ent.label_}")

# Output:
# Apple Inc.: ORG
# Steve Jobs: PERSON
# Cupertino: GPE
# California: GPE

Question answering

You can create systems that consume natural language questions and retrieve appropriate answers. Your system can also use entailment and contradiction detection to analyze the logical relationships between text blocks.

Intent recognition

You can recognize user intentions in conversational domains. Your dialog systems are conscious of the user’s goals, allowing buttons or voices to respond in kind.

Now, let’s move on to some general natural language-related tasks.

General Natural Language Tasks

This class of tasks pulls together some aspects of understanding while dealing with generation as well.

Machine translation

You can translate text across multiple languages while preserving context and meaning. Neural networks use encoder-decoder architectures to create linguistic outputs in the target language.

Let’s see how it’s done with the MarianMTModel and MarianTokenizer models:

from transformers import MarianMTModel, MarianTokenizer

# Load translation model
model_name = 'Helsinki-NLP/opus-mt-en-es'
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)

# Translate English to German
text = "Hello, how are you?"
translated = model.generate(**tokenizer(text, return_tensors="pt", padding=True))
print(tokenizer.decode(translated[0], skip_special_tokens=True))
# Output: Hallo, wie geht's dir?

Text summarization

Often you’ll need to shorten a long document into a more accessible summary – this is text summarization, and it’s a common NLP task. Your system retains key details and coherence while reducing the length of a document.

Speech recognition and text-to-speech

Using these techniques, you can turn speech into text (speech recognition) or text into natural audio (text-to-speech). These tasks close the gap between text and audio modalities.

Syntactic parsing

Here, you examine the grammatical construction to determine the syntactic relationships between words in the sentence. This critical task gives a structural analysis of the text to support more complex understanding tasks.

These tasks, when combined, create powerful applications for different industries and use cases in Natural Language Processing.

Conventional Machine Learning Methods for NLP

Instead of relying on manually created linguistic rules (where programmers specify patterns like "if a word ends with '-ing', it is likely a verb" or "sentences containing 'not' followed by positive words suggest negative sentiment"), ML approaches apply statistical methods to discover patterns automatically within the data.

These methods learn through examples and don’t require human experts to define every potential language structure explicitly. As a result, they are more scalable and adaptable across different languages and fields. Let’s look at some of them now.

Logistic Regression

For tasks involving binary classification, you can use logistic regression. Based on input features, it predicts event probability by learning linear decision boundaries. Consider the following example:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Sample data
texts = ["This is spam", "Normal email", "Buy now!", "Meeting tomorrow"]
labels = [1, 0, 1, 0]  # 1 = spam, 0 = not spam

# Convert text to features
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(texts)

# Train model
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.25)
model = LogisticRegression()
model.fit(X_train, y_train)

# Predict
new_text = vectorizer.transform(["Free money now"])
prediction = model.predict(new_text)
print(f"Prediction: {'Spam' if prediction[0] == 1 else 'Not Spam'}")

Typical uses include toxicity classification, sentiment analysis, and spam detection.

Naive Bayes

Using the premise that words are independent, Naive Bayes applies Bayes' Theorem.

To classify documents, it computes:

$$P(label|text) = P(label) × P(text|label) / P(text)$$

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

# Training data
texts = ["I love this product", "Terrible service", "Amazing quality", "Waste of money"]
labels = [1, 0, 1, 0]  # 1 = positive, 0 = negative

# Vectorize text
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Train Naive Bayes
clf = MultinomialNB()
clf.fit(X, labels)

# Predict sentiment
new_review = vectorizer.transform(["Great purchase"])
print(f"Sentiment: {'Positive' if clf.predict(new_review)[0] == 1 else 'Negative'}")

Common uses for this algorithm that you can try are spam detection and bug detection in software.

Decision Trees

Decision trees partition data sets recursively by choosing the feature that maximizes information gain at each split in a way that builds interpretable, tree-like models. Each internal node is a decision (on a feature), each branch is an outcome of the decision, and each leaf node is a classification.

Decision trees are especially useful for text classification and feature selection because the decision tree allows you to trace exactly how the model made the predicted classification.

Let’s see a code example that shows how the decision tree learns which words, converted to TF-IDF features, predict whether the sentiment of the text in question is positive or negative:

from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split

# Sample text data with labels
texts = [
    "I love this movie, it's fantastic",
    "Terrible film, waste of time",
    "Amazing performance and great story",
    "Boring and disappointing",
    "Excellent cinematography and acting",
    "Awful, would not recommend"
]
labels = [1, 0, 1, 0, 1, 0]  # 1 = positive, 0 = negative

# Convert text to TF-IDF features
vectorizer = TfidfVectorizer(max_features=20)
X = vectorizer.fit_transform(texts)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, labels, test_size=0.3, random_state=42)

# Train decision tree
clf = DecisionTreeClassifier(max_depth=3, random_state=42)
clf.fit(X_train, y_train)

# Make predictions
test_text = ["This movie is wonderful"]
test_vector = vectorizer.transform(test_text)
prediction = clf.predict(test_vector)

print(f"Text: {test_text[0]}")
print(f"Predicted sentiment: {'Positive' if prediction[0] == 1 else 'Negative'}")
print(f"Model accuracy: {clf.score(X_test, y_test):.2f}")

At each node, the decision tree asks a question: "Does the text have a high TF-IDF score for 'wonderful'?" Then the tree will branch accordingly based on the answer to the question until reaching a classification.

One key parameter in the above code is max_depth=3 – without it, the tree may become too complex and overfit. The parameter limits the complexity of the tree.

Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) automatically determines thematic structures in large collections of texts by treating documents as probabilistic mixtures of topics, and topics as distributions over words. This discovery approach uses unsupervised learning, which means that no labeled training data are needed to discover structured but hidden themes. LDA is suited for exploratory text analysis and organization of data in significant amounts of text.

Let’s see some code that generates a word frequency matrix from documents. In this code, LDA identifies two underlying topics based on patterns of word co-occurrence, a process that is a type of clustering analysis for text documents.

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

# Document collection
documents = [
    "Machine learning algorithms process data",
    "Deep learning uses neural networks",
    "Python is great for data science",
    "Neural networks learn from examples"
]

# Create document-term matrix
vectorizer = CountVectorizer(max_features=50)
doc_term_matrix = vectorizer.fit_transform(documents)

# Train LDA model
lda = LatentDirichletAllocation(n_components=2, random_state=42)
lda.fit(doc_term_matrix)

# Display topics
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words_idx = topic.argsort()[-5:]
    top_words = [feature_names[i] for i in top_words_idx]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

In this illustration, we could interpret Topic 0 as "data science and algorithms," and Topic 1 as "neural networks and deep learning." The LDA model will assign, in a mixed model fashion, a probability distribution of each document falling under the two topics. For instance, a document titled "neural networks for data processing" could be considered 60% Topic 1 and 40% Topic 0.

Deep Learning Models

Deep learning models automatically extract hierarchical representations from raw text without manual feature engineering. Applying deep learning to language processing is important because language understanding requires modeling not just individual words, but also phrases, sentences, and the context as a whole.

A neural architecture achieves this modeling by learning multiple layers of abstraction and can interpret the sentences in more complex ways, such as sentiment, intent, or topic.

Let’s illustrate how it works with an example showing a simplified deep learning model that can be used for text classification using TensorFlow/Keras. This specific example uses an embedding layer to map words to dense vectors that capture their semantic meaning, as well as a Bidirectional LSTM layer which is able to capture information from the past and future of a sequence and outputs to a Dense layer for binary classification.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Example sentences and labels
texts = ["I like this movie", "I hate this movie"]
labels = [1, 0]  # 1 = positive, 0 = negative

# Tokenize text and pad sequences
tokenizer = Tokenizer(num_words=50)
tokenizer.fit_on_texts(texts)
X = pad_sequences(tokenizer.texts_to_sequences(texts), maxlen=5)

# Simple model: embedding + LSTM + output
model = Sequential([
    Embedding(input_dim=50, output_dim=8, input_length=5),
    LSTM(4),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy')
model.fit(X, labels, epochs=5, verbose=0)

# Predict sentiment for new sentence
test_text = ["I love this"]
test_seq = pad_sequences(tokenizer.texts_to_sequences(test_text), maxlen=5)
pred = model.predict(test_seq)[0][0]

print(f"Sentiment score: {pred:.2f} (1=positive, 0=negative)")

The model learns these patterns from example sentences that have been labeled as positive or negative and then uses those learned patterns to predict the sentiment of new text input. This is an example of how deep learning models learn to automatically represent the text that is processed and then use that representation to interpret sequences of text for classification purposes, without any feature engineering.

Convolutional Neural Networks (CNNs)

CNNs apply the same pattern-detecting framework to the text as they do to image recognition. CNNs see documents as sequences, and when a convolutional filter is applied across the text, it detects patterns for various types of features, such as n-grams (sequences of symbols that are adjacent to one another), and meaningful phrases.

CNNs encompass multi-filter layers to detect different features. Each filter layer detects features that are continuously more abstract, going from simple combinations of words to capturing combinations of words that are consistently used in semantic patterns, creating an effective use for the text classification task. (Source: Yoon Kim)

Here is an example of the convolutional layer scanning through the text using filters. It detects meaningful patterns established through previous learning, such as the word "excellent" or "terrible waste," learning to treat each combination of words as expressing a positive or negative sentiment during a final classification step.

import tensorflow as tf
from tensorflow.keras.layers import Embedding, Conv1D, GlobalMaxPooling1D, Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Sample training data
texts = [
    "This movie is excellent and entertaining",
    "Terrible film, complete waste",
    "Amazing story and great acting",
    "Boring and poorly made"
]
labels = [1, 0, 1, 0]  # 1 = positive, 0 = negative

# Tokenize and pad sequences
tokenizer = Tokenizer(num_words=100)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
X = pad_sequences(sequences, maxlen=10)

# Build CNN model
model = Sequential([
    Embedding(input_dim=100, output_dim=32, input_length=10),  # Convert words to dense vectors
    Conv1D(filters=64, kernel_size=3, activation='relu'),  # Detect 3-word patterns
    GlobalMaxPooling1D(),  # Extract most important features
    Dense(1, activation='sigmoid')  # Binary classification
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X, labels, epochs=10, verbose=0)

# Test prediction
test_text = ["wonderful movie with great plot"]
test_seq = tokenizer.texts_to_sequences(test_text)
test_pad = pad_sequences(test_seq, maxlen=10)
prediction = model.predict(test_pad)

print(f"Sentiment probability: {prediction[0][0]:.2f}")
print(f"Classification: {'Positive' if prediction[0][0] > 0.5 else 'Negative'}")

The pooling layer analyzes this filtered text and brings forth the most substantial signals for measuring positive versus negative sentiments from the convolutional text features of the previous steps.

Recurrent Neural Networks (RNNs)

RNNs handle sequential data by tracking hidden states that reflect dependencies over time. At each time step, the RNN receives the current word and the previous hidden state as input and changes the hidden state, which reflects the accumulated context.

Here's a concrete example where, as the RNN reads the next word from left to right, it updates its hidden state to maintain the context.

import tensorflow as tf
from tensorflow.keras.layers import Embedding, SimpleRNN, Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Training data
texts = [
    "I really enjoyed this book",
    "The plot was confusing and dull",
    "Fantastic read, highly recommend",
    "Disappointing and poorly written"
]
labels = [1, 0, 1, 0]

# Prepare data
tokenizer = Tokenizer(num_words=100)
tokenizer.fit_on_texts(texts)
sequences = tokenizer.texts_to_sequences(texts)
X = pad_sequences(sequences, maxlen=10)

# Build RNN model
model = Sequential([
    Embedding(input_dim=100, output_dim=32, input_length=10),
    SimpleRNN(units=64, return_sequences=False),  # Process sequence and maintain hidden state
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X, labels, epochs=20, verbose=0)

# Test
test_text = ["amazing story highly engaging"]
test_seq = tokenizer.texts_to_sequences(test_text)
test_pad = pad_sequences(test_seq, maxlen=10)
prediction = model.predict(test_pad)

print(f"Sentiment probability: {prediction[0][0]:.2f}")

Longer sentences are more complex because the information contained in the hidden state is lost over an ever-increasing number of time steps. That's the motivation for the more sophisticated architectures of long short-term memory (LSTM) and gated recurrent unit (GRU).

Encoder-Decoder Architectures

These architectures have two neural networks which work together. The first encoder neural network takes the input text and reduces it to a dense, fixed-size representation but encodes the essential meaning. Then a second decoder network generates an output text based on the meaning representation.

These architectures learn a compressed representation of the input data, and they are often used for:

• Dimensionality reductions.

• Feature learning.

• Document clustering.

• Sequence-to-sequence tasks (for example, translations or summarizations).

The following example illustrates how to use a Text-to-Text Transfer Transformer (T5) encoder-decoder model to translate English into German. The encoder takes the input English sentence and builds its internal representation of the text, while the decoder generates the German translation based on the representation:

from transformers import T5Tokenizer, T5ForConditionalGeneration

# Load T5 model for text generation
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

# Translate text
input_text = "translate English to German: Hello, how are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

# Generate translation
outputs = model.generate(input_ids)
translation = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Translation: {translation}")

This architecture solves the issue of variable-length input and output in a very elegant way. The encoding neural network reduces the sentence to a fixed-size representation regardless of the input length. Subsequently, the decoder generates an output for whatever length it determines is appropriate based on the input length, whether it’s one sentence or six sentences.

Transformer Models

Unlike RNNs, in which text is processed sequentially (one word at a time), transformers use a processing mechanism that evaluates the sequence in parallel. This means that the transformer can simultaneously consider all of the words in a sentence and directly compute relationships between any two words, even apart in distance.

In the example below, "The girl didn't go to school because she was ill," the model directly connects "she" with "girl" despite other words between these two. This brings a faster ability to train on information and helps avoid the degradation of information through time steps. (Source: Vaswani and others).

In the example, BERT, one of the most well-known transformer models, performs sentiment classification on a text. Here’s how the transformer justifies text classification by understanding pre-trained language and only using minimal additional training:

from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Load pre-trained BERT
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# Prepare input
text = "This movie was fantastic!"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

# Get predictions
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

print(f"Prediction scores: {predictions}")

In the above code, the tokenizer converts the sequence of text into numerical tokens (which BERT understands) and special tokens, such as [CLS] (for classification), at the beginning of the list of tokens. BERT then models the entire length of the sentence using multiple layers, where each layer is able to learn abstract representations of meaning in each layer.

How to Use NLP in Various Industries

You can use NLP to solve issues in almost any sector, and there are many sector-specific implementations. You can choose to try the snippets below depending on the area you’re most interested in.

Tourism and Hospitality

You can use NLP techniques to build intelligent booking systems that comprehend natural language requests from clients. Important uses you can apply:

• Sentiment analysis monitors consumer feedback to spot patterns in satisfaction and problems with customer service.

• NER-enabled chatbots retrieve dates and locations from consumer inquiries such as "I need a flight to Paris next Tuesday."

Here’s an example:

from transformers import pipeline

# Load NER model
ner = pipeline("ner", grouped_entities=True)

# Extract booking information
query = "I need a hotel in London from December 15 to December 20"
entities = ner(query)

for entity in entities:
    print(f"{entity['entity_group']}: {entity['word']}")
# Output: LOC: London

Through machine translation, you can provide multilingual support to your customers in various languages. And an intent classification model based on BERT automatically identifies how to route your customers for service or makes bookings automatically for them.

Logistics and Supply Chain

You can automate document processing via NLP and optimize delivery routing using predictive algorithms. Let’s see the common areas of application:

• You can use OCR to process documents to automatically extract shipping information from invoices and customs forms. Here’s an example:

import pytesseract
from PIL import Image

# Extract text from shipping document
image = Image.open('invoice.png')
text = pytesseract.image_to_string(image)

# Parse extracted information
# (Add parsing logic based on document structure)

• Text classification can place shipments into categories based on descriptions, allowing for recursive sorting of shipments for transport.

• Predictive routing models can use historical delivery data and weather reports to create delivery schedules.

• Natural Language Generation takes technical data across logistics to create user-friendly tracking updates.

Retail and eCommerce

Within the eCommerse operations, you can personalize your customers’ shopping experience and optimize pricing with NLP techniques.

Some key applications that you can benefit from:

• Recommendation engines utilize word embeddings to learn product descriptions and corresponding user reviews to suggest relevant items. Here’s how, for instance:

from sentence_transformers import SentenceTransformer, util

# Load embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Product descriptions
products = [
    "Wireless Bluetooth headphones with noise cancellation",
    "USB-C charging cable for smartphones",
    "Noise-cancelling earbuds with long battery life"
]

# User query
query = "I need headphones that block outside noise"

# Calculate similarities
query_embedding = model.encode(query)
product_embeddings = model.encode(products)
similarities = util.cos_sim(query_embedding, product_embeddings)

# Find best match
best_match_idx = similarities.argmax()
print(f"Recommended product: {products[best_match_idx]}")

• Chatbots that include dialogue management can respond to inquiries from customers about products, orders, and returns.

• Sentiment analysis on social media tracks brand health and customer sentiment in real-time.

• Price optimization algorithms analyze competitors' pricing and market signals to change prices in real-time.

• Demand forecasting analyzes news and social sentiment to predict inventory needs.

Healthcare

Healthcare, with the great amount of data from patient records, is a natural area for NLP to optimize. You can support clinical decision-making and process medical records using specialized NLP systems.

Here are a few of the possible uses and an example:

• Clinical NER identifies conditions, medications, and treatments mentioned in the clinicians' notes within electronic health records. For instance:

import spacy

# Load medical NER model (requires installation of scispacy)
# pip install scispacy
# pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/en_core_sci_sm-0.5.1.tar.gz

nlp = spacy.load("en_core_sci_sm")

# Process clinical note
text = "Patient presents with hypertension and type 2 diabetes. Prescribed metformin 500mg."
doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text}: {ent.label_}")

• Clinical decision support systems scan descriptions of symptoms and provide suggestions for potential diagnoses to help a physician's decision-making.

• Literature mining scans clinical studies and identifies new treatment patterns or potential drug discovery targets.

Of course, NLP can also be used in patient assistance chatbots, as they can comprehend natural language and its nuances.

Financial Services

In the finance sector, there are unique bottlenecks you might face. Financial data security gaps and the risks of fraud are among the most threatening ones, as well as the regulatory fines that come with these issues.

With NLP, you can improve security mechanisms and create systems for detecting fraud.

You can also detect phishing attacks with high accuracy with ML classifiers and NLP using CNNs and RNNs combined. (Source: Saidat and others, ResearchGate).

Some other use cases include:

• Document analysis processes loans applications/contract to assess credit risk by automated analysis of documents.

• Fraud detection systems analyze transaction data and communication dat to identify suspicious activity. For example:

from transformers import pipeline

# Load zero-shot classification model
classifier = pipeline("zero-shot-classification")

# Analyze transaction description
description = "Wire transfer to offshore account for investment opportunity"
candidate_labels = ["legitimate transaction", "potential fraud", "suspicious activity"]

result = classifier(description, candidate_labels)
print(f"Classification: {result['labels'][0]} (Score: {result['scores'][0]:.4f})")

• Automated compliance monitoring scans messages for adherence with regulations.

• Robo-advisors leverage natural language interfaces to engage with clients while providing investment advice.

Apart from these uses, conventional chatbots also provide assistance by using NLP techniques. OCR algorithms are widely used for document analysis – but we’ve mentioned those other use cases, so we won’t discuss them further here.

Legal Industry and Compliance Regulations

Even more than financial services, the legal sector depends on strict requirements, laws, and regulations. NLP techniques can help you improve safety, security, and efficiency in processing legal documents.

Key examples of how it can be applied:

• Multimodal authentication is a secure identity verification process consisting of a combination of facial recognition, voice recognition, and natural language processing.

• Speaker recognition uses automatic speech-to-text encoding and intent recognition to process a verbal response to security questions.

• Contract analysis scans legal documents to identify key terms, deliverables, and dates, extracting that information automatically. As an example, you can try the following snippet with the spaCy library installed:

import spacy

nlp = spacy.load("en_core_web_sm")

# Extract dates and obligations from contract
contract_text = "The agreement shall commence on January 1, 2026 and continue for a period of 12 months."
doc = nlp(contract_text)

for ent in doc.ents:
    if ent.label_ in ["DATE", "CARDINAL"]:
        print(f"{ent.label_}: {ent.text}")

• Compliance monitoring looks for possible regulatory infractions in legal communications.

These real-world examples show how NLP can be used to solve practical business issues and boost operational effectiveness. You can modify the samples to your case or discover other potential uses, but these are the most widespread ones for you to try.

How to Choose the Most Effective NLP Tools and Libraries

There is a great variety of tools and libraries that can help you learn how to use NLP or that you can use to implement NLP into a project. You should select the appropriate tools considering your project needs and background in the associated technologies.

Below are some popular tools you can choose to learn or check out, along with tips about when they’re most useful.

Hugging Face Transformers

Hugging Face Transformers has thousands of pre-trained models for text generation, classification, and question answering. It gives you more than 100 languages supported and is compatible with PyTorch and TensorFlow.

It also provides model hosting and datasets, and allows collaboration with community members. It will assist you with deep learning NLP applications that require top-notch software to implement the algorithms.

NLTK (Natural Language Toolkit)

NLTK is the primary package in Python for education and research concerning NLP. Developed at the University of Pennsylvania, it provides extensive packages for your tokenizers, stemmers, parsers, and semantic reasoning. It’s a great choice if you need to get learning concepts in NLP or conduct research projects.

spaCy

spaCy is a Python library developed to be production-ready, and has the fastest syntactic parser. It’s constructed using Cython for optimal performance and offers excellent named entity recognition. It will fit well if you need strong dependency parsing and a developer-friendly API. It’s easy for you to use spaCy for quick prototyping.

Google Cloud NLP

Google Cloud NLP also offers enterprise-level API services. It will fit your project if you need sentiment analysis, entity recognition, syntax analysis, automatic language identification, and simple, trouble-free scaling. And if you’re already in the Google Cloud ecosystem working with big volumes of customer feedback, it’s just what you need.

Amazon Comprehend

Comprehend is a fully-managed service from AWS for text analysis in the cloud. It supports the major functions you might want to cover: sentiment analysis, entity recognition, topic modeling, built-in protection of personally identifiable information (PII), and auto-scaling. And it’s perfect if you need a built-in integration with the AWS suite.

IBM Watson

Watson has NLP models specific to regulated industries (healthcare, finance, and so on). Its library offers pre-trained models in 20 programming languages. Its top features you can use are strong data controls, reliable REST API access, and truly compliance-ready outputs. These makes this tool a great choice if you’re in healthcare, finance, or legal industries.

TextBlob

TextBlob is a simplified library that’s a great option if you’re a beginner. It’s a user-friendly Python library for common NLP tasks. For your convenience, it offers a simplified API design, but still provides decent sentiment analysis, translation, spelling correction, and noun phrase extraction. Apart from beginner projects, it will fit your quick prototypes creation needs.

How to Prepare and Train NLP Systems

As you get ready to train your NLP model, you’ll need to prepare your data accurately to ensure its quality doesn’t hinder the outputs. Remember that poor quality data results in a poor performing model, so you’ll want to make sure you have solid data.

Understanding Data Quality and Preprocessing

Raw text data is messy and unstructured. It contains typos, slang, and irrelevant information that degrades the performance of your model.

Preprocessing is the operation that takes messy data and converts it into clean, structured text that models can accept as input.

Research shows that 85.4% of NLP research studies utilized some sort of restructuring/preprocessing to allow NLP models to process raw text. The key data quality components that were essential included accuracy (68.3%), relevance (34.1%) and comparability (31.7%). (Source: Nesca and others, NCBI).

Preprocessing comes down to a specific list of tasks you’ll need to perform. Let’s break them down.

Text Cleaning

Text cleaning is the process of standardizing the text format by removing anything that may affect model training. Raw text often contains extra elements (HTML tags, URLs, special characters, inconsistent use of capitalization, and excess whitespace) that add noise to your data.

The following example shows a cleaning pipeline that removes the above-mentioned elements. This function performs multiple steps of text cleaning:

import re

def clean_text(text):
    # Convert to lowercase
    text = text.lower()

    # Remove HTML tags
    text = re.sub(r'<[^>]+>', '', text)    

    # Remove URLs
    text = re.sub(r'http\S+|www.\S+', '', text)    

    # Remove special characters and numbers
    text = re.sub(r'[^a-zA-Z\s]', '', text)    

    # Remove extra whitespace
    text = ' '.join(text.split())    

    return text

# Example
raw_text = "Check out https://example.com! It's <b>AMAZING</b> :-)"
cleaned = clean_text(raw_text)
print(cleaned)
# Output: check out its amazing

The first step the model made was to convert everything to lower case for uniformity. Then it used regular expressions to parse the text to remove HTML tags, URLs, special characters, and numbers. Finally, it normalized whitespace by splitting and joining the text back together. The output is clean text, in a standardized format you can use for tokenization.

Tokenization

The next step is to divide the text into smaller digestible chunks that are easier for ML models to understand. These chunks are known as tokens.

Tokenization comes in three varieties:

• Word tokenization separates text according to punctuation and whitespace.

• Sentence tokenization uses punctuation cues to divide text into sentences.

• Subword tokenization breaks words up into more manageable chunks.

The example below addresses examples of word and sentence tokenizing by using NLTK (Natural Language Toolkit).

from nltk.tokenize import word_tokenize, sent_tokenize

text = "Natural language processing is exciting! It helps computers understand text."

# Word tokenization
words = word_tokenize(text)
print(f"Words: {words}")

# Sentence tokenization
sentences = sent_tokenize(text)
print(f"Sentences: {sentences}")

Notice that after word tokenization was performed, the punctuation marks '!' or '.' were considered individual tokens, as punctuation conveys meaning. Sentence tokenization correctly identified the boundaries of the two sentences, and despite the presence of an exclamation mark, it indicated that it had more complex rules beyond just splitting based on periods.

Stop Word Removal

Here, you reduce the text to the meaning without any extra details. You can do this by removing the commonly used words that have little semantic value – the “stop words”.

Common stop words include articles, prepositions, pronouns, auxiliary verbs and conjunctions. Here’s how to do it:

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download('stopwords')

text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text.lower())

stop_words = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word not in stop_words]

print(f"Original: {tokens}")
print(f"Filtered: {filtered_tokens}")
# Output: ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog']

Stemming and Lemmatization

In this next step, you’ll process the text further by reducing words to their root form. This will treat words with similar variations as a single token.

• To be precise, stemming is simply using heuristic rules that remove the endings of words. For example (running|runs|run) → run.

• Lemmatization uses morphological analysis and vocabulary. For example, (children|mice) → child|mouse.

Lemmatization typically gives better, more accurate outcomes (but calls for more computation at the same time).

Here’s how you can apply both:

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

words = ["running", "runs", "ran", "children", "better"]

print("Stemming:")
for word in words:
    print(f"{word} -> {stemmer.stem(word)}")

print("\nLemmatization:")
for word in words:
    print(f"{word} -> {lemmatizer.lemmatize(word, pos='v')}")

Expanding Contractions

AI systems need standardization. And “don’ts” and “you’res” are unacceptable for them. This is why, typically, you would want to expand the contraction to the real word for standardization. Here’s how you can do that:

import contractions

text = "I can't believe it's already 2025. We'll see what happens."
expanded = contractions.fix(text)
print(expanded)
# Output: I cannot believe it is already 2025. We will see what happens.

Correcting Spelling Errors

Orthographic, spelling, or grammar errors shouldn’t be in the data that you feed to your ML models. You can make corrections to these errors using statistical language models, which can predict the most likely intended word, edit distance algorithms that can find the closest valid word, or neural approaches that can learn patterns of common errors.

For example, let’s see how TextBlob, a library that uses a mix of a dictionary-based approach and contextual probability, detects and corrects misspellings.

from textblob import TextBlob

text = "Natral languag procesing is powrful"
corrected = TextBlob(text).correct()
print(f"Original: {text}")
print(f"Corrected: {corrected}")

TextBlob analyzes each word, identifies which ones are not in its dictionary, calculates edit distances, finds the most similar valid words, and selects corrections based on the frequency of word use in context.

Parts-of-Speech Tagging

Parts-of-speech tagging (POS) refers to assigning grammatical classification to words based on their role in a sentence. This is important because the same word can function as a different part of speech depending on the context. For example, "walk" can be a noun (like "an evening walk") or a verb (like "I walk to the office").

POS taggers rely on statistical models trained to predict the most likely grammatical role for a word given the context. The following code shows POS tagging using NLTK, which applies a pre-trained model that will tag grammatical structure.

import nltk

text = "The cat sat on the mat"
tokens = nltk.word_tokenize(text)
pos_tags = nltk.pos_tag(tokens)

for word, tag in pos_tags:
    print(f"{word}: {tag}")
# Output:
# The: DT (Determiner)
# cat: NN (Noun)
# sat: VBD (Verb, past tense)

The function pos_tag() assesses each token and assigns it a standardized notation. For example, DT indicates determiners (such as "the"), NN indicates singular nouns, VBD indicates past tense verbs, and IN indicates prepositions. The tagger can also use context to make these decisions: it can determine that "sat" is VBD and not NN because it appears after a noun and before a preposition, all of which are typical patterns of the English sentence.

Establishing and Labeling Datasets

For supervised learning tasks such as sentiment analysis, NER, or classification, unlabeled data is useless.

This is why, to create training datasets, you must annotate raw data with relevant labels. Models can learn patterns and make predictions thanks to this "ground truth." Let’s define the most common methods for labeling.

Automated Labeling Based on Libraries

You don’t always have to create labels from scratch. Libraries like TextBlob have built-in sentiment analysis models trained on large datasets that label text. They run a polarity score (a number that represents sentiment) and assign a categorical label.

For example, TextBlob sees a word choice, modifiers in particular (words like "very" or "not"), and grammatical patterns to run a polarity score from -1 (most negative) to +1 (most positive), with zero meaning a neutral sentiment.

In this example, we’re automatically labeling sentiments based on a pre-trained TextBlob sentiment analyzer:

from textblob import TextBlob

def label_sentiment(text):
    blob = TextBlob(text)
    polarity = blob.sentiment.polarity

    if polarity < 0:
        return "negative"
    elif polarity == 0:
        return "neutral"
    else:
        return "positive"

# Example
texts = [
    "I love this product!",
    "It's okay, nothing special",
    "Terrible experience, very disappointed"
]

for text in texts:
    label = label_sentiment(text)
    print(f"{text} -> {label}")

Manual Labeling

In many cases, automated library-based labeling isn’t an option. For domain-specific standards, you should annotate your data by hand to ensure accuracy and relevance.

For projects involving manual labeling:

• Establish precise labeling standards. Provide an annotation guideline that defines each label with clear criteria and edge cases. As an example, if annotating for customer support tickets, an example of potential criteria should be that "I need help resetting my password" is "Technical support," and another example is "When will my order arrive?" which is an example of "Order inquiry."

• For quality control, use several annotators. Have 2-3 people label the same data samples independently. For example, if annotating for medical symptoms, having multiple annotators better reduces the chance of bias from one person's labeling and may protect against clerical errors.

• Determine the inter-annotator agreement. Calculate Cohen's kappa or Fleiss' kappa scores to measure the consistency of agreement among annotators. A score of above 0.80 would signify very good agreement, while a score below 0.60 would indicate that the labeling guidelines were not clear enough to the annotators.

• Give instructions and illustrations. Create a reference document with 20-30 examples you pre-labeled, showing examples of typical use cases and edge cases. For example, in a sentiment analysis, case you can provide examples of when a sentiment would be neutral, with an example such as "This product is fine, I guess," even though it may have seemed slightly negative. A sentiment is also classified as a good positive example even though it contains two negatives: "Not bad at all.”

This is the gold standard for high-quality datasets, but it’s also time-consuming and expensive – labeling 10,000 customer reviews might take a week if done manually.

Approaches with Semi-Supervision

There are instances where you should combine the previous approaches. This semi-supervised method uses a small, manually labeled data set (high-quality data) and a large pool of unlabeled data (cheap, large amounts of data).

The method operates through iterative self-training, where you first train the model on your small dataset of signed data, then predict the labels on the unlabeled data using this model, then add the most confident predictions to your training data during training, and retrain. The self-training process is then repeated, improving and expanding your labeled data set gradually.

Here is an example of self-training in practice: this code demonstrates the semi-supervised workflow.

from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.svm import SVC

# Small labeled dataset + large unlabeled dataset
X_labeled = [[1, 2], [3, 4], [5, 6]]
y_labeled = [0, 1, 0]
X_unlabeled = [[2, 3], [4, 5], [6, 7]]

# Combine datasets (-1 represents unlabeled)
X_train = X_labeled + X_unlabeled
y_train = y_labeled + [-1, -1, -1]

# Self-training classifier
base_classifier = SVC(probability=True, gamma='auto')
self_training = SelfTrainingClassifier(base_classifier)
self_training.fit(X_train, y_train)

The code shows a version of the SelfTrainingClassifier that first trains on the three labeled examples, then uses the model to predict inputs and labels for the unlabeled data. The classifier then selects predictions where it has high confidence (for example, predictions that are >90% probability) while using them as newly signed data. The classifier then re-trains itself, and the process continues.

So how do you decide which approach will fit your needs? In most cases, the optimal one will depend on the following aspects:

• Available budget and time.

• Desired accuracy.

• Size of the dataset.

• Complexity of the domain.

As you see, approaches vary and can be mixed now and then. The key thing is to make sure the inputs for final pre-generation processing are cleaned, standardized, and labeled. Remember the key principle: “garbage in, garbage out”. Send gold instead, and good luck!

Conclusion

At this point, you should know the basics of working with NLP projects.

Throughout this article, you've learned:

• How to set up your NLP development environment using a set of tools and libraries.

• The five parts of NLP systems and how they are used to process language.

• How to conduct common tasks like NER, sentiment analysis, and text classification.

• How to choose the library to use to accommodate your project needs.

• How to prepare and label datasets for training.

• How to find the key practical applications of NLP tailored to your industry and use case.

Next steps

• Try to start with a simple project, like sentiment analysis, that uses pre-trained models.

• Practice preprocessing methods with your own text data.

• Use and try different libraries to see how to get the best output for your project.

• Build a full pipeline from preparing text data to deploying models.

• Continue to practice and see advanced applications like transformer models and fine-tuning.

Most importantly, keep in mind that NLP is an iterative process. Start small, test appropriately to get it to work, and then build in complexity when you are more comfortable and sure of your abilities and familiarity with the practices.

About the author

Hope you enjoyed the article and found it helpful. I’ve been a contributor to freeCodeCamp for more than 8 years, and to make this piece more precise and detailed, I used some expert help.

I’m grateful for the technical ideas of my co-workers at COAX Software who wished to stay anonymous. The company is a well-regarded AI/ML development company.

To find out more about me and read more content on tech and digital, you can visit my LinkedIn page.

Whether you build chatbots, recommendation systems, or search engines, understanding how close two sentences are in meaning can improve user experience – and this is what sentence similarity allows you to do.

Sentence Transformers make this process simple and efficient. In this guide, you will learn what sentence similarity is, how Sentence Transformers work, and how to write code to measure similarity between two sets of sentences.

Table of Contents

• What Is Sentence Similarity?

• Why Use Sentence Transformers

• Loading a Pre-trained Model

• Defining Sentences to Compare

• Converting Sentences into Embeddings

• Calculating Cosine Similarity

• Printing the Results

• Sample Output

• How to Interpret the Scores

• Real-World Applications of Sentence Similarity

  • Semantic Search

  • Duplicate Detection

  • Recommendation Systems

  • Chatbots and Virtual Assistants

  • Improving Performance with Larger Models

• Conclusion

What Is Sentence Similarity?

Sentence similarity is the process of comparing two sentences to see how close they are in meaning. It does not look at the exact words but focuses on the meaning behind them.

For example:

• “The cat is sitting outside”

• “The dog is playing in the garden”

Both sentences talk about animals outdoors, so they share some similarity even though they use different words.

This kind of understanding is essential for tasks like document clustering, duplicate detection, or semantic search.

Why Use Sentence Transformers

Traditional methods like Bag of Words relied on simple word matching or frequency counts. But these fail when words differ but the meaning stays the same.

Sentence Transformers solve this by using transformer-based language models like BERT or RoBERTa to create embeddings.

An embedding is a list of numbers that represents the meaning of a sentence. When two embeddings are close together in this high-dimensional space, their sentences are similar in meaning.

The Sentence Transformers library in Python makes this easy by providing pre-trained models that can generate embeddings for sentences.

Installing the Required Libraries

Before you start coding, make sure you install the required packages. Run this command to do so:

pip install -U sentence-transformers

This will install the Sentence Transformers library along with its dependencies.

Loading a Pre-trained Model

Sentence Transformers offers several pre-trained models. For this example, you will use the all-MiniLM-L6-v2 model. It’s lightweight, fast, and works well for most applications.

Here is how to load it in Python:

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("all-MiniLM-L6-v2")

Once loaded, this model can convert any sentence into its corresponding embedding.

Defining Sentences to Compare

You need two lists of sentences for comparison. Here is an example:

sentences1 = [
    'The cat sits outside',
    'A man is playing guitar',
    'The movies are awesome'
]

sentences2 = [
    'The dog plays in the garden',
    'A woman watches TV',
    'The new movie is so great'
]

Each sentence in sentences1 will be compared with the sentence at the same position in sentences2.

Converting Sentences into Embeddings

Now that you have sentences, you must convert them into embeddings using the model.

Add this code:

# Convert sentences to embeddings
embeddings1 = model.encode(sentences1, convert_to_tensor=True)
embeddings2 = model.encode(sentences2, convert_to_tensor=True)

The convert_to_tensor=True argument tells the model to return PyTorch tensors, which work well with similarity calculations.

Calculating Cosine Similarity

Once you have embeddings, you need a way to measure similarity. The cosine similarity metric is commonly used for this.

Cosine similarity looks at the angle between two vectors in a high-dimensional space. If the angle is small, the vectors are similar.

Add this code to compute similarity:

from sentence_transformers import util
# Compute cosine similarity
cosine_scores = util.cos_sim(embeddings1, embeddings2)

Now cosine_scores contains the similarity score for each sentence pair.

Printing the Results

To see the results clearly, format them like this:

# Print formatted results
for i in range(len(sentences1)):
    print(f"{sentences1[i]} \t\t {sentences2[i]} \t\t Score: {cosine_scores[i][i]:.4f}")

This will print each sentence pair along with its similarity score.

Sample Output

If you run this code, you will see a result similar to the below.

The third pair has the highest score because both sentences talk about movies in a positive way.

How to Interpret the Scores

The cosine similarity score ranges between -1 and 1.

• A score close to 1 means the sentences are very similar.

• A score near 0 means they are unrelated.

• Negative values mean the sentences are not related or even opposite in meaning.

In most real-world cases, you focus on values between 0 and 1. The higher the value, the closer the meanings.

Real-World Applications of Sentence Similarity

Sentence similarity has become a core part of many modern applications because it helps systems understand meaning rather than relying on exact words. This shift makes search, analysis, and recommendations far more accurate and useful.

Semantic Search

Traditional search engines depend on keyword matches. If the exact words are missing, results often become irrelevant. Semantic search solves this problem by looking at the meaning behind a query.

For example, if someone searches for “best ways to learn guitar,” the system can return results for “top tips to play the guitar” even though the keywords differ. This makes search experiences smoother and more intelligent.

Duplicate Detection

Large datasets often contain repeated or near-duplicate content. Manual checking is impossible when dealing with millions of records.

Sentence similarity automates this by detecting texts that carry the same meaning even if the wording changes slightly. This is especially useful in data cleaning, web scraping pipelines, or managing user-generated content.

Recommendation Systems

Recommendation engines work best when they understand context. For instance, if a user likes articles about “healthy cooking,” the system can recommend content on “nutritious recipes” or “quick healthy meals” using similarity scores. This approach goes beyond surface-level keywords and finds deeper connections in the text.

Chatbots and Virtual Assistants

Chatbots store a large set of possible user questions and answers. When someone types a new question, the system must find the most relevant response. By using sentence similarity, chatbots match user input with the closest existing query in meaning, not just words, leading to more accurate and natural conversations.

Improving Performance with Larger Models

The all-MiniLM-L6-v2 model is fast and accurate for small to medium tasks.

For more accuracy, you can try larger models like all-mpnet-base-v2, though they may require more memory and time to run.

Replace the model name in your code to use a different pre-trained model:

model = SentenceTransformer("all-mpnet-base-v2")

Conclusion

Sentence Transformers make it easy to measure sentence similarity using pre-trained models. By converting sentences into embeddings and comparing them with cosine similarity, you can build systems that understand meaning rather than relying on simple word matching.

With just a few lines of code, you can integrate this into chatbots, search engines, or recommendation systems and create more intelligent applications.

Hope you enjoyed this article. Signup for my free newsletter TuringTalks.ai for more hands-on tutorials on AI. You can also visit my website.

In the fast-evolving world of natural language processing (NLP), tokenizers play a pivotal role.

Tokenizers are the unsung heroes behind the scenes, making sense of human language for machines to understand.

Let’s dive into what tokenizers are and explore their use cases. We'll also introduce you to Huggingface, a leading platform in AI and NLP.

We'll also walk through a simple code example using the Huggingface Tokenizer library.

What are Tokenizers?

Imagine that you’re trying to teach a robot to understand and speak human languages. The first challenge you’d face is how to break down language into pieces the robot can digest. That’s where tokenizers come in.

Tokenizers dissect complex language into manageable pieces, transforming raw text into a structured form that AI models can easily process. This seemingly simple step is crucial, enabling machines to grasp the nuances of human communication.

Think of tokenizers as the chefs who chop ingredients before a meal is cooked. Without this step, preparing complex dishes (or understanding complex sentences) would be much harder.

Through tokenization, AI systems can recognize patterns, understand context, and generate responses that are increasingly similar to human interaction.

By breaking down the complexities of language into digestible bits, tokenizers not only enhance AI’s linguistic capabilities but also pave the way for more intuitive, efficient, and accurate machine learning models.

What are Huggingface Tokenizers?

Huggingface is a company at the forefront of AI and NLP.

They are best known for their Transformers library, which has made it easy to access state-of-the-art NLP models.

At the heart of their innovations is the tokenizers library, a powerful tool designed to convert text into a format that AI models can understand. This library is essential for developers and researchers working on AI projects.

Hugging Face’s tokenizers are not only efficient and fast but also support a wide range of languages, making them versatile tools for global NLP tasks. They are optimized for performance, ensuring that they can handle large volumes of text without compromising speed or accuracy.

What sets Hugging Face’s tokenizers apart is their integration with the Transformers library, another cornerstone of Hugging Face’s AI ecosystem.

This integration allows for seamless processing of text data, readying it for complex tasks like translation, summarization, and sentiment analysis.

The tokenizers library is continually updated, incorporating the latest research findings and community feedback to enhance its capabilities.

Simple Code Example of Huggingface Tokenizer Library

Let’s get our hands dirty with some code. We’ll use the Huggingface Tokenizer library to tokenize a simple sentence.

First, let's install the Huggingface Transformers library. (Use ! before the command if you are installing it in a Google Collab notebook).

pip install transformers

First, let's import the AutoTokenizer class from the Transformers library. AutoTokenizer is a factory class that can automatically load the tokenizer corresponding to a pre-trained model we specify (in this case, the bert-base-uncased model).

from transformers import AutoTokenizer

Next, we create an instance of the AutoTokenizer class by calling the from_pretrained method. This tokenizer is designed to work with the BERT model and is configured to not differentiate between uppercase and lowercase letters (hence 'uncased').

tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")

Now let’s declare a string for tokenizing.

text = "Hello, and welcome to the world of Tokenizers"

Let’s use the tokenize method of the tokenizer with the sample text as its argument.

tokens = tokenizer.tokenize(text)

The tokenizemethod splits the input text into a list of tokens or words/sub-words that the pre-trained model was trained on. For models like BERT, words might be split into smaller units (sub-words or characters) to handle out-of-vocabulary words more effectively.

We'll also convert the list of tokens into a list of integers (token IDs). Each integer corresponds to a specific token in the tokenizer’s vocabulary.

This conversion is necessary because machine learning models do not understand text directly; they work with numerical data.

token_ids = tokenizer.convert_tokens_to_ids(tokens)

We are done. let’s print both tokens and their corresponding IDs.

print("Tokens:", tokens)
print("Token IDs:", token_ids)

So this piece of code loads a pre-trained tokenizer for the BERT model, tokenizes a sample sentence and converts those tokens into their corresponding IDs. These IDs are what machine learning models process.

Here is the response:

Tokens: ['hello', ',', 'and', 'welcome', 'to', 'the', 'world', 'of', 'token', '##izer', '##s']
Token IDs: [7592, 1010, 1998, 6160, 2000, 1996, 2088, 1997, 19204, 17629, 2015]

These tokens and token IDs are crucial for training machine learning models. They convert text into a numerical format that models can process, enabling the understanding of language nuances.

Tokens like ##izer and ##s are examples of how the tokenizer deals with words or parts of words that might not be in its basic vocabulary.

The ## prefix indicates that these are sub-word units or suffixes attached to the preceding token without a space. This allows the model to handle a wide range of vocabulary, including new or uncommon words, by breaking them down into known subcomponents.

Conclusion

Tokenizers are foundational to NLP, and the Huggingface Transformers library provides an extensive toolkit for working with them.

By understanding and utilizing tokenizers, we can bridge the gap between human language and machine understanding, unlocking a wide range of applications in AI.

Whether you’re a seasoned developer or new to NLP, diving into tokenization methods is a great way to enhance your machine-learning skills.

Hope you enjoyed this article. If you have any questions, let me know in the comments. Visit turingtalks.ai for weekly byte-sized AI tutorials.

Have you ever wondered how modern AI achieves such remarkable feats, like understanding human language or generating text that sounds like it was written by a person?

A significant part of this magic stems from a groundbreaking model called the Transformer. Many frameworks released into the Natural Language Processing (NLP) space are based on the Transformer model, and an important one is the Hugging Face Transformer Library.

In this article, I’ll walk you through why this library is not just another piece of software, but a powerful tool for engineers and researchers alike. Then you'll see a practical example of how to use it.

What is the Hugging Face Transformer Library?

The Hugging Face Transformer Library is an open-source library that provides a vast array of pre-trained models primarily focused on NLP. It’s built on PyTorch and TensorFlow, making it incredibly versatile and powerful.

One of the first reasons the Hugging Face library stands out is its remarkable user-friendliness. Even if you’re not a deep learning expert, you can use this library with relative ease.

It offers straightforward interfaces that allow you to implement complex models with just a few lines of code. This simplicity opens the doors of advanced AI to a broader range of developers and researchers.

Pre-Trained and Ready to Go

The beauty of today’s deep learning models is that you don't have to train a model from scratch. Most models are pre-trained and your job as an AI engineer will be to train a model using custom data.

So imagine having access to a toolbox where each tool is tailored for a specific job. That’s what Hugging Face offers with its wide range of pre-trained models.

Whether you’re working on text classification, question answering, or language generation, there’s a model ready for you to use. This saves an enormous amount of time and resources as you don’t have to start from scratch.

While pre-trained models are fantastic, they might not fit every specific need. This is where Hugging Face truly shines. The library allows you to fine-tune models on your dataset, making it possible to customize the models to your specific requirements.

Community Support

What sets Hugging Face apart is not just its technical capabilities but also its vibrant community. By engaging with this community, you gain access to a wealth of knowledge and support.

Users continuously contribute to the library, adding new models and features, making it a living, evolving ecosystem. This collaborative spirit ensures that the library stays at the cutting edge of AI research and application.

Performance and Scalability

In the world of AI, performance is key, and the Hugging Face library doesn’t disappoint. It’s designed to handle large-scale models efficiently, which means you can work with some of the most advanced AI models without needing a supercomputer at your disposal.

Hugging Face is also not just about English. It supports multiple languages, which is essential for organizations and developers aiming to create AI applications for a diverse user base.

Popular Hugging Face Models

1. BERT (Bidirectional Encoder Representations from Transformers): BERT excels in understanding the context of a word in a sentence, making it effective for tasks like sentiment analysis, question-answering, and language understanding. It’s widely used in chatbots, search engines, and to enhance user interaction with AI systems.
2. GPT (Generative Pretrained Transformer): Known for its ability to generate human-like text, GPT is used for creative writing, generating conversational responses, and even writing code. It’s particularly popular in chatbots, automated content creation tools, and customer service applications.
3. DistilBERT: A streamlined version of BERT, DistilBERT offers similar capabilities but is faster and requires less computational power. It’s ideal for environments where resources are limited, like mobile applications, and is used in tasks like text classification and information extraction.
4. RoBERTa (Robustly Optimized BERT Approach): An optimized version of BERT, RoBERTa is trained on a larger dataset and for a longer time, leading to improved performance. It’s used in more complex NLP tasks like sentiment analysis, language inference, and text classification.
5. T5 (Text-To-Text Transfer Transformer): T5 converts all NLP problems into a text-to-text format, providing a versatile approach to tasks like translation, summarization, and question answering. Its adaptability makes it valuable in diverse applications, from automated translation services to information summarization tools.

Each of these models has its unique strengths, and you should choose them based on the specific requirements of your tasks. Make sure to balance factors like computational resources, complexity of the task, and the desired level of performance.

How to Use the Hugging Face Transformers Library

Let me show you how easy it is to work with the Hugging Face Transformers library. We will implement a simple summarization script that takes in a large text and returns a short summary.

We will first import pipeline from the transformers library. In Hugging Face, a “pipeline” is like a tool that helps you perform a series of steps to change data into the form you want.

from transformers import pipeline

The pipeline makes it simple to use these tools for different jobs, without needing to know all the complex details about how these tools work on the inside. For this example, we will use the "summarization" pipeline.

summarizer = pipeline("summarization")

And we are now ready to start using the summarization pipeline. Let's pass in a long chunk of text and see what the response is.

text = """
The development of the internet has been one of the most transformative events in human history, altering virtually every aspect of modern life. Initially conceived as a military and academic network in the late 1960s, the internet evolved rapidly through the 1970s and 1980s, expanding its reach and capabilities with each passing year. The introduction of the World Wide Web in the early 1990s was a critical moment, making the internet much more accessible and user-friendly, sparking a global revolution in communication, business, and entertainment. As a tool for information dissemination, the internet has been unparalleled, allowing instant access to vast amounts of data from all over the world. It has democratized information, breaking down barriers that once existed due to geography or social status. The internet has also had a profound impact on commerce, giving rise to e-commerce and transforming traditional business models. The ease of online shopping and the rise of digital marketplaces have reshaped consumer habits and expectations. Socially and culturally, the internet has connected people across the globe, facilitating the exchange of ideas and cultures in a way that was previously unimaginable. However, it has also raised concerns about privacy, data security, and the digital divide. The rapid dissemination of information has sometimes led to the spread of misinformation, posing challenges for societies in discerning truth from falsehood. As the internet continues to evolve, it poses new challenges and opportunities, shaping the future of human interaction, governance, and technology.
"""

summary = summarizer(text)
print(summary[0]['summary_text'])

Here is a sample response:

 The introduction of the internet in the 1970s and 1980s was a major event for the world's first time . As a result, the internet has been able to connect people across the globe . The internet has also raised concerns about privacy and security in the digital age of 21.

That's how easy it is to work with the Hugging Face Transformers library.

Ethical AI and Transparency: A Step Towards Responsible AI

Since AI ethics are increasingly under the spotlight, Hugging Face commits to transparency and responsible AI development. The open-source nature of the library promotes a level of transparency that’s essential for ethical AI development. Users can see exactly how models are built and make informed decisions about their use.

AI is a field that never stands still, and neither does the Hugging Face Transformer Library. It’s continuously updated with the latest breakthroughs in AI research. This means that when you use Hugging Face, you’re always at the forefront of AI technology.

Finally, the real test of any tool is its applications in the real world, and here, Hugging Face excels. It’s used by academics for cutting-edge research and by companies for practical applications like sentiment analysis, content generation, and language translation.

Conclusion

In summary, the Hugging Face Transformer Library is more than just a collection of AI models. It’s a gateway to advanced AI for people of all skill levels. Its ease of use and the availability of a comprehensive range of models make it a standout library in the world of AI.

Whether you’re a seasoned AI expert or just starting, the Hugging Face library is a useful resource that can help you achieve your AI goals.

Hope you enjoyed this article. Find more beginner-friendly tutorials on AI at turingtalks.ai.

Imagine you’re running a business and you want to know what your customers think about your product. Or maybe you’re a movie director wanting to gauge the public reaction to your latest release.

This is where sentiment analysis comes into play.

Sentiment analysis is a technique used in text analysis that helps in identifying and categorizing opinions expressed in a piece of text.

Sentiment analysis determines whether the expressed opinion in a document, a sentence or an entity feature/aspect is positive, negative, or neutral.

In a world where data is king, sentiment analysis is a crown jewel. It’s like having a superpower to understand the emotional tone behind words at scale.

Companies use it to understand customer feedback on products and services. Governments and organizations use it to get a sense of public opinion.

In social media management, sentiment analysis is used for brand monitoring, customer service, and market research.

It’s not just about understanding how many people are talking about your brand or product, but how they feel about it.

What is Hugging Face?

Now, let’s talk about Hugging Face. No, it’s not what you think. You don’t go around hugging faces.

In the world of AI, Hugging Face is quite the star. It’s an AI community and platform that provides state-of-the-art tools and models for Natural Language Processing (NLP).

Think of it as a toolbox that gives you the power to understand and generate human language. It’s like having a linguistic wizard by your side.

Hugging Face’s most popular offering is the ‘Transformers’ library. The Transformers library comes packed with APIs and tools that let you easily grab and train top-notch pre-trained models.

When you pick these pre-trained models, you’re cutting down on compute costs and carbon footprint. Plus, you save loads of time and resources that you’d otherwise spend training a model from scratch.

These models solve common tasks across various domains, like:

• Natural Language Processing (NLP): Here, you can do a bunch of cool stuff like text classification, spotting names or entities in text, answering questions, language modelling, summarizing, translating, handling multiple-choice questions, and even generating text.
• Computer Vision: This involves image classification, spotting and outlining objects in images, and more.
• Audio: You can work on recognizing speech automatically and classifying different types of sounds.
• Multimodal Tasks: These are tasks that mix it up, like answering questions based on tables, recognizing text in images (like scanned documents), pulling out information from these documents, classifying videos, and answering questions based on images.

The neat thing about Transformers is that they’re flexible with different frameworks. Whether you’re into PyTorch, TensorFlow, or JAX, Transformers has got you covered.

Its ease of use and comprehensive nature make it a go-to for researchers, developers, and businesses alike.

Code for Sentiment Analysis

Now that you know what sentiment analysis and Hugging Face are, let’s write some code. We’ll use Python and the Hugging Face transformers library to build a simple sentiment analyzer.

You can either use your terminal, install Python and run the code, or use a Google Colab notebook. I would recommend the latter since it comes pre-installed with Python.

Install the transformerslibrary with this command:

pip install transformers

If you are using a Colab notebook, use a ! symbol before the command for the notebook to treat it as a shell command (Colab executes code as Python by default).

!pip install transfomers

Once the installation is complete, you can start using the library. First, let's import pipeline from the transformers library.

from transformers import pipeline

In Hugging Face, a “pipeline” is like a tool that helps you perform a series of steps to change data into the form you want. The pipeline makes it simple to use these tools for different jobs, without needing to know all the complex details about how these tools work on the inside.

Now let’s load the sentiment-analysis pipeline.

sentiment_pipeline = pipeline("sentiment-analysis")

Now would you believe me if I said we are pretty much done? Our sentiment analysis model is ready and we can pass text to the pipeline and get the label as well as a sentiment score.

# Run sentiment analysis
result = sentiment_pipeline("Every new day brings a chance to create joyful memories and embrace new opportunities.")

# Print the result
print(result)

This is the output of the above code:

[{'label': 'POSITIVE', 'score': 0.9998821020126343}]

If you want to pass multiple sentences, pass an array of inputs to the pipeline.

result = sentiment_pipeline(["Every new day brings a chance to create joyful memories and embrace new opportunities.","Despite the effort, the project failed to meet expectations, leading to disappointment and frustration among the team."])
print(result)

Following will be the output of the above code:

[{'label': 'POSITIVE', 'score': 0.9998821020126343}, {'label': 'NEGATIVE', 'score': 0.9997937083244324}]

I hope you understand how powerful the Hugging Face Transformer library is. This is just a sample of the many pre-trained models that Hugging Face provides. Unless you are working on a unique problem, you should find a pre-trained model in Hugging Face available for you to work with.

Summary

In this article, we’ve learned about sentiment analysis and Hugging Face, a powerful tool in the world of NLP. Most importantly, you’ve taken your first steps in performing sentiment analysis by using the Hugging Face Transformers library.

Remember, what we’ve covered is just the tip of the iceberg. The field of NLP is vast and constantly evolving. The Hugging Face Transformers library is a powerful ally in your journey through AI. It simplifies complex tasks and gives you access to pre-trained models, saving you time and resources.

Hope you enjoyed this article. Find more beginner-friendly articles on AI at turingtalks.ai

For example, you can analyze a large volume of textual content to extract a common theme. Companies and businesses utilize this technique to understand public opinion about their brand. This lets them make informed decisions and improve their services.

The ability to extract themes from a large amount of textual data is referred to as topic identification.

In this article, you will learn how to utilize NLP techniques for topic identification, enhancing your skillset as a data scientist. So sit back, because it's gonna be an interesting journey.

What is Topic Identification?

Topic identification, simply put, is a sub-field under natural language processing. It involves the process of automatically discovering and organizing the main themes or topics present in a collection of textual data.

There are several Natural Language Processing (NLP) techniques you can use to identify themes in text, from simple ones to more algorithm based techniques. In this article we will look at the common NLP techniques used for topic identification. We'll discuss these in more detail below.

I recently tweeted about the essence of NLP. It really is purely statistics, because there are different manipulations you can do to ensure that numbers serve as representations for text (since computers don't understand text).

Requirements for this Project

In order for you to be able to follow along and get hands-on practical experience while learning, you should have Python 3.x installed on your machine.

We'll also use the following libraries: Gensim, Scikit-Learn, and NLTK. You can install them using the Pip package installer with the following command:

pip install gensim nltk scikit-learn

Techniques Used in NLP for Topic Identification

There are various techniques you can use for topic identification. In this article, you will learn about some common NLP techniques that work quite well, from simple and effective methods to more advanced ones.

Bag of Words

Bag of Words (BoW) is a common representation used in NLP for textual data. You can use it to count the frequency at which each word occurs in a document.

BoW, in the context of topic identification, is based on the assumption that the more frequently a word occurs in a document, the more important it is. Then you can use those more common words to infer what the document is all about.

Bag of words is the simplest technique used to identify topics in NLP. While Bag of Words is simple and efficient, it is highly affected by stop words, which are common words in text data (like "the," "and," "is," and so on).

But once you eliminate the issue of stop words from the text, allowing you to perform effective text processing (using techniques like normalization), BoW can still prove effective in identifying some main topics.

Let's look at how you can use BoW to identify the topic below.

How to implement of Bag of Words in Python

A bit of background about the example article we'll use here: I got it from the BBC, and it's titled "US lifts ban on imports of latest Apple watch." The article discusses the lifted ban on Apple's latest watches, Ultra 2 and Series 9.

Now let's go over how to implement the bag of words in Python. I'll break this code block up into sections and explain each part as I go to make it a bit more easy to digest.

#import necessary libraries
from collections import Counter
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

article = "Apple's latest smart watches can resume being sold in the US after the tech company filed an emergency appeal with authorities.\
Sales of the Series 9 and Ultra 2 watches had been halted in the US over a patent row.\
The US's trade body had barred imports and sales of Apple watches with technology for reading blood-oxygen level.\
Device maker Masimo had accused Apple of poaching its staff and technology. \
It comes after the White House declined to overturn a ban on sales and imports of the Series 9 and Ultra 2 watches which came into effect this week.\
Apple had said it strongly disagrees with the ruling.\
The iPhone maker made an emergency request to the US Court of Appeals, which proved successful in getting the ban lifted."

In the above code, we're importing the necessary libraries that we'll use to implement the BoW.

We'll use the Counter library to count the frequency of each word, and the word_tokenize library to tokenize the document into individual word tokens so they can be counted. Lastly, the stopwords library will remove stop words from the document.

# Initialize english stopwords
english_stopwords = stopwords.words("english")

#convert article to tokens
tokens = word_tokenize(article)

#extract alpha words and convert to lowercase
alpha_lower_tokens = [word.lower() for word in tokens if word.isalpha()]

#remove stopwords
alpha_no_stopwords = [word for word in alpha_lower_tokens if word not in english_stopwords]

#Count word
BoW = Counter(alpha_no_stopwords)

#3 Most common words
BoW.most_common(3)

In the above code, we use the first line of code to extract all stop words in the English language. Then, the second line tokenizes the article string into individual words. The third line of code normalizes each word into lowercase and only extracts alphabetic words from the article. The last two lines of code are used to count the frequency of each word and select the most common three words.

Below is the output of the BoW model:

[('watches', 4), ('us', 4), ('apple', 3), ('emergency', 2)]

From this, we can infer that the article is all about "Apple's watches in the US". As you can see, with the simplicity in reasoning behind the bag of words, it is still possible to infer a bit of knowledge about the article.

Latent Dirichlet Allocation

Latent Dirichlet Allocation, or LDA for short, is a popular probabilistic model used in NLP and machine learning for topic modeling (using algorithms to identify topics). It is based on the assumption that documents are mixtures of topics, and topics are mixtures of words.

Simply put, LDA is an NLP technique used to identify the topic to which a document belongs based on the words contained in the document.

LDA operates on the bag-of-words representation of documents, where each document is represented as a vector of word frequencies. You can implement LDA using the Gensim library in Python (which is an open source library used for topic modelling and document similarity analysis).

Steps for implementing LDA include:

• Import Libraries: First step is to import the necessary libraries you will be utilizing.

• Data Preparation: Convert raw data to a document format then tokenize, remove stop words, and optionally perform stemming or lemmatization.

• Create Dictionary and Corpus: Build a dictionary with unique word IDs. Then form a bag of words corpus representing document-word frequency.

• Train LDA Model: Use the document-word frequency and dictionary to train the LDA model, setting the desired number of topics.

• Print Topics: Explore and print the discovered topics.

# Import the necessary libraries
from gensim.corpora.dictionary import Dictionary
from gensim.models import LdaModel
from nltk import sent_tokenize, word_tokenize
from nltk.corpus import stopwords

article = "Apple's latest smart watches can resume being sold in the US after the tech company filed an emergency appeal with authorities. \
Sales of the Series 9 and Ultra 2 watches had been halted in the US over a patent row. \
The US's trade body had barred imports and sales of Apple watches with technology for reading blood-oxygen level. \
Device maker Masimo had accused Apple of poaching its staff and technology. \
It comes after the White House declined to overturn a ban on sales and imports of the Series 9 and Ultra 2 watches which came into effect this week. \
Apple had said it strongly disagrees with the ruling. \
The iPhone maker made an emergency request to the US Court of Appeals, which proved successful in getting the ban lifted."

The above lines of code include the necessary libraries that we'll use to implement the LDA.

The first line of code contains the Dictionary object. Then, the second line imports the LDA model, and the third line of code contains the sent_tokenize, which we'll use to convert the article into document. After that, word_tokenize will tokenize the document into individual words. Lastly, we have the stop_words library.

# convert article to documents
documents = sent_tokenize(article)

#toeknize and normalize the document
tokenized_words = [word_tokenize(doc.lower()) for doc in documents]

# remove stops words and onl extract alphabets
cleaned_token = [[word for word in sentence if word not in english_stopwords and word.isalpha()]
                 for sentence in tokenize_words]

# create a dictionary
dictionary = Dictionary(cleaned_token)

# Create a corpus from the document
corpus = [dictionary.doc2bow(text) for text in cleaned_token]

The above lines of code include the preprocessing steps that will be performed on the article, including converting the article to a document, normalizing, and tokenizing the document into individual words.

The next part removes stopwords from the text and then extracts words and numbers from the document. After that, we create a dictionary, which is a map between each word and its numerical identifier. The last line of code then creates a corpus of the document.

# Build the LDA model
model = LdaModel(corpus=corpus, id2word=dictionary, num_topics=3)

# Print the topics
print("Identified Topics:")
for idx, topic in lda_model.print_topics():
    print(f"Topic {idx + 1}: {topic}")

The above code is used to train the model on the corpus and then prints the top 3 topics from the article.

Below is the output of the LDA Model:

Identified Topics:
Topic 1: 0.045*"9" + 0.045*"ultra" + 0.044*"sales" + 0.044*"2" + 0.043*"series" + 0.043*"watches" + 0.029*"apple" + 0.028*"ruling" + 0.028*"disagrees" + 0.028*"said"
Topic 2: 0.051*"maker" + 0.035*"ban" + 0.035*"us" + 0.031*"emergency" + 0.031*"made" + 0.031*"successful" + 0.031*"court" + 0.031*"lifted" + 0.031*"request" + 0.031*"proved"
Topic 3: 0.055*"apple" + 0.054*"us" + 0.054*"watches" + 0.031*"sales" + 0.031*"technology" + 0.031*"imports" + 0.031*"authorities" + 0.031*"barred" + 0.031*"appeal" + 0.031*"filed"

The LDA technique shows some improvement as compared to BoW method. We can still obtain a more information that the article is all about a ban related to Apple ultra series watches in the US.

Non-Negative Matrix Factorization

Non-Negative Matrix Factorization (NMF), just like LDA, is another topic modeling technique that uncovers latent topics in a collection of documents.

But instead of relying on BoW, it relies on the Term Frequency-Inverse Document Frequency (TF-IDF) representation to capture and retrieve hidden themes or topics from the documents.

By incorporating TF-IDF information, NMF is able to weigh the importance of terms, thereby identifying more hidden patterns. You can perform NMF using the Scikit-learn library.

Steps for performing NMF

• Import necessary libraries

• Data Preparation: Convert text into document, then perform necessary data preparation like removing stop words. The TF-IDF function in Scikit-Learn has as an argument that does that.

• Convert the document to a TF-IDF matrix using the TF-IDF vectorizer in Scikit-learn

• Apply the NMF function on the TF-IDF matrix and specify the numbers of topic you want and the number of words in each topic

• Lastly, interpret your result.

# import the necessary libraries
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import NMF

article = "Apple's latest smart watches can resume being sold in the US after the tech company filed an emergency appeal with authorities. \
Sales of the Series 9 and Ultra 2 watches had been halted in the US over a patent row. \
The US's trade body had barred imports and sales of Apple watches with technology for reading blood-oxygen level. \
Device maker Masimo had accused Apple of poaching its staff and technology. \
It comes after the White House declined to overturn a ban on sales and imports of the Series 9 and Ultra 2 watches which came into effect this week. \
Apple had said it strongly disagrees with the ruling. \
The iPhone maker made an emergency request to the US Court of Appeals, which proved successful in getting the ban lifted."

The above code contains the libaries that we'll use to implement NMF and the article itself.

# convert article to documents
documents = sent_tokenize(article)

# Create a TF-IDF vectorizer
tfidf_vectorizer = TfidfVectorizer(stop_words='english').fit_transform(document)

# Apply NMF
num_topics = 5  # Set the number of topics you want to identify
nmf_model = NMF(n_components=num_topics, init='random', random_state=42)
nmf_matrix = nmf_model.fit_transform(tfidf)

The above code converts the article into documents. Then it creates a Term-Frequency Inverse Document Frequency matrix of the article document. The last three lines of code then define the number of topics and create the topics from the document matrix using the NMF.

Below is the output of the NMF Model:

Topic #1: ultra, series, sales, watches, row, halted, patent, white, house, effect
Topic #2: lifted, court, iphone, getting, request, successful, proved, appeals, ban, maker
Topic #3: disagrees, strongly, ruling, said, apple, body, blood, level, trade, oxygen
Topic #4: filed, resume, appeal, latest, tech, authorities, sold, smart, company, emergency
Topic #5: technology, apple, accused, masimo, device, staff, poaching, maker, trade, level

You can see that NMF reveals more insights concerning the themes of the document. For example, you can tell that another company called Masimo is accusing Apple of a patent infringement in their Ultra series watches.

How to Choose Which Technique to Use?

I recommend experimenting with all the approaches in order to gain different perspectives concerning the contents of your document.

Bag of Words and LDA are based on how frequently words occur, making these techniques useful for inferring the biggest/most general themes about the document.

On the other hand, when using NMF, which is based on TF-IDF, less frequent words can be used to infer additional topics and provide a different perspective on the document.

For example, NMF was able to identify key terms like "Masimo" and "accused," whereas LDA was not able to do this. So depending on your needs, go ahead and experiment with all the approaches to see which one is able to yield better results.

Conclusion

In this article, you've learned about topic identification and how you can use it to extract themes or topics from a large document.

We covered some different techniques you can use to identify topic including simple ones like BoW and more advanced ones like LDA and NMF.

Happy learning, and see you in the next one.

What is Hugging Face 🤗?

If you are interested in Artificial Intelligence and Natural Language Processing, you have probably heard of Hugging Face – the company named after a cute emoji.

Hugging Face is not only a company, but also a platform that is transforming the fields of AI and NLP through open source and open science.

Hugging Face offers a platform called the Hugging Face Hub, where you can find and share thousands of AI models, datasets, and demo apps. The Hub is like the GitHub of AI, where you can collaborate with other machine learning enthusiasts and experts, and learn from their work and experience.

Hugging Face’s mission is to democratize good machine learning, one commit at a time. Whether you are a beginner or a professional, you can benefit from the amazing resources and tools that Hugging Face provides.

In this post, I'll guide you through the basics of Hugging Face. You'll learn how to create your Hugging Face account, set up your development environment, and use some of the pre-trained models that are available on the Hub. Let’s get started! 🚀

Here's what we'll cover:

1. What can you do on the Hugging Face Platform?
   • Download and fine-tune existing Open Source models
   • Run models directly from Hugging Face
   • Add/create your own model
   • Use existing datasets
   • Create/browse demo apps (also known as Spaces)
   • Join or create an organization
   • Create a portfolio
   • Learn AI skills
2. Hugging Face terminology
3. How to get started with Hugging Face
   • Create a Hugging Face account
   • Set up your environment
4. How to use pre-trained models in Hugging Face
5. How to find the right pre-trained model
6. What's next?

What Can You Do on the Hugging Face Platform?

Here are some of the awesome things you can do on Hugging Face:

Download and fine-tune existing Open Source models:

Why start from scratch when you can leverage the power of over 450k models that are already available on the Hugging Face model library?

You can easily download these models and fine-tune them on your own custom dataset with just a few lines of code. This way, you can save time and resources, and still get a model that suits your specific needs.

You can use these models to perform various tasks, such as:

1. Natural language processing (for example, translation, summarization, and text generation)
2. Audio-related functions (for example, automatic speech recognition, voice activity detection, and text-to-speech)
3. Computer vision tasks (for example, depth estimation, image classification, and image-to-image processing),
4. Multimodal models capable of handling diverse data types (text, images, audio) and producing multiple types of output.

Run Models directly from Hugging Face:

If you don’t want to set up these models on your own machines, you can simply use Hugging Face’s Transformer library to connect to these models, send requests, and receive outputs.

Add/create your own model:

If you have a brilliant idea for a new model, or you want to improve an existing one, you can also add/create your own model on Hugging Face.

The platform will host your model, and allow you to provide additional information, upload essential files, and manage different versions. You can also choose whether your models are public or private, so you can decide when or if you want to share them with the world.

Once your model is ready, you can access it directly from Hugging Face, send requests, and retrieve the outputs for integration into any applications you are developing.

Use existing datasets:

A good model needs a good dataset. Hugging Face provides a repository of over 90,000 datasets that you can use and feed into your models.

You can take an in-depth look inside the dataset using the dataset viewer. You can also contribute your own datasets to the repository, and help the machine learning community grow.

Screenshot of dataset viewer

Create/browse demo apps (also known as Spaces):

Hugging Face’s Spaces are Git repositories that allow you to showcase your machine learning applications. You can also browse and try out the Spaces created by other users, and find inspiration for your next AI app.

With thousands of ML apps to choose from, you will never run out of fun and interesting things to do.

Here are a few cool Spaces you can check out:

• OpenAI's Whisper: Transcribe long-form microphone or audio inputs with the click of a button.
• AI Comic Factory: Create your own comic books.
• QR Code AI Art Generator: Generate beautiful QR codes using AI.
• Stable Video Diffusion (Img2Vid - XT): Generate 4s video from a single image.
• Video-LLaMA: Audio-Visual Language Model for Video Understanding.

Join or create an organization:

You can join or create your own organization on Hugging Face. This allows you to showcase your work and collaborate with other members from your university, lab, or company. You can also work on private datasets, models, and spaces with your organization.

Create a portfolio:

You can create a professional portfolio on Hugging Face to showcase your work and start building your reputation. This can help you land jobs related to AI model training, integration, and development.

Hugging Face provides the basic computing resources for running the demo app, including 16 GB of RAM, 2 CPU cores, and 50 GB of disk space for free. You can also upgrade your hardware for improved and faster performance with paid options.

Learn AI Skills:

Hugging Face is an excellent platform for learning AI skills. It offers a comprehensive set of tools and resources for training and using models. This includes demos, use cases, documentation, and tutorials that guide you through the entire process of using these tools and training models.

You can also learn from the experts and the community on Hugging Face, and improve your AI knowledge and skills.

Hugging Face Terminology

There are some terms you'll need to know to get the most out of working with Hugging Face.

Pretrained model: A model that has been trained on a large dataset for a specific task before being made available for use.

Inference: Inference is the process of using a trained model to make predictions or draw conclusions about new, unseen data based on the learned patterns from the training data.

Transformers: Transformers are models that can handle text-based tasks, such as translation, summarization, and text generation. They use a special architecture that relies on attention mechanisms to capture the relationships between words and sentences.

Tokenizer: A tokenizer is a process that breaks down text into smaller units called tokens. Tokens are usually words or subwords that can be used for natural language processing (NLP) tasks.

How to Get Started with Hugging Face

To get started with HuggingFace, you will need to set up an account and install the necessary libraries and dependencies. Don’t worry, it’s easy and fun!

Here are the steps you need to follow:

Create a Hugging Face Account

Signing up as a Community individual contributor is free of charge. You can also opt for a ‘Pro’ plan or a customized plan for Organizations if you need more features and resources.

Go to the Hugging Face website and click on “Sign Up” to create a free account.

Then enter your email address and a password. Click next and complete your profile and security check.

Setting up a Hugging Face account

Congratulations, you are now a Hugging Face member! 🎉 You will be directed to the Hugging Face ‘Welcome’ page, where you can find more information and tips on how to use the platform.

As a bonus, you also get a Git-based hosted repository where you can create your Models, Datasets and Spaces. You can do this directly using the website or using the CLI. If you prefer the latter, you can check the detailed instructions on the ‘Welcome’ page under the ‘Programmatic access’ section.

Hugging Face welcome screen showing options to create a new model, browse the docs, and set up programmatic access

Set Up Your Environment

Before you start using the Hugging Face hub programmatically, you will need to set up your environment.

Step 1: Install Python and Pip:

Make sure you have Python 3.8 or higher installed on your system. You will also need Pip, the package manager for Python, to install the Hugging Face libraries. If you don’t have Python, you can install it by following the instructions here.

Step 2: Install HuggingFace libraries:

Open a terminal or command prompt and run the following command to install the HuggingFace libraries:

pip install transformers

This will install the core Hugging Face library along with its dependencies. To have the full capability, you should also install the datasets and the tokenizers library.

pip install tokenizers, datasets

Step 3: Set up a development environment:

Choose a code editor or IDE of your choice, such as Jupyter Notebook, PyCharm, or Visual Studio Code. Create a new project directory and set up a virtual environment to isolate your project dependencies. You can find more information on how to do this here.

With these steps completed, you have successfully set up Hugging Face on your system and are ready to start exploring its features and capabilities. Let’s go! 🚀

How to Use Pre-Trained Models in Hugging Face

One of the best things about Hugging Face is that it gives you access to thousands of pre-trained models that can perform various tasks on different types of data. Whether you are working with text, vision, audio, or a combination of them, you can find a model that suits your needs.

Hugging Face has two main libraries that provide access to pre-trained models: Transformers and Diffusers. The Transformers library handles text-based tasks, such as translation, summarization, and text generation. Diffusers can handle image-based tasks, such as image synthesis, image editing, and image captioning.

You have already installed the transformers library during the environment setup. Let’s see how you can use it to work with pre-trained models.

Step 1: Visit the PyPI page

To learn more about the transformers library, you can visit its page on PyPI, the Python Package Index.

Go to PyPi and search for ‘transformers’. Click on the latest version of the transformers library displayed in the search result. You will see a brief introduction of the library, as well as some useful links and information.

Step 2: Download and use pre-trained models

The transformers library provides APIs to quickly download and use pre-trained models on a given text, fine-tune them on your own datasets, and then share them with the community on Hugging Face’s model hub.

Step 3: Use the pipeline() method

To use a pre-trained model on a given input, Hugging Face provides a pipeline() method, an easy-to-use API for performing a wide variety of tasks.

The pipeline() method makes it simple to use any model from the Hub for inference on any language, computer vision, speech, and multimodal tasks.

Let’s try to perform a task using the pipeline() method.

Task: Sentiment analysis:

Let’s use the pipeline() method to classify positive versus negative texts provided by the user:

from transformers import pipeline

# Load the pre-trained sentiment analysis model
sentiment_analysis = pipeline(
"sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

input_text = [
"It’s a great app, my biggest problem is the card readers regularly do not connect. Which is very poor customer service for us because we have to manually enter our customers debit cards, which takes time. This slows down our efficiency."
]

# Perform sentiment analysis on the input text
result = sentiment_analysis(input_text)

# Print the result
print(result)

The pipeline statement downloads and caches the pretrained model used by the pipeline, while the statement result = sentiment_analysis(input_text) evaluates it on the given text.

Output:

[{'label': 'NEGATIVE', 'score': 0.9996176958084106}]

Here, the answer is "NEGATIVE" with a confidence of 99.96%.

Task: Automatic speech recognition

Let’s try another task that involves speech recognition.

from transformers import pipeline

transcriber = pipeline(task="automatic-speech-recognition",
                       model="openai/whisper-small")
result = transcriber(
    "https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac")

print(result)

Output:

{'text': ' I have a dream that one day this nation will rise up and live out the true meaning of its creed.'}

You can see how easy it is to get a pre-trained model up and running using Hugging Face's libraries.

How to find the right pre-trained model

But how can you find the right pre-trained model if you want to perform a specific task?

This is actually quite easy. You can browse the models on the Hugging Face website, and filter them by task, language, framework, and more. You can also search for models and datasets by keyword and sort them by trending, most likes, most downloads, or by recent updates.

Searching for models

Each model has a model card that contains important information, such as model details, inference example, training procedure, community interaction features, and link to the files. You can also try the model on the model card page by using the Inference API section.

Inference API

You can also check the list of spaces that are using that particular model and further explore the spaces by clicking on the space link.

Spaces

What's Next?

In this guide, you have learned the basics of Hugging Face, and how to use its libraries, models, datasets, and spaces. But there is so much more to discover and enjoy!

Here are some tips on how to make the most of Hugging Face:

• Dive into Hugging Face’s Spaces: Spaces are where the magic happens. You can find and try out thousands of machine learning applications created by the community, and see what’s trending and popular. You can also create your own spaces and showcase your work to the world.
• Explore the Hugging Face documentation and tutorials: If you want to learn more about the Hugging Face platform and its features, you can check out the documentation and tutorials. They provide detailed information and guidance on how to use the tools and resources that Hugging Face offers. You can also find information about common ML/AI tasks, such as text classification, image generation, and speech recognition, on the tasks page.
• Visit the learn section: If you are interested in acquiring new skills and knowledge in AI and NLP, you can visit the ‘learn’ page that displays courses from Hugging Face. Here, you can learn from the experts and the best practices in the field, and apply them to your own projects.
• Join the Hugging Face community: Machine learning is more fun when collaborating! You can join the Hugging Face community on platforms like GitHub, Discord, and Twitter to connect with other users and stay updated on the latest developments. You can also share your feedback, questions, and ideas with the community, and help Hugging Face grow and improve.

Hugging Face is not just a platform for AI and NLP – it's also a playground for your curiosity and creativity. You can experiment with new models, expand your AI knowledge, and enrich your AI toolkit with various tools and resources. So, keep learning, keep exploring. There is always something new and exciting to discover with Hugging Face. 😊

For centuries, this dance was uniquely human. But with the advent of modern computing, a new question emerged: can machines understand our language?

The answer, as many of us know, is a resounding “yes!” — but how do they do it? Let’s look at how Natural Language Processing (NLP) helps computers decode and derive context from our language.

The Building Blocks: Tokens

Imagine reading a sentence.

To make sense of it, your brain breaks it down, recognizing individual words and their roles. Computers do something similar called tokenization.

Tokenization splits a piece of text into smaller units, or “tokens”, which are typically words or subwords. This is the computer’s first step in processing text data.

For example, the sentence “Computers are smart” would be tokenized into [‘Computers’, ‘are’, ‘smart’].

Understanding Word Forms: Stemming and Lemmatization

Once a computer tokenizes a text, it needs to understand different word forms.

Consider the words “running”, “runner”, and “ran”. To us, they are related. But a computer sees them as separate words. Enter stemming and lemmatization.

Stemming

Stemming simplifies words to their foundational form. For example, in this example, variations like “running”, “runner”, or “runs” are all stripped down to the basic root, which is “run”.

Stemming helps simplify the text data, making it easier for algorithms to analyze and process. While it’s useful for certain tasks, it’s important to note that stemming can sometimes lead to inaccurate results, as it might trim words too much and lose some of their original meaning.

For more nuanced tasks, other techniques like lemmatization might be more appropriate.

Lemmatization

Lemmatization reduces a word to its base or canonical form, called a lemma.

Unlike stemming, which simply trims words, lemmatization considers the context and meaning of the word. It ensures that the words are transformed into a valid base form. For instance, the word “better” might be lemmatized to “good”, and “running” would be lemmatized to “run”.

By using lemmatization, we can group different forms of a word together so that they’re treated as a single item. This is useful when analyzing text data, as it helps in recognizing that different word forms are essentially conveying the same concept.

Lemmatization often requires more computational resources than stemming since it has to consider word meanings and structures. It’s also typically dependent on dictionaries or morphological analysis tools.

Understanding Context with Syntax and Semantics

Words interact with each other, influencing their meanings based on their neighbouring words. To grasp this context, computers analyze both syntax and semantics.

Take the word “bat” as an example. In the sentence “I played with the bat,” “bat” refers to a sporting tool. However, in the sentence “The bat flew in the night,” “bat” indicates a flying mammal.

Through syntax, computers determine a word’s function in a sentence, and with semantics, they interpret its exact meaning given that function.

The Power of Word Embeddings

Computers are great with numbers, but not so much with words.

To bridge this gap, words are often converted into vectors of numbers in a process called word embedding. These vectors capture the semantic meaning of words.

Words with similar meanings tend to have similar vectors. This numerical representation allows computers to perform mathematical operations on words, leading to tasks like finding word similarities or even analogies.

I recently published an article on word embeddings and you can read the full article here.

The Final Piece: Machine Learning

All the above processes feed into machine learning models.

These models, trained on vast datasets, use patterns in the text to make determinations. Datasets can include various examples and scenarios, allowing the models to learn and recognize patterns, trends, and relationships within the text.

Once trained, when these models encounter new textual information, they analyze it by looking for familiar patterns they’ve learned. For example, is a given piece of text positive or negative in sentiment? Or a review stating “The movie was captivating,” versus “It was a dull watch.”

These models can then power products like language translation and transformers. There are more steps involved in breaking down language for NLP, but these are all the ones that you will use almost on a daily basis as an AI engineer.

Summary

The journey from text to meaning is a complex one, even for humans. From breaking down sentences to understanding context and leveraging the power of machine learning, computers have come a long way in deciphering human language.

As technology continues to advance, we can only anticipate even more deep interactions between humans and machines, facilitated by the power of Natural Language Processing.

If you found this article interesting, join my newsletter and I ll send you an email with my content every Friday.

What Are Word Embeddings?

Word embeddings serve as the digital DNA for words in the world of natural language processing (NLP). In essence, word embeddings convert words into numerical vectors (a fancy term for arrays of numbers). These vectors can be processed by machine learning algorithms.

Think of these vectors as a numeric fingerprint for each word. For example, the word “apple” might be represented by a numerical vector like [0.2, -0.4, 0.7].

The main benefit of word embeddings is their ability to capture the semantic essence of words. In simpler terms, they help machines understand the meaning and nuances behind each word.

For example, if “apple” is close to “fruit” in this numerical space but far from “car,” the machine understands that an apple is more related to fruits than to vehicles.

Beyond individual meaning, word embeddings also encode relationships between words. Words that often appear together in the same context will have similar or ‘closer’ vectors.

Word embeddings

For example, in the numerical space, the vectors representing “king” and “queen” might be closer to each other than those representing “king” and “apple.” This is because the algorithm has learned from numerous texts that “king” and “queen” often appear in similar settings, such as discussions about royalty, while “king” and “apple” do not.

Why Do We Need Word Embeddings?

Traditional language models treated words as separate, isolated entities.

For instance, the word “dog” might be represented as a unique identifier, say 1, while the word "cat" as 2. This approach fails to capture the relationship between "dog" and "cat," which are both animals and pets.

Word embeddings solve this problem by placing words with similar meanings or contexts close to each other in a multi-dimensional space.

Algorithms for Generating Embeddings

Word2Vec

Researchers at Google developed Word2Vec, which employs neural networks to generate word embeddings. The model processes a large text corpus and outputs high-quality word vectors.

It determines these embeddings by analyzing the context in which words appear, based on the idea that words found in similar contexts likely share semantic meaning.

GloVe (Global Vectors for Word Representation)

Stanford researchers developed GloVe, which constructs a large table to monitor the frequency with which words co-occur in a text dataset. The model then employs mathematical methods to simplify this table, generating numerical vectors for individual words.

These vectors encapsulate both the meaning and the relationships among words, laying the groundwork for various machine-learning tasks related to language.

FastText

Facebook’s AI Research lab created FastText, which improves upon the Word2Vec model by viewing words as assemblies of smaller character strings, or character n-grams.

This method enables the model to more effectively capture the intricacies of languages that have complex word structures and to incorporate words not present in the original training data. Consequently, FastText yields a more adaptable and comprehensive language model useful for a diverse set of machine-learning tasks.

Word Embeddings and GPTs

Word embeddings and GPTs

Word embeddings are a foundational component in GPT models like GPT-2, GPT-3, and GPT-4. However, the architecture and approach are a bit more advanced compared to simpler models that solely rely on word embeddings.

In traditional models that use word embeddings like Word2Vec or GloVe, each word is converted into a fixed vector in a pre-defined space. These vectors are then used as the input to other machine learning algorithms for tasks like classification, clustering, or even in sequence-to-sequence models for machine translation.

In contrast, GPT models use a variant known as “transformer embeddings,” which not only embeds individual words but also considers the context in which a word appears.

This is essential for understanding the meaning of words that can change based on their surrounding words. For example, the word “bank” could mean a financial institution or the side of a river, depending on the context.

The GPT architecture takes a sequence of words (or more precisely, tokens) as input and processes them through multiple layers of transformer blocks. These blocks output a new sequence of vectors that represent not just the individual words, but also their relationships with all other words in the input sequence.

This sequence is then used for NLP tasks, from text completion to translation and summarization.

So, while GPT models do use embeddings, they are far more dynamic and context-aware than traditional word embeddings. The embeddings in GPT models are part of a larger, more complex system designed to understand and generate human-like text based on the input it receives.

Conclusion

Word embeddings offer an effective and computationally efficient way to represent words as vectors, capturing the intricacies of language in a form that machines can understand. They lie at the heart of many NLP applications, improving the accuracy and sophistication of language models.

As technology continues to evolve, so will the methods for generating and utilizing word embeddings, promising even more robust and nuanced language processing capabilities in the years to come.

If you found this article interesting, join my newsletter and I'll send you an email with my content every Friday.

spaCy excels at large-scale information extraction tasks. It's written from the ground up in carefully memory-managed Cython.

spaCy is designed to help us build real products, or gather real insights. It's built with 73+ languages, and supports custom models built with Pytorch and Tensorflow. It's robust and has rigorously evaluated accuracy.

You may not know much about Cython. So let's have a quick look at it.

What is Cython?

Cython is a Python compiler that makes writing C extensions for Python as easy as Python itself. Cython is based on Pyrex, but supports more cutting edge functionality and optimizations.

To put in simple words, it's a Python to C compiler.

Quoting from Wikipedia,

Cython is a programming language, a superset of the Python programming language, designed to give C-like performance with code that is written mostly in Python with optional additional C-inspired syntax.

So you might be wondering if you need to learn Cython to help you fine-tune your spaCy models.

Well, don't worry. You don't need to learn Cython to to work with spaCy. I just wanted to make sure you're aware of what it is to help you get the most out of this tutorial.

Prerequisites

Basic knowledge of spaCy

The official documentation site of spaCy provides a lot of information about the tool. Alternatively, you can read my other tutorial which gives some basic information about spaCy.

Basic knowledge of how to gather data

In order to fine-tune any model, you need to have the data ready. And it should be good quality data.

In this tutorial, let's assume we've built a event management software. We want to add voice assistance to our software. We have a module that converts voice input into text. Our next step is to process this text and extract data from the given sentence using spaCy.

We have to gather some basic sentences that we hear from people trying to schedule a event. Here are a few examples:

1. Schedule event for visit to Trivandrum on July 18
2. Create event happening tomorrow on AI
3. Schedule Pongal celebration event in Oaks HOA at June 20, 2023

Similarly, we have to collect prompts related to event scheduling. The more data you collect and input, more accurate your model will be.

I created 7 sentences, which is much too small for a event management software company to train its model. But from a demo standpoint, it should be enough.

How to Pre-Process the Data

Collecting data covers just one part of the equation. We need to pre-process the data and transform it in a way that spaCy can easily understand. We should also define what kind of data (tags) should be identified from the given sentences.

Let's take the following sentence as an example:

"Schedule event for visit to Trivandrum on July 18".

Let's try to split out some tags from above sentence:

• Schedule – this belongs to the "action" tag
• event – this belongs to the "domain" tag
• visit to Trivandrum – this belongs to the "name" tag
• July 18 – this belongs to the "date" tag

Every tag defined above may contain alternatives in other sentences. For an example, we may input the following sentences:

1. Cancel client meeting scheduled tomorrow
2. Change time of mall visit to 6 PM

From the above sentences, the action tags are "Cancel" and "Change". Similarly data for each tag may vary for each sentence.

Our next step is to teach spaCy about the words for each tag. We need to prepare a JSON file that contains examples with the tags and their indices.

For example, in the above sentence ("Schedule event for visit to Trivandrum on July 18"), the index for the "action" tag starts from 0 (indices always start from 0 in Python) and ends at 7.

Similarly for all 7 sentences I've chosen, I've prepared the index for each tag and created the JSON file.

How to Fine-Tune the spaCy Model

Let's try to fine-tune spaCy with the data that we have.

Create a folder and download the above JSON file and place it into the folder. Create a new file named custom_model.ipynb.

All the following sections below need a code block to be created. Create a code block wherever you see a heading. Here's a sample screenshot.

Creating and running blocks of code

Import spaCy

import spacy

Load the pre-trained model

nlp = spacy.load("en_core_web_lg")

Import the JSON file

Import the above downloaded JSON file.

import json

with open('./event_schedule_data.json', 'r') as f:
    data = json.load(f)

Convert the data

Convert the data read from JSON file into tuple of dictionaries containing original text and entities.

training_data = []
for example in data['examples']:
    temp_dict = {}
    temp_dict['text'] = example['content']
    temp_dict['entities'] = []
    for annotation in example['annotations']:
        start = annotation['start']
        end = annotation['end'] + 1
        label = annotation['tag_name'].upper()
        temp_dict['entities'].append((start, end, label))
    training_data.append(temp_dict)
print(training_data[0])

The above code will convert the data to the required format and print the first dictionary in the tuple, which will look something like below:

{'text': 'Schedule a calendar event in Teak oaks HOA about competitions happening tomorrow', 'entities': [(0, 8, 'ACTION'), (11, 25, 'DOMAIN'), (29, 42, 'HOA'), (49, 71, 'EVENT'), (72, 80, 'DATE')]}

Import training libraries

from spacy.tokens import DocBin
from tqdm import tqdm
from spacy.util import filter_spans

nlp = spacy.blank('en')

Train the model

The below code will create a custom model with the data that we give. A binary file named train.spacy will be generated at the end.

doc_bin = DocBin()
for training_example in tqdm(training_data):
    text = training_example['text']
    labels = training_example['entities']
    doc = nlp.make_doc(text)
    ents = []
    for start, end, label in labels: 
        span = doc.char_span(start, end, label=label, alignment_mode="contract")
        if span is None:
            print("Skipping entity")
        else:
            ents.append(span)
    filtered_ents = filter_spans(ents)
    doc.ents = filtered_ents
    doc_bin.add(doc)

doc_bin.to_disk("train.spacy")

Create config files

Create a new file named base_config.cfg and copy the below code into it.

Create another file named config.cfg and copy the below code into it.

Don't worry. These are default configurations that I've taken from their official documentation and I've not made any changes to it.

Initialize spaCy with the config files

Run the following command in the notebook code block to initialize spaCy with the config file. This config file will be used to train the spaCy model with our generated custom model.

!python -m spacy init fill-config base_config.cfg config.cfg

Train spaCy model

Run the following command to train the spaCy model:

!python -m spacy train config.cfg --output ./ --paths.train ./train.spacy --paths.dev ./train.spacy

This may take some time depending on your system configuration. Ideally not too long (around 5 to 10 minutes). At the end, it'll generate 2 folders named model-best and model-last.

Load the best model

nlp_ner = spacy.load("model-best")

Test our model

Let's test our model with the following input.

"Could you please reserve a team brainstorming session on coming Wednesday at 11 AM?"

doc = nlp_ner("Could you please reserve a team brainstorming session on coming Wednesday at 11 AM?")

spacy.displacy.render(doc, style="ent")

You should be surprised to see the output.

Entity representation of our input sentence

That's great right?

Eventually, you may have a question: as a programmer, how can I get this data in my backend code?

Well, that's something everyone asks.

spaCy has an answer for it. You can expose the above data as JSON.

Convert extracted data to JSON

json_obj = doc.to_json()
json_obj

This will show a similar output like the one below.

JSON of our test sentence input

Write a REST API and expose this data as JSON. That's it. But remember, spaCy will give you only the indices, you have parse your sentence and extract words in between those indices.

Conclusion

In this article, we learnt about how to customize and fine-tune a pre-trained spaCy model with the data that corresponds to our domain knowledge.

Similarly you can also train with your domain specific data. The model that you fine-tune will be private to you unless you expose it to the public. So it's best suited for training with the domain data that is not publicly available.

If you wish to learn more about NLP/Machine Learning, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

NLP is a sub-field of artificial intelligence. It focuses on enabling machines to understand, interpret, and generate human language.

In this tutorial, we'll explore the fundamental concepts of NLP and we'll look at a particular implementation with spaCy. This will showcase its immense potential to revolutionize various industries.

Let's have a quick look at Natural Language Processing before we begin with spaCy.

What is Natural Language Processing?

NLP involves the intersection of linguistics, computer science, and machine learning. Its primary objective is to bridge the gap between human language and machine understanding.

NLP encompasses a wide range of tasks, including Text Classification, Named Entity Recognition (NER), Sentiment Analysis, and more.

Text Classification

This involves categorizing text into predefined classes or categories based on its content.

This has applications in sentiment analysis, spam detection, topic classification, and more.

Named Entity Recognition (NER)

This involves identifying and extracting named entities such as names, organizations, locations, and dates from text.

NER is crucial for information extraction, question answering systems, and recommendation engines.

Sentiment Analysis

This involves determining the sentiment or emotion expressed in a piece of text, whether it's positive, negative, or neutral.

Sentiment analysis is extensively used for brand monitoring, customer feedback analysis, and social media monitoring.

Challenges in Natural Language Processing

While NLP has made significant advancements, several challenges persist:

1. Human language is inherently ambiguous, making it challenging sometimes for machines to accurately understand and interpret meaning.
2. Different languages, dialects, slang, and cultural nuances add complexity to NLP tasks, requiring models to be language-specific and adaptable.
3. Capturing contextual information and understanding the underlying semantics of text remains a significant challenge for NLP algorithms.
4. NLP models heavily rely on training data, and biased or low-quality data can result in biased or inaccurate predictions, leading to potential ethical concerns.

What is spaCy?

In the world of Natural Language Processing (NLP), spaCy has emerged as a powerful and efficient library, revolutionizing the way developers and researchers work with text data.

spaCy is an open-source Python library designed specifically for NLP tasks such as part-of-speech tagging, named entity recognition, dependency parsing, and more.

It was developed with the goal of providing industrial-strength performance, while still being easy to use and integrate into existing workflows.

spaCy is built on the latest research and implements state-of-the-art techniques, making it an ideal choice for both beginners and experienced NLP practitioners.

Key Features of spaCy

Linguistic Annotations

spaCy provides a wide range of pre-trained models that can quickly analyze text and extract various linguistic features. These features include part-of-speech tags, named entities, syntactic dependencies, sentence boundaries, and more.

The pre-trained models are trained on large corpora and have high accuracy, allowing developers to focus on their specific NLP tasks without worrying about training models from scratch.

Tokenization and Sentence Segmentation

Tokenization is a crucial step in NLP that breaks down text into individual words or subwords. spaCy's tokenization algorithms are highly efficient and language-specific, allowing for accurate and customizable tokenization.

spaCy can also automatically segment text into sentences, making it easy to work with text data at a granular level.

Entity Recognition

Named Entity Recognition (NER) is the task of identifying and classifying named entities such as persons, organizations, locations, dates, and more.

spaCy's NER capabilities are exceptional, providing out-of-the-box support for multiple languages. It allows developers to train custom NER models using their own labeled data, enabling domain-specific entity recognition.

Dependency Parsing

Dependency parsing involves analyzing the grammatical structure of a sentence by determining the relationships between words.

spaCy's dependency parsing is based on efficient algorithms and achieves high accuracy. It provides a rich set of syntactic annotations, including the head of each word, the dependency label, and the subtree structure.

This information is invaluable for tasks like information extraction, question answering, and sentiment analysis.

Customization and Extensibility

One of spaCy's major strengths is its flexibility and extensibility. Developers can easily customize and fine-tune spaCy's models to adapt to specific domains or improve performance on specific tasks.

The library also provides a straightforward API for adding custom components, such as new tokenizers, entity recognizers, or syntactic parsers, making it a versatile tool for research and development.

Performance and Scalability

spaCy is known for its exceptional performance and scalability. The library is implemented in Cython, a programming language that compiles Python-like code into highly efficient C/C++ modules. This allows spaCy to process text data blazingly fast, making it suitable for large-scale NLP applications and real-time systems.

Named Entity Recognition Example in spaCy

Let's try to implement NER using spaCy.

I'll be using Google Colab. Google Colab is a hosted Jupyter Notebook service that requires no setup to use and provides free access to computing resources, including GPUs and TPUs.

You can use Kaggle instead or run it on your own computer if you'd like. Since spaCy is a pre-trained model, it does not required much computing power to get started.

But I'd advise that you setup Anaconda on your machine if you're working on Machine learning problems.

Navigate to https://colab.research.google.com and click on the "New Notebook" button.

Google Colab Console

On the header, enter a name of your file. Ensure your file name ends with .pynb extension.

Change file name and create a code block

Click on the "+ Code" button to create a code block.

By default, Google Colab is packed with some machine tools and Python libraries pre-installed. So, we don't have to worry about installations and getting our development environment ready.

But it doesn't come with the spacy library.

Run the following command inside the code block to install the spacy library.

!pip install -U spacy

Install spacy library

Choose whichever option you want and proceed. The major difference between each one of them is the amount of data it has been trained with.

• Small – en_core_web_sm
• Medium – en_core_web_md
• Large – en_core_web_lg
• Transformer – en_core_web_trf

Our next step is to download one of these models. Add a code block and choose any of the ones from the above list and run the following command. I'll be downloading the large model.

!python -m spacy download en_core_web_lg

Download pre-trained model

Add a code block and run the following command to load the model.

import spacy
nlp = spacy.load("en_core_web_lg")

Alright. We're all set.

Let's try to split entities from a sentence. Add a code block and run the following block of code:

doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)

In the above code, we're asking the spaCy model to find the entities from the sentence "Apple is looking at buying U.K. startup for $1 billion".

We're then iterating through each entity, and displaying the entity, start and end characters index in the sentence, and the entity label.

You should be seeing the following output:

Named Entity Recognition example 1 from spaCy

The above output describes that "Apple" is an entity and it is present from index 0 to index 5 in the given sentence and it is an Organisation (ORG).

If you're confused about the index, remember that it starts from 0. The first 5 characters in our given input text is "Apple". So, it is from 0 to 5.

Similarly it figures out "U.K." as an entity and describes it as a Geopolitical entity (GPE). It labels "$1 billion" as a Money (MONEY) entity.

Let's try a different sentence this time.

"Prime Minister of India Narendra Modi met US President Joe Biden at Washington DC".

Let's see what are the entities it finds out. Add a code block and run the following code:

doc = nlp("Prime Minister of India Narendra Modi met US President Joe Biden at Washington DC")

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)

Named Entity Recognition example 2 from spaCy

That's awesome, isn't it?

It has identified "India", "US", and "Washington DC" as Geopolitical entities (GPE). It has also identified "Narendra Modi" and "Joe Biden" as Person entities (PERSON).

Try to input different sentences and play around with it. I'm sure you'll be amazed at its capabilities in identifying entities.

Conclusion

In this tutorial, we learnt about NLP with a simple implementation using the spaCy library.

Natural Language Processing holds immense potential to transform the way we interact with machines and analyze vast amounts of textual data. spaCy has become a go-to library for many NLP practitioners due to its powerful features, ease of use, and exceptional performance.

If you wish to learn more about NLP/Machine Learning, subscribe to my email newsletter (https://5minslearn.gogosoon.com/) and follow me on social media.

Topic Modeling answers the question: "Given a text corpus of many documents, can we find the abstract topics that the text is talking about?"

In this tutorial, you’ll:

• Learn about two powerful matrix factorization techniques - Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF)
• Use them to find topics in a collection of documents

By the end of this tutorial, you'll be able to build your own topic models to find topics in any piece of text.📚📑

Let's get started.

Table of Contents

1. What is Topic Modeling?
2. TF-IDF Score Equation
3. Topic Modeling Using Singular Value Decomposition (SVD)
4. What is Truncated SVD or k-SVD?
5. Topic Modeling Using Non-Negative Matrix Factorization (NMF)
6. 7 Steps to Use SVD for Topic Modeling
7. How to Visualize Topics as Word Clouds
8. How to Use NMF for Topic Modeling
9. SVD vs NMF – An Overview of the Differences

What is Topic Modeling?

Let's start by understanding what topic modeling is.

Suppose you're given a large text corpus containing several documents. You'd like to know the key topics that reside in the given collection of documents without reading through each document.

Topic Modeling helps you distill the information in the large text corpus into a certain number of topics. Topics are groups of words that are similar in context and are indicative of the information in the collection of documents.

The general structure of the Document-Term Matrix for a text corpus containing M documents, and N terms in all, is shown below:

Structure of the Document-Term Matrix

Let's parse the matrix representation:

• D1, D2, ..., DM are the M documents.
• T1, T2, ..., TN are the N terms

To populate the Document-Term Matrix, let’s use the widely-used metric—the TF-IDF Score.

TF-IDF Score Equation

The TF-IDF score is given by the following equation:

where,

• TF_ij is the number of times the term Tj occurs in the document Di.
• dfj is the number of documents containing the term Tj

A term that occurs frequently in a particular document, and rarely across the entire corpus has a higher IDF score.

I hope you’ve now gained a cursory understanding of the DTM and the TF-IDF score. Let’s now go over the matrix factorization techniques.

Topic Modeling Using Singular Value Decomposition (SVD)

The use of Singular Value Decomposition (SVD) for topic modeling is explained in the figure below:

Singular Value Decomposition on the the Document-Term Matrix D gives the following three matrices:

• The left singular vector matrix U. This matrix is obtained by the eigen decomposition of the Gram matrix D.D_T—also called the document similarity matrix. The i,j-th entry of the document similarity matrix signifies how similar document i is to document j.
• The matrix of singular values S, which (values) signify the relative importance of topics.
• The right singular vector matrix V_T, which is also called the term topic matrix. The topics in the text reside along the rows of this matrix.

If you'd like to refresh the concept of eigen decomposition, here's an excellent tutorial by Grant Sanderson from 3Blue1Brown. It explains eigenvectors and eigenvalues visually.

Embedded content

It's totally fine if you find the working of SVD a bit difficult to understand. 🙂 For now, you may think of SVD as a black box that operates on your Document-Term Matrix (DTM) and yields 3 matrices, U, S, and V_T. And the topics reside along the rows of the matrix V_T.

Note: SVD is also called Latent Semantic Indexing (LSI).

What is Truncated SVD or k-SVD?

Suppose you have a text corpus of 150 documents. Would you prefer skimming through 150 different topics that describe the corpus, or would you be happy reading through 10 topics that can convey the content of the corpus?

Well, it's often helpful to fix a small number of topics that best convey the content of the text. And this is what motivates k-SVD.

As matrix multiplication requires a lot of computation, it's preferred to choose the k largest singular values, and the topics corresponding to them. The working of k-SVD is illustrated below:

Topic Modeling Using Non-Negative Matrix Factorization (NMF)

Non-negative Matrix Factorization (NMF) works as shown below:

Non-negative Matrix Factorization acts on the Document-Term Matrix and yields the following:

• The matrix W which is called the document-topic matrix. This matrix shows the distribution of the topics across the documents in the corpus.
• The matrix H which is also called the term-topic matrix. This matrix captures the significance of terms across the topics.

NMF is easier to interpret as all the elements of the matrices W and H are now non-negative. So a higher score corresponds to greater relevance.

But how do we get matrices W and H?

NMF is a non-exact matrix factorization technique. This means that you cannot multiply W and H to get back the original document-term matrix V.

The matrices W and H are initialized randomly. And the algorithm is run iteratively until we find a W and H that minimize the cost function.

The cost function is the Frobenius norm of the matrix V - W.H, as shown below:

The Frobenius norm of a matrix A with m rows and n columns is given by the following equation:

7 Steps to Use SVD for Topic Modeling

1️⃣ To use SVD to get topics, let's first get a text corpus. The following code cell contains a piece of text on computer programming.

text=["Computer programming is the process of designing and building an executable computer program to accomplish a specific computing result or to perform a specific task.",

      "Programming involves tasks such as: analysis, generating algorithms, profiling algorithms' accuracy and resource consumption, and the implementation of algorithms in a chosen programming language (commonly referred to as coding).",

      "The source program is written in one or more languages that are intelligible to programmers, rather than machine code, which is directly executed by the central processing unit.",

      "The purpose of programming is to find a sequence of instructions that will automate the performance of a task (which can be as complex as an operating system) on a computer, often for solving a given problem.",

      "Proficient programming thus often requires expertise in several different subjects, including knowledge of the application domain, specialized algorithms, and formal logic.",

      "Tasks accompanying and related to programming include: testing, debugging, source code maintenance, implementation of build systems, and management of derived artifacts, such as the machine code of computer programs.",

      "These might be considered part of the programming process, but often the term software development is used for this larger process with the term programming, implementation, or coding reserved for the actual writing of code.",

      "Software engineering combines engineering techniques with software development practices.",

    "Reverse engineering is a related process used by designers, analysts and programmers to understand and re-create/re-implement"]

The text for which you need to find topics is now ready.

2️⃣ The next step is to import the TfidfVectorizer class from scikit-learn's feature extraction module for text data:

from sklearn.feature_extraction.text import TfidfVectorizer

You'll use the TfidfVectorizer class to get the DTM populated with the TF-IDF scores for the text corpus.

3️⃣ To use Truncated SVD (k-SVD) discussed earlier, you need to import the TruncatedSVD class from scikit-learn's decomposition module:

from sklearn.decomposition import TruncatedSVD

▶ Now that you've imported all the necessary modules, it's time to start your quest for topics in the text.

4️⃣ In this step, you'll instantiate a Tfidfvectorizer object. Let's call it vectorizer.

vectorizer = TfidfVectorizer(stop_words='english',smooth_idf=True) 
# under the hood - lowercasing,removing special chars,removing stop words
input_matrix = vectorizer.fit_transform(text).todense()

So far, you've:

☑ collected the text,
☑ imported the necessary modules, and
☑ obtained the input DTM.

Now you'll proceed with using SVD to obtain topics.

5️⃣ You'll now use the TruncatedSVD class that you imported in step 3️⃣.

svd_modeling= TruncatedSVD(n_components=4, algorithm='randomized', n_iter=100, random_state=122)
svd_modeling.fit(input_matrix)
components=svd_modeling.components_
vocab = vectorizer.get_feature_names()

6️⃣ Let’s write a function that gets the topics for us.

topic_word_list = []
def get_topics(components): 
  for i, comp in enumerate(components):
    terms_comp = zip(vocab,comp)
  sorted_terms = sorted(terms_comp, key= lambda x:x[1], reverse=True)[:7]
     topic=" "
     for t in sorted_terms:
      topic= topic + ' ' + t[0]
     topic_word_list.append(topic)
     print(topic_word_list)
  return topic_word_list
get_topics(components)

7️⃣ And it's time to view the topics, and see if they make sense. When you call the get_topics() function with the components obtained from SVD as the argument, you'll get a list of topics, and the top words in each of those topics.

Topic 1: 
  code programming process software term computer engineering

Topic 2: 
  engineering software development combines practices techniques used

Topic 3: 
  code machine source central directly executed intelligible

Topic 4: 
  computer specific task automate complex given instructions

And you have your topics in just 7 steps. Do the topics look good?

How to Visualize Topics as Word Clouds

In the previous section, you printed out the topics, and made sense of the topics using the top words in each topic.

Another popular visualization method for topics is the word cloud. In a word cloud, the terms in a particular topic are displayed in terms of their relative significance. The most important word has the largest font size, and so on.

!pip install wordcloud
from wordcloud import WordCloud
import matplotlib.pyplot as plt
for i in range(4):
  wc = WordCloud(width=1000, height=600, margin=3,  prefer_horizontal=0.7,scale=1,background_color='black', relative_scaling=0).generate(topic_word_list[i])
  plt.imshow(wc)
  plt.title(f"Topic{i+1}")
  plt.axis("off")
  plt.show()

The word clouds for topics 1 through 4 are shown in the image grid below:

Topic Clouds from SVD

As you can see, the font-size of words indicate their relative importance in a topic. These word clouds are also called topic clouds.

How to Use NMF for Topic Modeling

In this section, you'll run through the same steps as in SVD. You need to first import the NMF class from scikit-learn's decomposition module.

from sklearn.decomposition import NMF
NMF_model = NMF(n_components=4, random_state=1)
W = NMF_model.fit_transform(input_matrix)
H = NMF_model.components_

And then you may call the get_topics() function on the matrix H to get the topics.

Topic 1: 
  code machine source central directly executed intelligible

Topic 2: 
  engineering software process development used term combines

Topic 3: 
  algorithms programming application different domain expertise formal

Topic 4: 
  computer specific task programming automate complex given

Topic Clouds from NMF

For the given piece of text, you can see that both SVD and NMF give similar topic clouds.

SVD vs NMF – An Overview of the Differences

Now, let's put together the differences between these two matrix factorization techniques for topic modeling.

• SVD is an exact matrix factorization technique – you can reconstruct the input DTM from the resultant matrices.
• If you choose to use k-SVD, it's the best possible k-rank approximation to the input DTM.
• Though NMF is a non-exact approximation to the input DTM, it's known to capture more diverse topics than SVD.

Wrapping Up

I hope you enjoyed this tutorial. As a next step, you may spin up your own Colab notebook using the code cells from this tutorial. You only have to plug in the piece of text that you'd like to find topics for, and you'd have your topics and word clouds ready!

Thank you for reading, and happy coding!

References and Further Reading on Topic Modeling

• A Code-First Approach to Natural Language Processing by fast.ai
• Computational Linear Algebra by fast.ai

Cover Image: Photo by Brett Jordan on Unsplash

In machine learning, you need to have a large amount of data in order to achieve strong model performance.

Using a method known as data augmentation, you can create more data for your machine learning project. Data augmentation is a collection of techniques that manage the process of automatically generating high-quality data on top of existing data.

In computer vision applications, augmenting approaches are quite common. If you are working on a computer vision project (like image classification), for instance, you can apply dozens of techniques to each image: shift, modify color intensities, scale, rotate, crop, and more.

If you have a tiny dataset for your ML project or wish to reduce overfitting in your machine learning models, it may be a good idea to apply data augmentation approaches.

“We don’t have better algorithms. We just have more data.”- Peter Norvig

In the field of Natural Language Processing (NLP), the tremendous complexity of language makes it difficult to augment the text. The process of augmenting text data is more challenging and not as straightforward as you might expect.

In this article, you will learn how to use a library called TextAttack to improve data for natural language processing.

What is TextAttack?

TextAttack is a Python framework that was built by the QData team for the purpose of conducting adversarial attacks, adversarial training, and data augmentation in natural language processing.

TextAttack has components that can be utilized independently for a variety of basic natural language processing tasks, including sentence encoding, grammar checking, and word substitution.

TextAttack excels in performing the following three functions:

1. Adversarial attacks (Python: **textattack.Attack**, Bash: **textattack attack**).
2. Data augmentation (Python: **textattack.augmentation.Augmenter**, Bash: **textattack augment**).
3. Model training (Python: **textattack.Trainer**, Bash: **textattack train**).

For this article, we will focus on how to use the TextAttack library for data augmentation.

How to Install TexAttack

To use this library, make sure you have Python 3.6 or above in your environment.

Run the following command to install textAttack:

pip install textattack

Note: Once you have installed TexAttack, you can run it via the Python module or via the command line.

Data Augmentation Techniques for Text Data

The TextAttack library has various augmentation techniques that you can use in your NLP project to add more text data.

Here are some of the techniques that you can apply:

CharSwapAugmenter technique

This technique augments words by swapping characters out for other characters.

from textattack.augmentation import CharSwapAugmenter

text = "I have enjoyed watching that movie, it was amazing."

charswap_aug = CharSwapAugmenter()

charswap_aug.augment(text)

[‘I have enjoyed watching that omvie, it was amazing.’]

The Augmenter has swapped the word “movie” for “omvie”.

DeletionAugmenter technique

This one augments the text by deleting some parts of the text to make new text.

from textattack.augmentation import DeletionAugmenter

text = "I have enjoyed watching that movie, it was amazing."

deletion_aug = DeletionAugmenter()

deletion_aug.augment(text)

[‘I have watching that, it was amazing.’]

This method has removed the word “enjoyed” to create a new augmented text.

EasyDataAugmenter technique

This augments the text with a combination of different methods, such as:

• Randomly swapping the positions of the words in the sentence.
• Randomly removing words from the sentence.
• Randomly inserting a random synonym of a random word at a random location.
• Randomly replacing words with their synonyms.

from textattack.augmentation import EasyDataAugmenter

text = "I was billed twice for the service and this is the second time it has happened"

eda_aug = EasyDataAugmenter()

eda_aug.augment(text)

[‘I was billed twice for the service and this is the second time it has happen’, ‘I was billed twice for the one service and this is the second time it has happened’, ‘I billed twice for the service and this is the second time it has happened’,
‘I was billed twice for the this and service is the second time it has happened’]

As you can see from the augmented texts, it shows different results based on the methods applied. For example in the first augmented text, the last word has been modified from “happened” to “happen”.

WordNetAugmenter technique

This technique can augment the text by replacing it with synonyms from the WordNet thesaurus.

from textattack.augmentation import WordNetAugmenter

text = "I was billed twice for the service and this is the second time it has happened"

wordnet_aug = WordNetAugmenter()

wordnet_aug.augment(text)

[‘I was billed twice for the service and this is the second time it has pass’]

This method has changed the word “happened” to “pass” in order to create a new augmented text.

How to Create Your Own Augmenter

Importing transformations and constraints from textattack.transformations and textattack.constraints allows you to build your own augmenter from the ground up.

The following is an illustration of the use of the WordSwapRandomCharacterDeletion algorithm to produce augmentations of a string:

from textattack.transformations import WordSwapRandomCharacterDeletion
from textattack.transformations import CompositeTransformation
from textattack.augmentation import Augmenter

my_transformation = CompositeTransformation([WordSwapRandomCharacterDeletion()])
augmenter = Augmenter(transformation=my_transformation, transformations_per_example=3)

text = 'Siri became confused when we reused to follow her directions.'

augmenter.augment(text)

[‘Siri became cnfused when we reused to follow her directions.’, ‘Siri became confused when e reused to follow her directions.’, ‘Siri became confused when we reused to follow hr directions.’]

The output shows different augmented texts after implementing the WordSwapRandomCharacterDeletion method. For example, in the first augmented text, the method randomly removes the character “o” in the word “confused”.

Conclusion

In this article, you have learned the significance of data augmentation for your Machine Learning projects. You've also learned how to execute data augmentation for textual data using the TextAttack library.

To the best of my knowledge, these techniques are the most effective approaches available to do the task for your NLP project. Hopefully they’ll be of use to you in your work.

You can also try to use other available augmentation techniques from the TextAttack library such as:

• EmbeddingAugmenter
• CheckListAugmenter
• CLAREAugmenter

If you learned something new or enjoyed reading this article, please share it so that others can see it. Until then, see you in the next post!

You can also find me on Twitter @Davis_McDavid.

And you can read more articles like this here.

Tokenization is often regarded as a subfield of NLP but it has its own story of evolution. And now it underpins many state-of-the-art NLP models.

This post is all about training tokenizers from scratch by leveraging Hugging Face’s tokenizers package.

Before we get to the fun part of training and comparing the different tokenizers, I want to give you a brief summary of the key differences between the algorithms.

The main difference lies in the choice of character pairs to merge and the merging policy that each of these algorithms uses to generate the final set of tokens.

BPE Algorithm – a Frequency-based Model

Byte Pair Encoding uses the frequency of subword patterns to shortlist them for merging.

The drawback of using frequency as the driving factor is that you can end up having ambiguous final encodings that might not be useful for the new input text.

But it still has the scope of improvement in terms of generating unambiguous tokens.

Unigram Algorithm – a Probability-based Model

Next we have the Unigram model that approaches solving the merging problem by calculating the likelihood of each subword combination rather than picking the most frequent pattern.

It calculates the probability of every subword token and then drops it based on a loss function that is explained in this research paper.

Based on a certain threshold of the loss value, you can then trigger the model to drop the bottom 20-30% of the subword tokens.

Unigram is a completely probabilistic algorithm that chooses both the pairs of characters and the final decision to merge (or not) in each iteration based on probability.

WordPiece Algorithm

With the release of BERT in 2018, there came a new subword tokenization algorithm called WordPiece which can be considered an intermediary of BPE and Unigram algorithms.

WordPiece is also a greedy algorithm that leverages likelihood instead of count frequency to merge the best pair in each iteration but the choice of characters to pair is based on count frequency.

So, it is similar to BPE in terms of choosing characters to pair and similar to Unigram in terms of choosing the best pair to merge.

With the algorithmic differences covered, I tried to implement each of these algorithms (not from scratch) to compare the output generated by each of them.

How to Train the BPE, Unigram, and WordPiece Algorithms

Now, in order to have an unbiased comparison of outputs, I didn’t want to use pre-trained algorithms as that would bring size, quality, and the content of the dataset into the picture.

One way could be to code these algorithms from scratch using the research papers and then test them out. This is a good approach in order to truly understand the workings of each algorithm but you might end up spending weeks doing that.

I instead used Hugging Face’s tokenizers package that offers the implementation of all of today’s most used tokenizers. It also allowed me to train these models from scratch on my choice of dataset and then tokenize the input string of my own choice.

How to Train the Datasets

I chose two different datasets to train these models. One is a free book from Gutenberg which serves as a small dataset and the other one is the wikitext-103 which is 516M of text.

In the Colab, you can download the datasets first and unzip them (if required):

!wget http://www.gutenberg.org/cache/epub/16457/pg16457.txt

!wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip

!unzip wikitext-103-raw-v1.zip

Import the Required Models and Trainers

Going through the documentation, you’ll find that the main API of the package is the class Tokenizer.

You can then instantiate any tokenizer with your choice of model (BPE/ Unigram/ WordPiece).

Here, I imported the main class, all the models I wanted to test, and their trainers, as I want to train these models from scratch.

## importing the tokenizer and subword BPE trainer
from tokenizers import Tokenizer
from tokenizers.models import BPE, Unigram, WordLevel, WordPiece
from tokenizers.trainers import BpeTrainer, WordLevelTrainer, \
                                WordPieceTrainer, UnigramTrainer

## a pretokenizer to segment the text into words
from tokenizers.pre_tokenizers import Whitespace

How to Automate Training and Tokenization

Since I need to perform somewhat similar processes for three different models, I broke the processes into 3 functions. I’ll only need to call these functions for each model and my job here will be done.

So, what are these functions?

Step 1 - Prepare the tokenizer

Preparing the tokenizer requires us to instantiate the Tokenizer class with a model of our choice. But since we have four models (I added a simple Word-level algorithm as well) to test, we’ll write if/else cases to instantiate the tokenizer with the right model.

To train the instantiated tokenizer on the small and large datasets, we will also need to instantiate a trainer, in our case these would be BpeTrainer, WordLevelTrainer, WordPieceTrainer, and UnigramTrainer.

The instantiation and training will need us to specify some special tokens. These are tokens for unknown words and other special tokens that we’ll need to use later on to add to our vocabulary.

You can also specify other training arguments' vocabulary size or minimum frequency here.

unk_token = "<UNK>"  # token for unknown words
spl_tokens = ["<UNK>", "<SEP>", "<MASK>", "<CLS>"]  # special tokens

def prepare_tokenizer_trainer(alg):
    """
    Prepares the tokenizer and trainer with unknown & special tokens.
    """
    if alg == 'BPE':
        tokenizer = Tokenizer(BPE(unk_token = unk_token))
        trainer = BpeTrainer(special_tokens = spl_tokens)
    elif alg == 'UNI':
        tokenizer = Tokenizer(Unigram())
        trainer = UnigramTrainer(unk_token= unk_token, special_tokens = spl_tokens)
    elif alg == 'WPC':
        tokenizer = Tokenizer(WordPiece(unk_token = unk_token))
        trainer = WordPieceTrainer(special_tokens = spl_tokens)
    else:
        tokenizer = Tokenizer(WordLevel(unk_token = unk_token))
        trainer = WordLevelTrainer(special_tokens = spl_tokens)

    tokenizer.pre_tokenizer = Whitespace()
    return tokenizer, trainer

We’ll also need to add a pre-tokenizer to split our input into words as without a pre-tokenizer, we might get tokens that overlap several words: for instance we could get a "there is" token since those two words often appear next to each other.

Using a pre-tokenizer will ensure no token is bigger than a word returned by the pre-tokenizer.

This function will return the tokenizer and its trainer object which we can use to train the model on a dataset.

Here, we are using the same pre-tokenizer (Whitespace) for all the models. You can choose to test it with others.

Step 2 - Train the tokenizer

After preparing the tokenizers and trainers, we can start the training process.

Here’s a function that will take the file(s) on which we intend to train our tokenizer along with the algorithm identifier.

• ‘WLV’ - Word Level Algorithm

• ‘WPC’ - WordPiece Algorithm

• ‘BPE’ - Byte Pair Encoding

• ‘UNI’ - Unigram

def train_tokenizer(files, alg='WLV'):
    """
    Takes the files and trains the tokenizer.
    """
    tokenizer, trainer = prepare_tokenizer_trainer(alg)
    tokenizer.train(files, trainer) # training the tokenzier
    tokenizer.save("./tokenizer-trained.json")
    tokenizer = Tokenizer.from_file("./tokenizer-trained.json")
    return tokenizer

This is the main function that we’ll need to call for training the tokenizer. It will first prepare the tokenizer and trainer and then start training the tokenizers with the provided files.

After training, it saves the model in a JSON file, loads it from the file, and returns the trained tokenizer to start encoding the new input.

Step 3 - Tokenize the input string

The last step is to start encoding the new input strings and compare the tokens generated by each algorithm.

Here, we’ll be writing a nested for loop to train each model on the smaller dataset first followed by training on the larger dataset and tokenizing the input string as well.

Input string - “This is a deep learning tokenization tutorial. Tokenization is the first step in a deep learning NLP pipeline. We will be comparing the tokens generated by each tokenization model. Excited much?!😍”

small_file = ['pg16457.txt']
large_files = [f"./wikitext-103-raw/wiki.{split}.raw" for split in ["test", "train", "valid"]]

for files in [small_file, large_files]:
    print(f"========Using vocabulary from {files}=======")
    for alg in ['WLV', 'BPE', 'UNI', 'WPC']:
        trained_tokenizer = train_tokenizer(files, alg)
        input_string = "This is a deep learning tokenization tutorial. Tokenization is the first step in a deep learning NLP pipeline. We will be comparing the tokens generated by each tokenization model. Excited much?!😍"
        output = tokenize(input_string, trained_tokenizer)
        tokens_dict[alg] = output.tokens
        print("----", alg, "----")
        print(output.tokens, "->", len(output.tokens))

And here's the output:

Analysis of the output:

Looking at the output, you’ll see the difference in how the tokens were generated which in turn led to different number of tokens generated.

• A simple word level algorithm created 35 tokens no matter which dataset it was trained on.

• BPE algorithm created 55 tokens when trained on a smaller dataset and 47 when trained on a larger dataset. This shows that it was able to merge more pairs of characters when trained on a larger dataset.

• The Unigram model created similar (68 and 67) numbers of tokens with both the datasets. But you can see the difference in the generated tokens:

With larger dataset, merging came closer to generating tokens that are better-suited to encode real-world English language words that we often use.

WordPiece created 52 tokens when trained on a smaller dataset and 48 when trained on a larger dataset. The generated tokens have double ## to denote the use of a token as a prefix/suffix.

All three algorithms generated worse and better subword tokens when trained on a larger dataset.

How to Compare the Tokens

To compare the tokens, I stored the output of each algorithm in a dictionary and I’ll turn it into a dataframe to view the differences in tokens better.

Since the number of tokens generated by each model is different, I’ve added a token to make the data rectangular and fit a dataframe.

is basically nan in the dataframe.

import pandas as pd

max_len = max(len(tokens_dict['UNI']), len(tokens_dict['WPC']), len(tokens_dict['BPE']))
diff_bpe = max_len - len(tokens_dict['BPE'])
diff_wpc = max_len - len(tokens_dict['WPC'])

tokens_dict['BPE'] = tokens_dict['BPE'] + ['<PAD>']*diff_bpe
tokens_dict['WPC'] = tokens_dict['WPC'] + ['<PAD>']*diff_wpc

del tokens_dict['WLV']

df = pd.DataFrame(tokens_dict)
df.head(10)

Here's the output:

You can also look at the difference in tokens using sets:

To check out the code, head over to this Colab notebook.

Closing Thoughts and Next Steps

Based on the kind of tokens generated, WPC does seem to generate subword tokens that are more commonly found in the English language – but don’t hold me to this observation.

These algorithms are slightly different from each other and do a somewhat similar job of developing a decent NLP model. But much of the performance depends on the use case of your language model, the vocabulary size, speed and other factors.

This concludes our examination of tokenization algorithms. The next step to deep dive into this is to understand what embeddings are, how tokenization plays a vital role in creating these embeddings, and how they affect a model’s performance.

A further advancement to these algorithms is the SentencePiece algorithm which is a wholesome approach to the whole tokenization problem. But much of this problem is alleviated by HuggingFace, and even better – they have all the algorithms implemented in a single GitHub repo.

References and Notes

If you have questions about my analysis or any of my work in this post, I highly encourage you to check out these resources for a precise understanding of the workings of each algorithm:

1. Subword regularization: Improving Neural Network Translation Models with Multiple Subword Candidates by Taku Kudo

2. Neural Machine Translation of Rare Words with Subword Units - Research paper that discusses different segmentation techniques based BPE compression algorithm.

3. Hugging Face’s tokenizer package.

Connect with me

If you’re looking to get started in the field of data science or ML, check out my course on Foundations of Data Science & ML.

If you would like to see more of such content and you are not a subscriber, consider subscribing to my newsletter.

Have something to add or suggest, you can reach out to me via:

• YouTube

• Twitter

• LinkedIn

These companies have released state-of-the-art language models like BERT and GPT-2 and GPT-3. And GitHub Copilot and OpenAI codex are among some of the popular applications that are in the news lately.

As someone who has had very limited exposure to NLP, I decided to take it up as an area of research so I can learn more about it. My next few articles and videos will focus on sharing what I learn after dissecting some important components of NLP.

Main Components of NLP

NLP systems have three main components that help machines understand natural language:

1. Tokenization

2. Embedding

3. Model architectures

Top Deep Learning models like BERT, GPT-2, and GPT-3 all share the same components but with different architectures that distinguish one model from another.

In this article (and the notebook that accompanies it), we are going to focus on the basics of the first component of an NLP pipeline which is tokenization. It's an often overlooked concept, but it is a field of research in itself.

We have come so far from the traditional NLTK tokenization process. And though we have state-of-the-art algorithms for tokenization, it's always a good practice to understand its evolution and how we got to where we are now.

So, here's what we'll cover:

• What is tokenization?

• Why do we need a tokenizer?

• Types of tokenization – Word, Character, and Subword.

• Byte Pair Encoding Algorithm - a version of which is used by most NLP models these days.

The next part of this tutorial will dive into more advanced (or enhanced versions of Byte Pair Encoding) algorithms:

• Unigram Algorithm

• WordPiece – BERT transformer

• SentencePiece – End-to-End tokenizer system

What is Tokenization?

Tokenization is the process of representing raw text in smaller units called tokens. These tokens can then be mapped with numbers to further feed to an NLP model.

Here's an overly simplified example of what a tokenizer does:

## read the text and enumerate the tokens in the text
text = open('example.txt', 'r').read(). # read a text file

words = text.split(" ") # split the text on spaces

tokens = {v: k for k, v in enumerate(words)} # generate a word to index mapping

Here, we have simply mapped every word in the text to a numerical index. This is, of course, a very simple example and we have not considered grammar, punctuation, or compound words (like test, test-ify, test-ing, and so on).

So we need a more technical and accurate definition of tokenization for our work here. To take into account all punctuation and every related word, we need to start working at the character level.

There are multiple applications of tokenization. One of the use cases comes from compiler design where you might need to parse computer programs to convert raw characters into keywords of a programming language.

In deep learning, tokenization is the process of converting a sequence of characters into a sequence of tokens which further needs to be converted into a sequence of numerical vectors that can be processed by a neural network.

Why do we need a Tokenizer?

The need for a tokenizer came from the question "How can we make machines read?"

A common way of processing textual data is to define a set of rules in a dictionary and then look up that fixed dictionary of rules. But this method can only go so far, and we want machines to learn these rules from the text that it reads.

Now, machines don't know any language, nor do they understand sound or phonetics. They need to be taught from scratch and in such a way that they can read any language that's out there.

Quite a task, right?

Humans learn a language by connecting sound to the meaning and then we learn to read and write in that language. Machines can't do that, so they need to be given the most basic units of text to start processing the text.

That's where tokenization comes into play. It breaks down the text into smaller units called "tokens".

And there are different ways of tokenizing text which is what we'll learn now.

Different ways to tokenize text

To make the deep learning model learn from the text, we need a two-step process:

1. Tokenize – decide the algorithm we'll use to generate the tokens.

2. Encode the tokens to vectors

Word-based tokenization

As the first step suggests, we need to decide how to convert text into small tokens. A simple and straightforward method that most of us would propose is to use word-based tokens, splitting the text by spaces.

Problems with Word tokenizer

There's a high risk of missing words in the training data. With word tokens, your model won't recognize the variants of words that were not part of the data on which the model was trained.

So, if your model has seen foot and ball in the training data but the final text has football, the model won't be able to recognize the word and it will be treated with an <UNK> token.

Similarly, punctuation poses another problem. For example, let or let's will need individual tokens which is an inefficient solution. This will require a huge vocabulary to make sure you've thought of every variant of the word.

Even if you add a lemmatizer to solve this problem, you're adding an extra step in your processing pipeline.

It's also tough to handle slang and abbreviations. We use lots of slang and abbreviations in text these days, such as "FOMO", "LOL", "tl;dr" and so on. What do we do for these words?

Finally, what if the language doesn't use spaces for segmentation? For a language like Chinese, which doesn't use spaces for word separation, this tokenizer will fail completely.

After encountering these problems, researchers looked into another approach which involved tokenizing all the characters.

Character-based tokenization

To resolve the problems associated with word-based tokenization, data scientists tried an alternative approach of character-by-character tokenization.

This did solve the problem of missing words, as now we are dealing with characters that can be encoded using ASCII or Unicode. Now it could generate embedding for any word.

Every character, whether it was a space, apostrophe, colon, or whatever can now be assigned a symbol to generate a sequence of vectors.

But this approach had its own cons.

Problems with character-based models

First, this approach requires more computing resources. Character-based models will treat each character as a token. And more tokens means more input computations to process each token which in turn requires more compute resources.

For example, for a 5-word long sentence, you may need to process 30 tokens instead of 5 word-based tokens.

Also, it narrows down the number of NLP tasks and applications. With long sequences of characters, you can only use a certain type of neural network architecture.

This limits the type of NLP tasks we can perform. For applications like entity recognition or text classification, character-based encoding might turn out to be an inefficient approach.

Finally, there's a risk of learning incorrect semantics. Working with characters could generate incorrect spellings of words. Also, with no inherent meaning, learning with characters is like learning with no meaningful semantics.

What's fascinating is that for such a seemingly simple task, multiple algorithms have been written to find the optimal tokenization policy.

After understanding the pros and cons of these tokenization methods, it makes sense to look for an approach that offers a middle route. We'll want one that preserves the semantics with limited vocabulary that can generate all the words in the text on merging.

Subword Tokenization

With character-based models, we risk losing the semantic features of the word. And with word-based tokenization, we need a very large vocabulary to encompass all the possible variations of every word.

So, the goal was to develop an algorithm that could:

1. Retain the semantic features of the token, that is information per token.

2. Tokenize without demanding a very large vocabulary with a finite set of words.

To solve this problem, you can think of breaking down the words based on a set of prefixes and suffixes. For example, we can write a rule-based system to identify subwords like "##s", "##ing", "##ify", "un##" and so on, where the position of the double hash denotes prefix and suffixes.

So, a word like "unhappily" is tokenized using subwords like "un##", "happ", and "##ily".

The model only learns relatively few subwords and then puts them together to create other words. This solves the problems of memory requirement and effort required to create a large vocabulary.

Problems with the subword tokenization algorithm:

First of all, some of the subwords that are created as per the defined rules may never appear in your text to tokenize and may end up occupying extra memory.

Also, for every language, we'll need to define a different set of rules to create subwords.

To alleviate this problem, in practice, most modern tokenizers have a training phase that identifies the recurring text in the input corpus and creates new subword tokens. For rare patterns, we stick to word-based tokens.

Another important factor that plays a vital role in this process is the size of the vocabulary that the user sets. A large vocabulary size allows for more common words to be tokenized, whereas smaller vocabulary requires more subwords to be created to create every word in the text without using the <UNK> token.

Striking the right balance for your application is key here.

Byte Pair Encoding (BPE) Algorithm

BPE was originally a data compression algorithm that you use to find the best way to represent data by identifying the common byte pairs. We now use it in NLP to find the best representation of text using the smallest number of tokens.

Here's how it works:

1. Add an identifier (</w>) at the end of each word to identify the end of a word and then calculate the word frequency in the text.

2. Split the word into characters and then calculate the character frequency.

3. From the character tokens, for a predefined number of iterations, count the frequency of the consecutive byte pairs and merge the most frequently occurring byte pairing.

4. Keep iterating until you have reached the iteration limit (set by you) or until you have reached the token limit.

Let's go through each step (in the code) for some sample text. For coding this, I have taken help from Lei Mao's very minimalistic blog on BPE. I encourage you to check it out!

Step 1: Add word identifiers and calculate word frequency

Here's our sample text:

"There is an 80% chance of rainfall today. We are pretty sure it is going to rain."

## define the text first

text = "There is an 80% chance of rainfall today. We are pretty sure it is going to rain."

## get the word frequency and add the end of word (</w>) token ## at the end of each word

words = text.strip().split(" ")

print(f"Vocabulary size: {len(words)}")

Step 2: Split the word into characters and then calculate the character frequency

char_freq_dict = collections.defaultdict(int)
for word, freq in word_freq_dict.items():
    chars = word.split()
    for char in chars:
        char_freq_dict[char] += freq

char_freq_dict

Step 3: Merge the most frequently occurring consecutive byte pairings

import re

## create all possible consecutive pairs
pairs = collections.defaultdict(int)
for word, freq in word_freq_dict.items():
    chars = word.split()
    for i in range(len(chars)-1):
        pairs[chars[i], chars[i+1]] += freq

Step 4 - Iterate n times to find the best (in terms of frequency) pairs to encode and then concatenate them to find the subwords

It is better at this point to structure our code into functions. This means that we need to perform the following steps:

1. Find the most frequently occurring byte pairs in each iteration.

2. Merge these tokens.

3. Recalculate the character tokens' frequency with the new pair encoding added.

4. Keep doing this until there are no more pairs or you reach the end of the for a loop.

For detailed code, you should check out my Colab notebook.

Here’s a trimmed output of those 4 steps:

So as we iterate with each best pair, we merge (concatenate) the pair. You can see that as we recalculate the frequency, the original character token frequency is reduced and the new paired token frequency pops up in the token dictionary.

If you look at the number of tokens created, it first increases because we create new pairings – but the number starts to decrease after a number of iterations.

Here, we started with 25 tokens, went up to 31 tokens in the 14th iteration, and then came down to 16 tokens in the 50th iteration. Interesting, right?

How to improve the BPE algorithm

BPE algorithm is a greedy algorithm, which means that it tries to find the best pair in each iteration. And there are some limitations to this greedy approach.

So of course there are pros and cons of the BPE algorithm, too.

The final tokens will vary depending upon the number of iterations you have run. This also causes another problem: we now can have different tokens for a single text, and thus different embeddings.

To address this issue, multiple solutions have been proposed. But the one that stood out was a unigram language model that added subword regularization (a new method of subword segmentation) training that calculates the probability for each subword token to choose the best option using a loss function. We'll talk more about this in upcoming articles.

Do we use BPE in BERTs or GPTs?

Models like BERT or GPT-2 use some version of the BPE or the unigram model to tokenize the input text.

BERT included a new algorithm called WordPiece. It is similar to BPE, but has an added layer of likelihood calculation to decide whether the merged token will make the final cut.

Summary

In this blog, you've learned how a machine starts to make sense of language by breaking down the text into very small units.

Now, there are many ways to break text down and so it becomes important to compare one approach with another.

We started off by understanding tokenization by splitting the English text by spaces – but not every language is written the same way (that is using spaces to denote segmentation). So then we looked at splitting by character to generate character tokens.

The problem with characters was the loss of semantic features from the tokens at the risk of creating incorrect word representations or embeddings.

To get the best of both worlds, we looked at subword tokenization which was more promising. And finally we looked at the BPE algorithm to implement subword tokenization.

We'll look more into the next steps and advanced tokenizers like WordPiece, SentencePiece, and how to work with the HuggingFace tokenizer next week.

References and Notes

My post is actually an accumulation of the following papers and blogs that I encourage you to read:

1. Neural Machine Translation of Rare Words with Subword Units - Research paper that discusses different segmentation techniques based BPE compression algorithm.

2. GitHub repo on Subword NMT(Neural Machine Translation) - supporting code for the above paper.

3. Lei Mao’s blog on Byte Pair Encoding - I used the code in his blog to implement and understand BPE myself.

4. How Machines read - a blog by Cathal Horan.

If you’re looking to start in the field of data science or ML, check out my course on Foundations of Data Science & ML.

If you would like to get all my tutorials/blogs delivered directly to your inbox, consider subscribing to my newsletter here.

Have something to add or suggest, you can reach out to me via:

• YouTube

• Twitter

• LinkedIn