The Raspberry Pi Model B was introduced in 2012 as the first sellable unit, and the company has since released many more models. There are even low-cost models like the Raspberry Pi Zero Series, which is quite small and tailored to embedded systems applications. All the models operate on an operating system called the Raspberry Pi OS, a Linux flavor niched for Raspberry PI Computers.

In this tutorial, we’ll be using the Raspberry Pi 4 Model for a headless setup through a SSH connection using Visual Studio Code (VS Code). The Raspberry Pi 4 Model has a Quad-core ARM Cortex-A72 (64-bit) SoC at 1.5GHz, up to 8GB RAM options, video inputs, Ethernet shield, USB ports, MicroSD card slot for storage, USB-C power input and 40 General Purpose Inputs and Outputs Pins (GPIO). Impressive, right?

You’ll be able to use the Raspberry Pi as a personal computer, for home automation and IoT projects, robotics projects, network applications, educational tools, and Artificial Intelligence projects.

Table of Contents

• Understanding the Headless Setup

• Prerequisites

• Preparing the MicroSD Card

• How to Boot the Raspberry Pi

• Understanding the LED of the Raspberry Pi During Setup

• How to Establish an SSH Connection

• How to Set Up Visual Studio Code for Remote Development

• How to Write and Run the Code Remotely

• Conclusion

Understanding the Headless Setup

Many Raspberry Pi computers are sold with additional peripherals, including a keyboard, mouse, and monitor, that are essential for the Raspberry Pi's setup. A headless setup is the process of configuring the Raspberry Pi or preparing it for use without needing these peripherals. This entails operating the Raspberry Pi through a network protocol like SSH (Secure Shell) or VNC (Virtual Network Computing).

This is really helpful when you don’t need peripherals, as it lets you use your personal computer to connect to the Raspberry Pi without needing to purchase specialized peripherals. It’s also excellent for remote access. This headless setup is also essential for remote monitoring systems, such as surveillance systems with remote camera access, and IoT systems.

Remote development lets you write code and modify your Raspberry Pi and other devices connected to the GPIO pins through a headless configuration via SSH.

SSH guarantees a secure connection for transferring and modifying files, as well as transferring and debugging commands from one computer (your personal computer) to another computer (the Raspberry Pi). It restricts unauthorized access from any other system that aims to intercept the communication channel.

Prerequisites

Here’s what you’ll need to follow along with this tutorial:

Hardware Requirements

1. Raspberry Pi 4 or 5

2. MicroSD Card (8GB or higher recommended)

3. Flash Drive with SD Card Slot or a MicroSD Card Adapter

4. Power Supply (5V 2A/3A)

5. Network Connection (Wi-Fi, Pi, and laptop must be on the same network)

6. Personal Computer (Windows, macOS, Linux)

Software Requirements

1. Raspberry Pi Operating System (Raspberry Pi OS)

2. Visual Studio Code

3. Remote SSH Extension in VS Code

Preparing the MicroSD Card

The Raspberry Pi requires a MicroSD Card that serves as the storage of your the Raspberry Pi OS using Raspberry Pi Imager. The operating system of the Raspberry Pi provides a graphical interface to interact with the Raspberry Pi, store files and datasets, and write commands to get your Raspberry Pi working.

But the Raspberry Pi needs an empty MicroSD Card to install the Raspberry Pi OS in the MicroSD Card. Here are some step by step instructions that’ll show you how to get your MicroSD Card setup before inserting it back into the Raspberry Pi for SSH Connection.

Downloading and flashing Raspberry Pi OS

Insert your MicroSD Card into a flash drive with a SD Card slot

Aside from using a flash drive with an SD Card slot (so as to get the memory card connected to the computer), you can also use a SD Card adapter. Make sure it’s inserted into your computer where you have the Raspberry PI Imager downloaded to flash – that is, transfer the OS into the SD Card.

Download the Raspberry Pi Imager based on your PC’s operating system

This involves clicking the link and selecting your operating system (either MacOS, Windows or Linux operating system). The Raspberry Pi OS comes in these variants for different OSes

Next, install and open the Raspberry Pi imager

Click the Raspberry Pi Imager download, follow all the instructions during the installation process. Once this screen pops up, you’re good to go!

Choose your Raspberry Pi Device and operating system and select Storage

For each of the three configurations, you must select one sequentially. Select a device according to the type of Raspberry Pi you have, and various options will appear. I selected the Raspberry Pi 4, as it is my preferred device. You may choose between the Raspberry Pi 5 and the Raspberry Pi Zero 2 W, depending on your device requirements.

Next, proceed to the operating system – I would recommend choosing the 64-bit version. While many people opt for the legacy version (32-bit), I think the 64-bit version is best. Once you're finished, you can choose a storage option, and your MicroSD should appear. My storage is around 128GB, which is why you can see 125.1GB displayed there in the screenshot below:

Click on “Next” and edit the settings

It is a customary practice to keep your username as "pi", but it’s not required. The goal is to have something simple and easy to remember when setting up your SSH connection. It’s also helpful to make your password simple. I used 'roboticsai'.

Try to avoid using numbers simply to make things easier, because you may not be able to see what you are entering in the terminal. Then, make sure that your wireless LAN and SSID (WIFI or Hotspot name if you're using a phone, as well as the password for your WIFI or Hotspot) is the same network as the one linked to your computer.

Click on “SERVICES” and enable SSH. Then use password authentication for security and Click on “SAVE”.

After you've completed the changes in the General Section, go to the Services section and click the checkbox button “Enable SSH”. Once highlighted, make sure you pick “Use password authentication”, avoid the “RUN SSH-KEYGEN” button at the moment, and then click Save.

Click “YES” to apply the customizations, and the Raspberry Pi OS should get flashed into your SD Card.

Following the previous stage, you will be shown various buttons to apply the adjustments you have made. Pick yes, and the Raspberry Pi OS will be flashed or transferred to your Memory Card. This could take between 10 and 20 minutes to go from transferring to writing or customizing. Hold on and enjoy the process.

After a successful installation into the disk, remove your SD Card.

You will receive a successful popup like the one shown below. This demonstrates that all processes were completed successfully, and the Raspberry Pi OS is now installed.

How to Boot the Raspberry Pi

Eject the MicroSD safely from your computer

Once the installation is successful, eject the MicroSD safely from the computer.

Insert it “upside down” into the MicroSD card slot of your Raspberry Pi

To properly insert the MicroSD card, place it gently into the slot with the back or gold side facing upward. It will protrude slightly once it is inserted. You are good to go!

Connect the USB-C port of your Raspberry Pi to your computer. Give the Raspberry Pi some time to load

Get a USB-C cable and connect one end to your Raspberry Pi's USB-C port and the other to a laptop port. It should light up red, indicating that there is an adequate power source. You may also power your Raspberry Pi directly by plugging into a wall socket.

After a while, the memory card should begin to boot into the Raspberry Pi, and the green LED will blink for a while. In the next section, we’ll talk about the different states of the two LEDs during and after a successful boot.

Understanding the LED Status of the Raspberry Pi During Setup

The below table describes the LED statuses you might see when you power on your Raspberry Pi and the SD Card is in the slot.

How to Establish an SSH Connection

The SSH (Secure Shell) connection is a network protocol that allows two computers to safely communicate without leaking any information. It’s also used for remote command line execution and for file transfers between two computers.

To establish an SSH connection, you’ll have to complete a few steps. Then I’ll explain how to enable SSH using a Visual Studio Code extension

Create a wpa_supplicant.conf.txt in the same folder of your Raspberry Pi SD Card

Insert your MicroSD card back into the computer. Then the files that comprise the Raspberry Pi OS will appear on your computer. Create a new text (.txt) document on your computer, similar to the image below, under the SD Card storage section.

Add the code below, making sure that "ssid" is the name of your Wi-Fi network and "psk" is your network's password.

country=NG # Your 2-digit country code
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
network={
    ssid="Josiah"
    psk="roboticsai"
    key_mgmt=WPA-PSK
}

Save the file on the same SD Card

Once you've finished producing the text file, save it to the SD Card storage, as shown in the image below.

Create a .ssh folder

In your personal computer, create a .ssh folder if it doesn’t exist on your personal computer.

If it exists, the .ssh folder should contain files like id_rsa, known_hosts, and config files. It shouldn’t be empty.

After a successful boot, open your terminal or command line application on your personal computer.

Make sure that the Raspberry Pi is connected to the same network before moving ahead. Once your wifi or mobile hotspot is switched on, make sure it’s the same password as the wpa_supplicant.conf.txt and the settings page while installing the Raspberry Pi.

As long as the SD card is in the Raspberry Pi and there is adequate power supply for at least 2-5 minutes, the Raspberry Pi will get connected to the wifi or your mobile hotspot.

How to Resolve Connection Problems

If there is no connection, reinstall the Raspberry Pi OS Imager on the SD Card again. Then you can also change the network AP Band from 5GHz to 2.5GHz or vice-versa. This can be very tricky.

It should get connected after trying this. Just make sure that the passwords are consistent and that you don’t accidentally have the caps lock key switched on while typing, for example.

To confirm if the Raspberry Pi is connected using the command line interface, use the ping command – it shows the devices connected to the device.

ping raspberrypi.local

After running the above command, you should see an image showing the connection once it’s successful like this:

For establishing an SSH connection using the terminal, run the code below:

ssh pi@raspberrypi.local

This will result in a request for a password. If it shows an error like the image below, it means you have to delete the known_hosts.old and known_hosts if either or both exist in the .ssh folder in your PC. This is because the keys are conflicting with each other. Then re-run the above code ssh pi@raspberrypi.local in your terminal.

After successful entry, type “yes” in the terminal.

Connection Closed should show when the connection is successful.

How to Set Up Visual Studio Code for Remote Development

Download and install Visual Studio Code if you don’t have it already.

Then, click on the VS Code extension and search for Remote - SSH by Microsoft and install it to your machine.

Next, click on the “Remote Explorer” icon that looks like a monitor. Select the SSH config in your C:\Users\{name}\.ssh\config folder.

Make sure the config has this command:

Host raspberrypi.local
    HostName raspberrypi.local
    User pi

Enter your username as raspberrypi.local and input your password – the same as the password during loading Raspbian OS.

After inputting the correct password, it should start downloading the server.

Congratulations! The image below has a blue rectangle button showing SSH:raspberrypi.local which shows a successful SSH Connection through Visual Studio Code. This also means you can start remote development as we discussed earlier in this tutorial.

How to Write and Run the Code Remotely

Create a new file on your VS Code. This way, you’re creating files and writing to them directly. Go to the terminal and type the commands to create a folder and a file:

Create a new file and write in your code

Create a new file and name it led.py on your Visual Studio Code. It should be in the same folder as test-raspberry on the Raspberry Pi remote network through the SSH connection on VSCode.

Once you have your file created, you can write in your code such as blinking LED to a Raspberry Pi, as you can see in the code below:

from gpiozero import LED
from time import sleep

# Set the GPIO pin where the LED is connected
led = LED(17)  # Replace 17 with your GPIO pin number

# Blink the LED in a loop
while True:
    led.on()        # Turn LED on
    sleep(1)        # Wait for 1 second
    led.off()       # Turn LED off
    sleep(1)        # Wait for 1 second

After writing this code in the new file you’ve created, run the code by typing the command below in your terminal:

python led.py

As soon as this command is sent, the LED positive terminal is connected to the GPIO 17 according to the code and the negative terminal is connected to the GND GPIO pin of the Raspberry Pi. The image from Random Nerd Tutorials below shows the GPIO pins and their number to understand the connection. Just note that the connection of the LED is beyond the scope of this tutorial.

The LED should start blinking each second according to the code. With this, you can now control your Raspberry Pi (a tiny computer) with another computer (your personal computer) through an SSH connection on Visual Studio Code.

Conclusion

In this tutorial, you went through the whole process of setting up a headless Raspberry Pi for remote development using VS Code.

This offers a wide range of benefits: there’s no need for external peripherals, it provides remote access from anywhere within your network, and it leverages efficient coding and debugging with VS Code integration.

You can use this to deploy web servers and IoT dashboards, and you can explore with automating processes using Python scripts and GPIO control.

Both of these examples work based on classification tasks, as does the facial recognition feature on your phone.

When building a classification algorithm, real-world data often has a non-linear relationship. And many machine learning classification algorithms struggle with non-linear algorithms. But in this article, we'll be looking at how Support Vector Machine (SVM) kernel functions can help to solve this problem. We’ll go in-depth into a Python implementation of non-linear classification and SVM kernel functions.

Prerequisites

1. Basic Understanding of Machine Learning

2. Linear Algebra Basics

3. Basic Python Programming Skills

4. Understanding of Data Visualization

5. A Google Colab or Jupyter Notebook Account

Table of Contents

1. Overview of the Support Vector Machine (SVM) Technique

2. Fundamentals of SVM

3. SVM Objective Function

4. Understanding Kernel Functions

5. Popular Kernel Functions

6. How to Choose the Right Kernel

7. SVM Kernel Implementation

8. Conclusion

Overview of the Support Vector Machine (SVM) Technique

Support Vector Machine (SVM) is a supervised learning algorithm. It uses a hyperplane that divides features inside a feature space into distinct categories. It’s effective for both classification and regression applications.

By identifying the optimal dividing line or plane that will serve as the decision boundary, SVM seeks to maximize the margin between the various target variables. It’s primarily utilized in classification tasks and is very helpful in ignoring outliers. It categorizes the data points of the features in the dataset into distinct outputs or classes.

SVM seeks to achieve the optimal maximum margin and an ideal or near-perfect separation. There are various applications for SVM, such as image classification, face detection, text classification, image classification, and bioinformatics. SVM is also efficient in linear and non-linear classification problems.

Importance of Kernel methods in SVM

Nonlinear classification is a sort of classification that involves categorizing features that have non-linear, curved, or complex decision boundaries. Decision boundaries are regions of space that separate two different classes.

In linear classification tasks, the region of space between the different classes such as if the email is spam or not can be easily separated with a straight line. But in non-linear relationships, it could have a circular, parabola, or a complex-shape decision boundary.

Non-linear classification tasks have patterns that cannot be discovered by linear models. This is because the features have a non-linear relationship with each other.

SVM as a linear classification algorithm isn’t efficient for a non-linear data. To handle this sort of data, it will require a kernel method, which is the core topic of this article.

A kernel method is a technique used in SVM to transform non-linear data into higher dimensions. For example, if the data has a complex decision boundary in a 2-Dimensional space (as I’ll explain further in the later part of this article), it can be transformed into a 3-Dimensional space. This allows efficient classification just with a linear plane.

The goal of the article is to teach you about SVM kernels and their application to non-linear classification tasks.

Fundamentals of SVM

Linear Classifiers and Margin Maximization

Linear classifiers are classification algorithms that make predictions by using a straight line of best fit as a decision boundary between two or more categories.

Marginal planes are used to determine the support vector in the classification task. Support vectors are the data points in the dataset that are used to separate the different target variable categories – they are data points very close to the decision boundary.

In the image below, the marginal planes are the yellow lines, while the hyperplane is the red line. The hyperplane serves as the line of best fit or decision boundary. The data points that are closest to the marginal plane are the support vectors – the data points encircled in green in the image below.

The marginal plane aims to achieve a maximum margin between its plane and the hyperplane – both having equal distance from hyperplane to achieve the best classification. The hyperplane in the image above shows a perfect linear relationship between feature x1 and feature x2. The support vectors also help to establish the location of the marginal plane.

We have the hard margin and the soft margin, serving as model optimization methodologies for the SVM. The hard margin shows that you cannot find a data point of feature x1 in the same area where there are feature x2 data points and vice versa. It used to describe a perfect classification by the algorithm. The image above gives a representation of a hard margin.

A soft margin shows that the classification is imperfect, because you can find some data points of feature x1 in the same area where we have data points of feature two, which could be caused by outliers. The image below gives a representation of soft margin.

SVM Objective Function

For a binary classification, such as a dog or a cat, the dog can be represented as class 1 and cat as -1. This shows that the decision boundary or hyperplane is the determining factor. Any value above the plane is given as 1, and the class below the plane is given as -1.

The mathematical function for the hyperplane is given as:

$$f(x) = \mathbf{w}^T\mathbf{x} + b$$

$$\begin{array}{l} \text{ The variables used are:} \\ \mathbf{w}: \text{Weight vector (defining the orientation of the hyperplane)} \\ b: \text{Bias term (defining the position of the hyperplane)} \\ \mathbf{x}: \text{Input feature vector} \\ \\ \text{The classification decision is based on the sign of } f(x)\text{:} \\ f(x) > 0: \text{Class 1} \\ f(x) < 0: \text{Class -1} \end{array}$$

Hard Margin SVM

The Hard Margin SVM ensures all the data points are all properly classified without error, ensuring that the data points don’t find themselves in the other part of the hyperplane, and also maximizing the margin. It’s an effective method for a “noise-free” dataset. This is achieved by minimizing an objective function given below:

$$\begin{array}{l} \text{Hard Margin SVM Objective Function:} \ \min_{\mathbf{w},b} \frac{1}{2}\|\mathbf{w}\|^2 \\ \\ \text{Subject to:} \\ y_i(\mathbf{w}^T\mathbf{x}_i + b) \geq 1, \,\, \forall i \\ \\ \text{Where:} \\ y_i: \text{ Class label of the }i\text{-th sample } (+1 \text{ or } -1) \\ \mathbf{x}_i: \text{ Feature vector of the }i\text{-th sample} \end{array}$$

This constraint given above in the objective function ensures that all the data points are not misclassified and the stay outside the margin.

Soft Margin SVM

The Soft Margin SVM is lenient, as it allows some misclassifications. It’s suitable for real-world datasets, which are noisy, and it handles non-linearly separable data. It introduces a slack variable that penalizes incorrect predictions.

$$\begin{array}{l} \text{Objective Function:} \ \min_{\mathbf{w},b,\xi} \frac{1}{2}\|\mathbf{w}\|^2 + C\sum_{i=1}^n \xi_i \\ \\ \text{Subject to:} \\ y_i(\mathbf{w}^T\mathbf{x}_i + b) \geq 1 - \xi_i, \,\, \forall i \\ \xi_i \geq 0, \,\, \forall i \\ \\ \text{Where:} \\ \xi_i: \text{ Slack variables representing the degree of misclassification or} \\ \text{margin violation.} \\ C: \text{ Regularization parameter controlling the trade-off between} \\ \text{margin maximization and error minimization.} \end{array}$$

The hyperparameter C helps to control the penalty for a balance between margin maximization and error minimization. A large C value minimizes the classification errors, but causes a smaller margin. A small C value allows some misclassifications but causes a larger margin.

Nonlinear Classification Problems

Non-linear classification problems include datasets with non-linear patterns that are difficult for linear SVM models to capture. This is a drawback, but SVM kernels can help.

Non-linear classification contains datasets with complicated relationships and linear models like linear regression will not be able to accurately generate predictions or identify trends.

Understanding Kernel Functions

In kernel functions, we transform the dataset used in the classification task into a higher dimensional feature space. This line of action enables the hyperplane (a linear decision boundary) to split the data as linearly separable data.

For example, if a dataset contains three features in a 2D plane, the kernel function converts the data to a 3D plane, making it much simpler to partition the dataset using a basic hyperplane. This technique can be used to capture non-linear relationships in data.

To provide a clearer mental image, consider three distinct feature sets in the 2D plane (x and y). This can be taken to a 3D plane by the kernel machine, where features x1 and feature x2 may be in the x-y plane, which is readily divided by a simple hyperplane, and feature x3 may be in the y-z plane, which is already separated.

The Kernel Trick Explained

Transformation into a higher dimensional space is computationally intensive and is not the best option. But we know the importance of kernel functions in classifying non-linear data. So, what’s the way forward to still achieve the same feat while bypassing the cost of computation? It’s called the kernel trick. The kernel trick explains the “magic power” of the kernel functions.

The kernel trick is the computation of the inner or dot product between the data points in the original dimensional space instead of transforming the data into a higher-dimensional space before doing the computation.

The right side of the equation below shows the dot product of ϕ(x), representing the transformed vector into a higher dimensional space (which is not efficient). It’s the same as a kernel function at the left hand side:

$$K(x_i, x_j) = \phi(x_i) \cdot \phi(x_j)$$

The purpose of the kernel trick is to perform computation based on the data point in its original dimensional space, instead of performing calculations on complex data that might require an infinite number of dimensions.

Mathematical Implementation of the Kernel Trick

Suppose we have two classes of data that are non-linear in the 2D space representing the original feature space. No straight line can separate these points because they lie diagonally across the origin.

$$\begin{array}{l} \textbf{Mapping Without Kernel Trick: }\\ \\ \begin{align*} \textbf{The 2D data is given as: } \\ & \mathbf{x}_1 = (1,1), & y_1 = +1 \\ & \mathbf{x}_2 = (-1,-1), & y_2 = -1 \end{align*} \\ \\ \textbf{Let's use a mapping function: } \\ \\ \phi(x, y) = (x^2, \sqrt{2}xy, y^2)\ \\ \\ Mapping\ \mathbf{x}_1 \ and \ \ \mathbf{x}_2: \\ \\ \begin{array}{l} - \ \phi(\mathbf{x}_1) = (1^2, \sqrt{2}(1)(1), 1^2) = (1, \sqrt{2}, 1) \\ \\ -\ \phi(\mathbf{x}_2) = ((-1)^2, \sqrt{2}(-1)(-1), (-1)^2) = (1, \sqrt{2}, 1) \end{array} \\ \\ \\ \textbf{Dot Product in Higher-Dimensional Space:} \\ \\ \phi(\mathbf{x}_1) \cdot \phi(\mathbf{x}_2) = (1)(1) + (\sqrt{2})(\sqrt{2}) + (1)(1) = 1 + 2 + 1 = 4 \\ \\ \\ \begin{array}{l} \text{This is the dot product of }\mathbf{x}_1\text{ and }\mathbf{x}_2\text{ after explicitly} \\ \text{mapping them to the higher-dimensional space.} \end{array} \end{array}$$

$$\begin{array}{l} \textbf{Using the Kernel Trick: }\\ \\ \textbf{Polynomial Kernel Definition:} \\ \\ K(\mathbf{x}_i, \mathbf{x}_j) = (\mathbf{x}_i^\top \mathbf{x}_j + c)^d \\ \\ \textbf{For this example:} \\ \\ d = 2 \ (\text{degree of the polynomial}), \quad c = 0 \ (\text{no bias term}) \\ \\ \textbf{Given: } \\ \\ \mathbf{x}_1 = (1, -1), \quad \mathbf{x}_2 = (-1, -1) \\ \\ \textbf{Compute } K(\mathbf{x}_1, \mathbf{x}_2): \\ \\ \begin{align*} K(\mathbf{x}_1, \mathbf{x}_2) &= ((1)(-1) + (1)(-1))^2 \\ &= (-1 - 1)^2 \\ &= (-2)^2 \\ &= 4 \end{align*} \\ \\ \begin{array}{l} \text{Using the kernel trick, we directly compute the dot product in the higher} \\ \text{dimensional space without explicitly mapping the points.} \end{array} \end{array}$$

Popular Kernel Functions

Linear kernel

For a dataset that is linearly separable, the linear kernel is ideal. When used for non-linear data sets, which are the main topic of this article, it may result in underfitting and create a linear decision boundary. It’s provided as the input feature vectors' dot product.

This kernel merely constructs the hyperplane or line of best fit to divide the data points. It does not perform any particular transformation to a higher dimension.

$$Linear Kernel Function: K(x_i, x_j) = x_i \cdot x_j$$

Polynomial kernel

The polynomial kernel transforms the data into a polynomial feature space of order d. It does a dot product on the feature vector with a constant c, all within the degree of d. The higher the degree of the polynomial, the better the kernel captures the relationships in the nonlinear dataset.

$$Polynomial Kernel Function: K(x_i, x_j) = (x_i \cdot x_j + c)^d$$

Gaussian or Radial Basis Function (RBF) kernel

The Gaussian kernel, also known as the RBF kernel, is often used in SVM to map the input feature vector to an infinite-dimensional feature space using a Gaussian function. This kernel can handle more complex relationships.

$$RBF Kernel Function: K(x_i, x_j) = \exp(-\gamma \|x_i - x_j\|^2)$$

Sigmoid kernel

The sigmoid kernel acts similarly to the activation function in neural networks. It functions similarly to a two-layered perception network and can map data into a higher-dimensional feature space.

$$Sigmoid Kernel Function: K(x_i, x_j) = \tanh(\alpha(x_i \cdot x_j) + c)$$

There are other kernel functions such as Laplacian kernels, hyperbolic kernels, exponential kernels, and custom kernels that you can look into if you’re curious.

How to Choose the Right Kernel

The various kernel functions are applied based on the linear and nonlinear relationships in the feature space. The linear kernel is simple and fast, and it works well with linearly separable data but not with high-dimensional data.

The polynomial kernel is well-suited for data with non-linear or polynomial relationships, as well as low-dimensional data. The RBF kernel is ideal for dense data that you have no prior knowledge of. Finally, the sigmoid kernel works well for binary and categorical data points.

SVM Kernel Implementation

Let’s now go through an example showing how you can use this technique.

Step 1: Import the necessary libraries

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from mpl_toolkits.mplot3d import Axes3D
from sklearn.preprocessing import StandardScaler

Step 2: Generate the non-linear dataset

The non-linear dataset used in this article is a circle dataset from sklearn.datasets. We used 1500 samples with a random_state of 46 to keep the dataset consistent for reproducibility. We added a Gaussian noise to the data of 10%. This function generate_circle_data is implemented to generate the dataset used in the article.

def generate_circle_data(n_samples=1500, noise=0.10, random_state=46):
    """
    Generate two concentric circles dataset.

    Parameters:
    -----------
    n_samples : int
        The total number of points generated
    noise : float
        Standard deviation of Gaussian noise added to the data
    random_state : int
        Random seed for reproducibility

    Returns:
    --------
    X : array of shape [n_samples, 2]
        The generated samples
    y : array of shape [n_samples]
        The integer labels (0 or 1) for class membership of each sample
    """
    return make_circles(n_samples=n_samples, 
                       noise=noise, 
                       random_state=random_state)

Step 3: Plot the 2D Data

The data generated above comes in 2D form. Each color represents the two different data samples. The data points were plotted which allows us to see it as a circular dataset using the Matplotlib library.

def plot_2d_data(X, y, title="2D Circle Dataset"):
    """
    Plot the 2D dataset with different colors for each class.

    Parameters:
    -----------
    X : array-like of shape (n_samples, 2)
        The input samples
    y : array-like of shape (n_samples,)
        The target values (class labels)
    title : str
        The title of the plot
    """
    plt.figure(figsize=(8, 6))
    plt.scatter(X[:, 0], X[:, 1], c=y, marker='.', cmap='viridis')
    plt.title(title)
    plt.xlabel('X₁')
    plt.ylabel('X₂')
    plt.colorbar(label='Class')
    plt.grid(True, alpha=0.3)
    plt.show()

The output image of the dataset is given below:

Step 4: Transform into a Higher-Dimensional Space

The data in 2D is transformed into a 3D space using the polynomial kernel. We achieved this by creating a third feature X3 so it can be mapped into a higher dimensional space for easy separation.

def transform_to_3d(X):
    """Transform 2D data to 3D using radius-based transformation"""
    X1 = X[:, 0].reshape(-1, 1)
    X2 = X[:, 1].reshape(-1, 1)
    # Modified transformation to create better separation
    X3 = X1**2 + X2**2
    return np.hstack((X1, X2, X3))

Step 5: Plot the 3D Transformation

The next step is to plot the 3D transformed dataset. It now looks like a U-shaped bowl, and is separated with a hyperplane after fitting a LinearSVC model from the sklearn library as the kernel we’re using. This shows a practical example of the concepts you’ve learned so far:

def plot_3d_transformation_with_separator(X_transformed, y, title="3D Transformed Dataset with Linear Separator"):
    """Plot the 3D transformed dataset with a clear linear separating plane"""

    # Scale the transformed features
    scaler = StandardScaler()
    X_scaled = scaler.fit_transform(X_transformed)

    # Fit linear SVM with adjusted parameters for better separation
    svm = LinearSVC(C=1.0, dual="auto", max_iter=5000)
    svm.fit(X_scaled, y)

    # Create the 3D plot
    fig = plt.figure(figsize=(12, 8))
    ax = fig.add_subplot(111, projection='3d')

    # Plot the two classes with different colors and markers for clarity
    class_0 = y == 0
    class_1 = y == 1

    ax.scatter(X_transformed[class_0, 0], 
              X_transformed[class_0, 1], 
              X_transformed[class_0, 2],
              c='blue', 
              marker='o',
              label='Class 0',
              alpha=0.6)

    ax.scatter(X_transformed[class_1, 0], 
              X_transformed[class_1, 1], 
              X_transformed[class_1, 2],
              c='red', 
              marker='^',
              label='Class 1',
              alpha=0.6)

    # Create a grid for the separator plane
    x_min, x_max = X_transformed[:, 0].min() - 0.2, X_transformed[:, 0].max() + 0.2
    y_min, y_max = X_transformed[:, 1].min() - 0.2, X_transformed[:, 1].max() + 0.2

    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 50),
                        np.linspace(y_min, y_max, 50))

    # Get the separating plane coefficients
    w = svm.coef_[0]
    b = svm.intercept_[0]

    # Calculate z coordinates of the plane
    grid_points = np.c_[xx.ravel(), yy.ravel(), np.zeros(xx.ravel().shape[0])]
    scaled_grid = scaler.transform(grid_points)

    # Calculate the separator plane
    z = (-w[0] * scaled_grid[:, 0] - w[1] * scaled_grid[:, 1] - b) / w[2]
    z = z.reshape(xx.shape)
    z = scaler.inverse_transform(np.c_[xx.ravel(), yy.ravel(), z.ravel()])[:, 2].reshape(xx.shape)

    # Plot the separating plane with adjusted transparency
    surface = ax.plot_surface(xx, yy, z, alpha=0.3, cmap='coolwarm')

    # Customize the plot
    ax.set_xlabel('X₁')
    ax.set_ylabel('X₂')
    ax.set_zlabel('X₁² + X₂²')
    ax.set_title(title)

    # Add legend
    ax.legend()

    # Adjust the viewing angle for better visualization
    ax.view_init(elev=20, azim=45)

    # Add text description
    ax.text2D(0.05, 0.95, 
              "Polynomial Kernel Transformation:\nΦ(x₁,x₂) → (x₁,x₂,x₁²+x₂²)\n\nClasses are linearly separable\nin transformed space", 
              transform=ax.transAxes, 
              bbox=dict(facecolor='white', alpha=0.8))

    plt.show()

def main():
    # Generate and plot the dataset
    X, y = generate_circle_data()

    # Transform and plot 3D data with clear separator
    X_transformed = transform_to_3d(X)
    plot_3d_transformation_with_separator(X_transformed, y)

if __name__ == "__main__":
    main()

The main function is a function of functions that put together all the other functions such as generate_circle_data, transform_to_3d and plot_3d_transformation_with_separator together to establish the model. The image below shows a better separation with the aid of the polynomial kernel.

Here’s the full code:

Conclusion

In this article, you learned about the efficiency of SVM kernels for non-linear classification applications. The various functions demonstrated computational efficiency by changing input data into higher dimensional data, as shown in the example, without requiring vast amounts of storage or processing.

SVM can be used in a variety of classification tasks, including image and text classification, and it has proven to be extremely efficient.

References

1. Park, H., & Son, J.-H. (2021). Machine learning techniques for THz imaging and time-domain spectroscopy. Sensors, 21(4), 1186. https://doi.org/10.3390/s21041186

2. Scikit-learn developers. (2024). Support vector machines. Scikit-learn.https://scikit-learn.org/1.5/modules/svm.html