The Raspberry Pi Model B was introduced in 2012 as the first sellable unit, and the company has since released many more models. There are even low-cost models like the Raspberry Pi Zero Series, which is quite small and tailored to embedded systems applications. All the models operate on an operating system called the Raspberry Pi OS, a Linux flavor niched for Raspberry PI Computers.

In this tutorial, we’ll be using the Raspberry Pi 4 Model for a headless setup through a SSH connection using Visual Studio Code (VS Code). The Raspberry Pi 4 Model has a Quad-core ARM Cortex-A72 (64-bit) SoC at 1.5GHz, up to 8GB RAM options, video inputs, Ethernet shield, USB ports, MicroSD card slot for storage, USB-C power input and 40 General Purpose Inputs and Outputs Pins (GPIO). Impressive, right?

You’ll be able to use the Raspberry Pi as a personal computer, for home automation and IoT projects, robotics projects, network applications, educational tools, and Artificial Intelligence projects.

Table of Contents

• Understanding the Headless Setup

• Prerequisites

• Preparing the MicroSD Card

• How to Boot the Raspberry Pi

• Understanding the LED of the Raspberry Pi During Setup

• How to Establish an SSH Connection

• How to Set Up Visual Studio Code for Remote Development

• How to Write and Run the Code Remotely

• Conclusion

Understanding the Headless Setup

Many Raspberry Pi computers are sold with additional peripherals, including a keyboard, mouse, and monitor, that are essential for the Raspberry Pi's setup. A headless setup is the process of configuring the Raspberry Pi or preparing it for use without needing these peripherals. This entails operating the Raspberry Pi through a network protocol like SSH (Secure Shell) or VNC (Virtual Network Computing).

This is really helpful when you don’t need peripherals, as it lets you use your personal computer to connect to the Raspberry Pi without needing to purchase specialized peripherals. It’s also excellent for remote access. This headless setup is also essential for remote monitoring systems, such as surveillance systems with remote camera access, and IoT systems.

Remote development lets you write code and modify your Raspberry Pi and other devices connected to the GPIO pins through a headless configuration via SSH.

SSH guarantees a secure connection for transferring and modifying files, as well as transferring and debugging commands from one computer (your personal computer) to another computer (the Raspberry Pi). It restricts unauthorized access from any other system that aims to intercept the communication channel.

Prerequisites

Here’s what you’ll need to follow along with this tutorial:

Hardware Requirements

1. Raspberry Pi 4 or 5

2. MicroSD Card (8GB or higher recommended)

3. Flash Drive with SD Card Slot or a MicroSD Card Adapter

4. Power Supply (5V 2A/3A)

5. Network Connection (Wi-Fi, Pi, and laptop must be on the same network)

6. Personal Computer (Windows, macOS, Linux)

Software Requirements

1. Raspberry Pi Operating System (Raspberry Pi OS)

2. Visual Studio Code

3. Remote SSH Extension in VS Code

Preparing the MicroSD Card

The Raspberry Pi requires a MicroSD Card that serves as the storage of your the Raspberry Pi OS using Raspberry Pi Imager. The operating system of the Raspberry Pi provides a graphical interface to interact with the Raspberry Pi, store files and datasets, and write commands to get your Raspberry Pi working.

But the Raspberry Pi needs an empty MicroSD Card to install the Raspberry Pi OS in the MicroSD Card. Here are some step by step instructions that’ll show you how to get your MicroSD Card setup before inserting it back into the Raspberry Pi for SSH Connection.

Downloading and flashing Raspberry Pi OS

Insert your MicroSD Card into a flash drive with a SD Card slot

Aside from using a flash drive with an SD Card slot (so as to get the memory card connected to the computer), you can also use a SD Card adapter. Make sure it’s inserted into your computer where you have the Raspberry PI Imager downloaded to flash – that is, transfer the OS into the SD Card.

Download the Raspberry Pi Imager based on your PC’s operating system

This involves clicking the link and selecting your operating system (either MacOS, Windows or Linux operating system). The Raspberry Pi OS comes in these variants for different OSes

Next, install and open the Raspberry Pi imager

Click the Raspberry Pi Imager download, follow all the instructions during the installation process. Once this screen pops up, you’re good to go!

Choose your Raspberry Pi Device and operating system and select Storage

For each of the three configurations, you must select one sequentially. Select a device according to the type of Raspberry Pi you have, and various options will appear. I selected the Raspberry Pi 4, as it is my preferred device. You may choose between the Raspberry Pi 5 and the Raspberry Pi Zero 2 W, depending on your device requirements.

Next, proceed to the operating system – I would recommend choosing the 64-bit version. While many people opt for the legacy version (32-bit), I think the 64-bit version is best. Once you're finished, you can choose a storage option, and your MicroSD should appear. My storage is around 128GB, which is why you can see 125.1GB displayed there in the screenshot below:

Click on “Next” and edit the settings

It is a customary practice to keep your username as "pi", but it’s not required. The goal is to have something simple and easy to remember when setting up your SSH connection. It’s also helpful to make your password simple. I used 'roboticsai'.

Try to avoid using numbers simply to make things easier, because you may not be able to see what you are entering in the terminal. Then, make sure that your wireless LAN and SSID (WIFI or Hotspot name if you're using a phone, as well as the password for your WIFI or Hotspot) is the same network as the one linked to your computer.

Click on “SERVICES” and enable SSH. Then use password authentication for security and Click on “SAVE”.

After you've completed the changes in the General Section, go to the Services section and click the checkbox button “Enable SSH”. Once highlighted, make sure you pick “Use password authentication”, avoid the “RUN SSH-KEYGEN” button at the moment, and then click Save.

Click “YES” to apply the customizations, and the Raspberry Pi OS should get flashed into your SD Card.

Following the previous stage, you will be shown various buttons to apply the adjustments you have made. Pick yes, and the Raspberry Pi OS will be flashed or transferred to your Memory Card. This could take between 10 and 20 minutes to go from transferring to writing or customizing. Hold on and enjoy the process.

After a successful installation into the disk, remove your SD Card.

You will receive a successful popup like the one shown below. This demonstrates that all processes were completed successfully, and the Raspberry Pi OS is now installed.

How to Boot the Raspberry Pi

Eject the MicroSD safely from your computer

Once the installation is successful, eject the MicroSD safely from the computer.

Insert it “upside down” into the MicroSD card slot of your Raspberry Pi

To properly insert the MicroSD card, place it gently into the slot with the back or gold side facing upward. It will protrude slightly once it is inserted. You are good to go!

Connect the USB-C port of your Raspberry Pi to your computer. Give the Raspberry Pi some time to load

Get a USB-C cable and connect one end to your Raspberry Pi's USB-C port and the other to a laptop port. It should light up red, indicating that there is an adequate power source. You may also power your Raspberry Pi directly by plugging into a wall socket.

After a while, the memory card should begin to boot into the Raspberry Pi, and the green LED will blink for a while. In the next section, we’ll talk about the different states of the two LEDs during and after a successful boot.

Understanding the LED Status of the Raspberry Pi During Setup

The below table describes the LED statuses you might see when you power on your Raspberry Pi and the SD Card is in the slot.

How to Establish an SSH Connection

The SSH (Secure Shell) connection is a network protocol that allows two computers to safely communicate without leaking any information. It’s also used for remote command line execution and for file transfers between two computers.

To establish an SSH connection, you’ll have to complete a few steps. Then I’ll explain how to enable SSH using a Visual Studio Code extension

Create a wpa_supplicant.conf.txt in the same folder of your Raspberry Pi SD Card

Insert your MicroSD card back into the computer. Then the files that comprise the Raspberry Pi OS will appear on your computer. Create a new text (.txt) document on your computer, similar to the image below, under the SD Card storage section.

Add the code below, making sure that "ssid" is the name of your Wi-Fi network and "psk" is your network's password.

country=NG # Your 2-digit country code
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
network={
    ssid="Josiah"
    psk="roboticsai"
    key_mgmt=WPA-PSK
}

Save the file on the same SD Card

Once you've finished producing the text file, save it to the SD Card storage, as shown in the image below.

Create a .ssh folder

In your personal computer, create a .ssh folder if it doesn’t exist on your personal computer.

If it exists, the .ssh folder should contain files like id_rsa, known_hosts, and config files. It shouldn’t be empty.

After a successful boot, open your terminal or command line application on your personal computer.

Make sure that the Raspberry Pi is connected to the same network before moving ahead. Once your wifi or mobile hotspot is switched on, make sure it’s the same password as the wpa_supplicant.conf.txt and the settings page while installing the Raspberry Pi.

As long as the SD card is in the Raspberry Pi and there is adequate power supply for at least 2-5 minutes, the Raspberry Pi will get connected to the wifi or your mobile hotspot.

How to Resolve Connection Problems

If there is no connection, reinstall the Raspberry Pi OS Imager on the SD Card again. Then you can also change the network AP Band from 5GHz to 2.5GHz or vice-versa. This can be very tricky.

It should get connected after trying this. Just make sure that the passwords are consistent and that you don’t accidentally have the caps lock key switched on while typing, for example.

To confirm if the Raspberry Pi is connected using the command line interface, use the ping command – it shows the devices connected to the device.

ping raspberrypi.local

After running the above command, you should see an image showing the connection once it’s successful like this:

For establishing an SSH connection using the terminal, run the code below:

ssh pi@raspberrypi.local

This will result in a request for a password. If it shows an error like the image below, it means you have to delete the known_hosts.old and known_hosts if either or both exist in the .ssh folder in your PC. This is because the keys are conflicting with each other. Then re-run the above code ssh pi@raspberrypi.local in your terminal.

After successful entry, type “yes” in the terminal.

Connection Closed should show when the connection is successful.

How to Set Up Visual Studio Code for Remote Development

Download and install Visual Studio Code if you don’t have it already.

Then, click on the VS Code extension and search for Remote - SSH by Microsoft and install it to your machine.

Next, click on the “Remote Explorer” icon that looks like a monitor. Select the SSH config in your C:\Users\{name}\.ssh\config folder.

Make sure the config has this command:

Host raspberrypi.local
    HostName raspberrypi.local
    User pi

Enter your username as raspberrypi.local and input your password – the same as the password during loading Raspbian OS.

After inputting the correct password, it should start downloading the server.

Congratulations! The image below has a blue rectangle button showing SSH:raspberrypi.local which shows a successful SSH Connection through Visual Studio Code. This also means you can start remote development as we discussed earlier in this tutorial.

How to Write and Run the Code Remotely

Create a new file on your VS Code. This way, you’re creating files and writing to them directly. Go to the terminal and type the commands to create a folder and a file:

Create a new file and write in your code

Create a new file and name it led.py on your Visual Studio Code. It should be in the same folder as test-raspberry on the Raspberry Pi remote network through the SSH connection on VSCode.

Once you have your file created, you can write in your code such as blinking LED to a Raspberry Pi, as you can see in the code below:

from gpiozero import LED
from time import sleep

# Set the GPIO pin where the LED is connected
led = LED(17)  # Replace 17 with your GPIO pin number

# Blink the LED in a loop
while True:
    led.on()        # Turn LED on
    sleep(1)        # Wait for 1 second
    led.off()       # Turn LED off
    sleep(1)        # Wait for 1 second

After writing this code in the new file you’ve created, run the code by typing the command below in your terminal:

python led.py

As soon as this command is sent, the LED positive terminal is connected to the GPIO 17 according to the code and the negative terminal is connected to the GND GPIO pin of the Raspberry Pi. The image from Random Nerd Tutorials below shows the GPIO pins and their number to understand the connection. Just note that the connection of the LED is beyond the scope of this tutorial.

The LED should start blinking each second according to the code. With this, you can now control your Raspberry Pi (a tiny computer) with another computer (your personal computer) through an SSH connection on Visual Studio Code.

Conclusion

In this tutorial, you went through the whole process of setting up a headless Raspberry Pi for remote development using VS Code.

This offers a wide range of benefits: there’s no need for external peripherals, it provides remote access from anywhere within your network, and it leverages efficient coding and debugging with VS Code integration.

You can use this to deploy web servers and IoT dashboards, and you can explore with automating processes using Python scripts and GPIO control.

This VPN will allow you to access your home network from anywhere as if you’re still at home. So why is this useful, you might ask? Well, it allows you to use your home network IP, no matter where you are, which is a good for privacy.

In this article, we’ll use Tailscale, an open-source mesh VPN (Virtual Private Network) service that streamlines connecting devices and services securely across different networks. It enables encrypted point-to-point connections using the open-source WireGuard protocol. This means that only devices on your private network can communicate with each other.

Table of Contents

• Prerequisites

• Install Raspberry Pi OS Lite (32-bit)

• Boot The Raspberry Pi

• SSH Into The Raspberry Pi and Login

• Install Tailscale on Raspberry Pi

• Key Expiry

• Configuring the Raspberry Pi as an Exit Node

• Conclusion

Prerequisites

• Raspberry Pi (I am working with a Raspberry Pi 5)

• Raspberry Pi Imager

• A Micro SD Card (8GB is enough)

• A Micro SD Card reader for your computer.

• Home Router

• A Tailscale account

Install Raspberry Pi OS Lite (32-bit)

We’ll start this process by installing the Raspberry Pi OS Lite (32-bit) on the micro SD card we have. We will be making use of the Raspberry Pi Imager software which is available for free here.

When you run the imager software, pick the Raspberry Pi Device, which for me is a Raspberry Pi 5.

Then in Operating System, click on Raspberry Pi OS (other), then scroll down to Raspberry Pi OS Lite (32-bit)

Next, select your SD card which you have inserted into the card reader, and the card reader into the computer. Your screen should look similar to what you see below. Click on next.

After next, you should see a pop-up asking if you would like to apply OS customisation settings.

Next, click on edit settings. Enable set hostname and write the name you want to give the Pi. For this tutorial, I will be using dapivpn. Then enable set username and password. Pick a username and a strong and secure password

You can enable configure wireless LAN if you plan to use Wifi, but if you are team Ethernet cable, you can skip this. I will be using WiFi in this tutorial though.

Now you’ll need to enable set local settings and pick your correct time zone and keyboard layout.

After that, go to the Services tab, then enable SSH and click on “Use password authentication”. Then click save, then yes on the apply customisation screen, and yes again. Remember this will erase all the data on the SD card, so make sure you’re using one without any important files on it.

This is how your Raspberry Pi Imager should look now:

Boot the Raspberry Pi

After this is done, take the SD card and insert it into your Raspberry Pi. Then plug the power cable into the Raspberry Pi and wait some minutes for it to boot properly. You will know it is ready when the green LED light stays on.

Now you should go to your router and set a static IP to the Raspberry Pi. For mine, I set it to 192.168.8.21.

SSH into the Raspberry Pi and Login

Open up your command line terminal. Type “ssh <pi username>@<raspberry_pi_ip_address>”. For me, this would be:

ssh danpi@192.168.8.21

Then type in the password you used. You should see your username and the Pi hostname and this confirms you have logged in successfully to it.

Type in:

sudo apt update && sudo apt upgrade -y

You run this command to make sure everything is up to date locally.

Now reboot your Pi after this by typing:

sudo reboot

Install Tailscale on Raspberry Pi

Now you’re going to add Tailscale’s package signing key and repository.

curl -fsSL https://pkgs.tailscale.com/stable/debian/bookworm.noarmor.gpg | sudo tee /usr/share/keyrings/tailscale-archive-keyring.gpg >/dev/null 
curl -fsSL https://pkgs.tailscale.com/stable/debian/bookworm.tailscale-keyring.list | sudo tee /etc/apt/sources.list.d/tailscale.list

Install Tailscale using these commands:

sudo apt-get update
sudo apt-get install tailscale

Next, you need to connect your Pi to your Tailscale network and authenticate. You can do that with the following command:

sudo tailscale up

Your browser should look like this.

To locate the Tailscale IPv4 address for the Raspberry Pi, run this command:

tailscale ip -4

You can also see it on the Tailscale dashboard in your browser.

At this point, you’re done installing Tailsacle and you just need to do some finishing touches.

Key Expiry

There is something you need to know when it comes to adding a device to Tailsacle. By default, and as a security feature, Tailscale requires devices to re-authenticate after a certain period of time has elapsed, usually 180 days.

If the re-authentication does not occur, keys expire and the connection stops working. It’s up to you to choose what you prefer, as this is a security feature that comes with some inconvenience.

I will be disabling the key expiry on the Raspberry Pi, as I fully trust it. To do this, you need to:

• Open the Machines page of the Tailscale admin console.

• Find the Raspberry Pi on the row and select the option menu there.

• Click on the Disable Key Expiry option. You should see an Expiry Disable label below the machine name.

How to Configure the Raspberry Pi as an Exit Node

Another thing you’ll need to know about when it comes to Tailscale is what an exit node is. A Tailscale exit node is a designated device in your Tailscale network that routes all of your internet traffic through it. No matter where you are, once you have this device activated as an exit node, when you turn on Tailscale, it routes your internet traffic through the device.

Ideally, you want a device that is powered on 24/7 to serve as your exit node. That’s why we are picking the Raspberry Pi, as it is a low-powered computer.

We are already 90% of the way, as we have Tailscale running on our Pi. Remember to also have Tailscale installed on as many devices on your local network as possible. What’s left is to allow your Pi to act as an exit node, so all your internet traffic or LAN traffic routes through it, giving you access to:

• Local network devices at home

• Your home public IP

• Internal services like NAS, printers, cameras, and so on

To do this, SSH into your Raspberry Pi and follow these steps:

• Enable IP Forwarding. IP forwarding allows your Raspberry Pi to pass traffic between its network interfaces. Run the commands below line by line:

    echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf
  
    echo "net.ipv6.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.conf
  
    sudo sysctl -p /etc/sysctl.conf
  

• Advertise the Raspberry Pi as an exit node:

    sudo tailscale up --advertise-exit-node
  

• Open the Machines page of the Tailscale admin console.

• Find the Raspberry Pi on the row. You should see an Exit Node label on its name.

• Click on the options menu there and select Edit Route Settings.

• Check the box for Use as an exit node, then save.

Now you should see the option of routing the internet through an exit node when you open up your Tailscale app on mobile or PC or anywhere you have it installed. When you see that option, you will also see the Raspberry Pi as an exit node option. You can also add more devices as an exit node if you want more options.

Conclusion

Using the Tailscale app on other devices, you can now route traffic securely through the Raspberry Pi by selecting it as an exit node. Tailscale also provides clear, step-by-step guides tailored to each device type for setting up and using an exit node.

You can now be away from your home internet but still connect to the internet as if you were home. See you next time.

Why have an on-premise artifact manager? There are many reasons for it:

• Use your private infrastructure: You may have proprietary code that needs to be safeguarded.

• Faster artifact download speeds: If you constantly download the same artifacts over the Internet, you can cache them on a central location, for the benefit of your multiple users across multiple servers by caching them.

• Control what artifacts make it to your build chain: Centralize the location of the artifacts, ensure they are approved for usage, and also confirm than they do not contain malicious code.

• Segregate who can have access to your artifacts: You may have more strict requirements on who can access some artifacts within your own organization.

In this article I will show you how you can download, install, and configure the OSS version of Nexus 3 using an Ansible playbook.

Nexus 3 will run on an Orange PI 5 computer with 8 GB or RAM, but this provisioning can be done on any machine with the minimum requirements. Part of the setup will consist of setting a proxy for PyPI.org, for the machines listed on my inventory file.

What you need to run the code from this tutorial

1. An Internet connection to download the source code for the Ansible playbook, Nexus, and PIP modules

2. Two or more Linux machines (I used Debian, Armbian and Fedora IOT), with at least 8 GB of RAM. My cluster has a mix of Raspberry PI 4 and an OrangePI 5.

3. Ansible controller will run on the Fedora machine, but any server can be the controller. Installation instructions for Ansible are easy to follow.

Playbook Organization

I divided the tasks in groups and the resulting playbook looks like this:

[josevnz@dmaf5 Nexus3OnOrangePI]$ tree -N ansible/
ansible/
├── inventories
│   └── home
│       └── hosts.yaml
├── roles
│   ├── clients
│   │   ├── tasks
│   │   │   └── main.yaml
│   │   └── templates
│   │       └── pip.conf.j2
│   └── nexus
│       ├── files
│       │   └── swagger.json
│       ├── tasks
│       │   ├── download.yaml
│       │   ├── install.yaml
│       │   ├── main.yaml
│       │   ├── post_install.yaml
│       │   ├── pre_install.yaml
│       │   ├── repositories.yaml
│       │   ├── third_party.yaml
│       │   └── user.yaml
│       └── templates
│           ├── logrotate.nexus3.j2
│           ├── nexus3.service.j2
│           ├── nexus.rc.j2
│           └── nexus.vmoptions.j2
├── site.yaml
├── vars
│   ├── clients.yaml
│   └── nexus.yaml
└── vault
    ├── nexus_password.enc
    └── README.md

13 directories, 21 files

Now a little bit of explaining:

• There are two roles: ‘nexus’ and ‘clients’. The nexus role is used to setup the artifact management software, while the client role sets up the pip settings on every machine.

• Vars contains variables used on each role, separated by files to make their usage more clear

• We have passwords, and we managed them using Ansible vault feature.

• The file ‘site.yaml’ Orchestrates the role execution:

- hosts: all
  tags: clients
  vars_files:
    - vars/clients.yaml
  roles:
    - clients
- hosts: nexus_server
  tags: nexus
  become_user: root
  become: true
  vars_files:
    - vars/nexus.yaml
  roles:
    - nexus

Now let’s move on to see the universe where the playbook will be executed.

The Host Inventory

In my case it is quite simple – I have two main groups: ‘clients’ and the machine where the Nexus 3 server itself will run:

all:
  children:
    nexus_server:
      hosts:
        orangepi5.home:
    home_lab:
      hosts:
        dmaf5.home:
        raspberrypi.home:
        orangepi5.home:

The next important task is to download and configure Nexus 3.

How to Install Nexus 3

The file main.yaml describes the order and purpose of each installation task for the Nexus role:

# Tasks listed here are related to the remote Nexus 3 server
# Included tasks are called in order
---
  - include_tasks: third_party.yaml
  - include_tasks: pre_install.yaml
  - include_tasks: download.yaml
  - include_tasks: install.yaml
  - include_tasks: post_install.yaml
  - include_tasks: user.yaml
  - include_tasks: repositories.yaml

Let’s see first what I like to call the “core tasks”:

1. third_party.yaml: In here we install the OpenJDK8 (Nexus 3 is written in Java) and logrotate to take care of the stale logs.

2. pre_install.yaml: A lot happens here, like creating required directories for nexus, dedicated non-privileged user that will run the process.

3. download.yaml: As the name says, we get a fresh version of the Nexus 3 OSS software and make sure it has the right checksum. We don’t want to install malware from the Internet.

Then come the tasks that fall into the “customized installation group”:

1. install.yaml: Unpack the software, prepare the systemd unit to start it automatically, setup JVM settings for Nexus, and deploy the logrotate configuration.

2. post_install.yaml: Exciting stuff happens here – the software is installed, and we run it for the first time. We also change the default password using the REST API, so we can move to the customization stage.

3. user.yaml: Here we prepare to provide our end users with proper access to the services offered by Nexus. We do this using a combination of the REST-API and Ansible client code:

# https://help.sonatype.com/repomanager3/installation-and-upgrades/post-install-checklist
# https://help.sonatype.com/repomanager3/integrations/rest-and-integration-api
---
- name: Enable anonymous user
  tags: anonymous
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/security/anonymous"
    method: PUT
    body_format: raw
    status_code: [ 200, 202, 204 ]
    headers:
      Content-Type: application/json
    body: |-
      { "enabled" : true, "userId" : "anonymous", "realmName" : "NexusAuthorizingRealm" }
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true
- name: Enable Docker security realm
  tags: docker_realm
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/security/realms/active"
    method: PUT
    body_format: raw
    status_code: [ 200, 202, 204 ]
    headers:
      Content-Type: application/json
    body: |-
      [ "NexusAuthenticatingRealm", "NexusAuthorizingRealm", "DockerToken" ]
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true

The logic is easy to follow, by using the ‘PUT’ http method you can tell is a modification operation (meaning existing roles and users already exist). Error detection is done by getting the HTTP codes returned by Nexus.

Next step is to prepare our local PyPi proxy. This is a multistep task and will be described in detail next.

How to Set Up PyPI Proxy on Nexus 3

The last file on the Nexus 3 role is ‘repositories.yaml’. In here we go through the following steps:

1. Check if the proxy was already setup (GET or read only operation)

2. If it doesn’t exist, create a new one (POST method with JSON payload with details to create whole new repository)

Notice than this playbook doesn’t offer the option to update repository settings. It is possible to do with the REST API, but I will leave that as an exercise to the reader.

The tasks to prepare the PyPi proxy are shown below:

# Create proxy for repositories
# https://help.sonatype.com/repomanager3/integrations/rest-and-integration-api
# PyPi: https://pip.pypa.io/en/stable/user_guide/
---
- name: Check if the PyPi proxy exists
  tags: pypi_proxy_exists
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/repositories/pypi/proxy/python_proxy"
    method: GET
    body_format: raw
    status_code: [ 200, 202, 204, 404 ]
    headers:
      Content-Type: application/json
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true
  register: python_local
- name: Create PyPI proxy
  tags: pypi_proxy_create
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/repositories/pypi/proxy"
    method: POST
    body_format: raw
    status_code: [ 201 ]
    headers:
      Content-Type: application/json
    body: |-
      {
        "name": "python_proxy",
        "online": true,
        "storage": {
          "blobStoreName": "default",
          "strictContentTypeValidation": true
        },
        "proxy": {
          "remoteUrl": "https://pypi.org/",
          "contentMaxAge": -1,
          "metadataMaxAge": 1440
        },
        "negativeCache": {
          "enabled": true,
          "timeToLive": 1440
        },
        "httpClient": {
          "blocked": false,
          "autoBlock": true,
          "connection": {
            "retries": 0,
            "timeout": 60,
            "enableCircularRedirects": false,
            "enableCookies": true,
            "useTrustStore": false
          }
        }
      }
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true
  when: python_local.status == 404

We are almost there. Now we need to tell our PyPi clients than we should use our local Nexus and not the direct PyPi site to get our Python libraries.

How to Set the Clients

The clients role is much simpler and only requires deploying a template for pip.conf with enough information to force the search on our new repository:

# Tasks here are meant to be used on our clients user
---
- name: Create installation directory for pip.conf
  tags: pip_basedir
  ansible.builtin.file:
    state: directory
    path: ""
    owner: ""
    group: ""
    mode: "u+rwx,go-rwx"
- name: Copy pip.conf file
  tags: pip_copy
  ansible.builtin.template:
    src: pip.conf.j2
    dest: "/pip.conf"
    owner: ""
    group: ""
    mode: u=rxw,g=r,o=r

The resulting file gets deployed on ‘~/.config/pip/pip.conf’ of every machine:

# https://pip.pypa.io/en/stable/topics/configuration/
[global]
timeout = 60
[install]
index = http://orangepi5.home:8081/repository/python_proxy/pypi
index-url = http://orangepi5.home:8081/repository/python_proxy/simple/
trusted-host = orangepi5.home

The file above shows an example of how the final version of the file will look once deployed on my cluster (yours will be different with the resolved URL).

It is time now to run the whole playbook and see what it looks like.

How to Run the Playbook

To run the playbook, we pass a few arguments:

1. The location of our host inventory

2. The location of the encrypted password file and a master file containing the master password to unlock the contents of the protected file

3. And finally the location of our main playbook file

cd ansible
ansible-playbook --inventory  inventories --extra-vars @vault/nexus_password.enc --vault-password-file $HOME/vault/ansible_vault_pass site.yaml

How to test the new PyPI proxy

To test our new proxy, we will install Python Rich using pip and a virtual environment.

josevnz@orangepi5:~$ python3 -m venv ~/virtualenv/rich
(rich) josevnz@orangepi5:~$ . ~/virtualenv/rich/bin/activate
(rich) josevnz@orangepi5:~$ pip install rich
Looking in indexes: http://orangepi5.home:8081/repository/python_proxy/simple/
Collecting rich
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/rich/13.3.4/rich-13.3.4-py3-none-any.whl (238 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 238.7/238.7 KB 14.8 MB/s eta 0:00:00
Collecting pygments<3.0.0,>=2.13.0
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/pygments/2.15.0/Pygments-2.15.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 23.8 MB/s eta 0:00:00
Collecting markdown-it-py<3.0.0,>=2.2.0
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/markdown-it-py/2.2.0/markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.5/84.5 KB 6.9 MB/s eta 0:00:00
Collecting mdurl~=0.1
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/mdurl/0.1.2/mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: pygments, mdurl, markdown-it-py, rich
Successfully installed markdown-it-py-2.2.0 mdurl-0.1.2 pygments-2.15.0 rich-13.3.4

And then we can confirm than the cache was indeed used by seeing the new artifacts on the new repository:

See the PyPi artifacts

Let’s see a demo of the client in action, installing something else:

Further Customization Using the REST-API

Every Nexus installation allows you to download a JSON file that describes the API supported by the server. For example, in my server you can get a copy like this from my orangepi5.home server:

curl --fail --remote-name http://orangepi5.home:8081/service/rest/swagger.json

Also, the UI allows you to try the other REST API endpoints to customize your installation.

REST API testing

Conclusion

I recommend spending some time and reading the Nexus 3 book to get yourself familiar with the features this tool can offer.

The community prepared Debian and RPM installers, if you need this kind of setup as opposed to using Ansible.

Nexus 3 has lots of configurable settings. We covered only the surface here. While preparing this article I found 'ThoTeam Nexus3-oss repository' with a very complete and up-to-date playbook, but it was way more complex than anything I required for my home lab.

Archiva is another Open Source artifact manager, it is more limited in functionality but also simpler to setup.

There is a post-installation checklist with some tasks I did not need to complete for my home lab. Please check it out to make sure your setup is complete.

There was a chip shortage during the pandemic that has resulted in increased Raspberry Pi prices. Other world events have also skyrocketed the price of Raspberry Pis, and the manufacturing of a few models has even been stopped. You can read more about it here.

Because of this, I felt that switching to a Raspberry Pi alternative would be a good option for a project I wanted to work on.

Le Potato is similar to Raspberry Pi in terms of appearance, configuration, and so on. It also has the ability to run numerous operating systems such as Ubuntu, Debian, Raspbian, Android, and others.

But unfortunately, it does not come with a pre-installed wifi module, whereas Raspberry Pi has the wifi module pre-installed.

In this article, I'll give you clear step-by-step instructions to install an external wifi adapter driver in Le Potato running Ubuntu OS. For those who are running other operating systems, you can try the following steps, but I cannot assure you that it'll definitely work.

Let's have a quick look at my accessories.

Here's my Le Potato device:

Le Potato Device

And here's my Zebronics External Wifi Adapter:

Zebronics External Wifi Adapter

Trial and Error – What Didn't Work

Before I found my final solution and ended up installing the wifi driver and being able to access the internet with my wifi adapter, I tried many approaches. But none of them worked out.

Here's what I tried along the way:

1. I tried to install the driver given on the CD that was delivered along with the wifi adapter. But, I couldn't understand the steps they asked me to follow and finally ended up with a lot of errors.
2. I downloaded the exact driver for this device from the Zebronics official site. Again that did not pay off well.
3. I tried to install some open source drivers from GitHub forked by many people from the Realtek source. This also did not work out as expected.

Finally, I found an answer from the Ubuntu Q&A forum and I was able to install it on the first try. Though the steps were not so clear initially, I managed to figure them out. So I'll explain how to do it here.

How to Install the Wifi Adapter Driver for Le Potato in Ubuntu

Follow the below steps to install the driver on your device:

Install the dependencies

The first step is to install the required software.

You need to install git, dkms, build-essential, and linux-headers for your system architecture.

You can install them all together in a single command:

sudo apt-get install -y build-essential git dkms linux-headers-$(uname -r)

If you have been prompted (yes/no) while running the above command, just hit y (which is basically agreeing to install the software in your system).

Download the driver source

Drivers for some devices will not be available in any installable/executable formats. In such cases, you should download, compile, and install the source code on the machine directly. Unfortunately, this driver also falls under this category.

We can download the source of this driver from GitHub. Run the following command in your terminal to download the source code:

git clone https://github.com/kelebek333/rtl8188fu

Build and install the driver

Before building and installing the driver, you need to know about the dkms command in Linux. If you know about dkms, you can skip this paragraph and move on to the next paragraph.

DKMS stands for Dynamic Kernel Module Support. It's a program/framework that lets you install the supplementary versions of kernel modules. A package can be compiled and installed into the kernel tree. DKMS is called automatically upon installation of new Ubuntu kernel-image packages, and so modules added to DKMS will be automatically carried across updates.

This is the source package that we downloaded in the previous step. We need to add, compile, and install the source package into our kernel tree.

Run the following commands sequentially to add, compile and install the driver package:

Add Source to Kernel

sudo dkms add ./rtl8188fu

Compile source package

sudo dkms build rtl8188fu/1.0

Install the package into the kernel tree

sudo dkms install rtl8188fu/1.0

Copy the firmware

The compiled binary firmware file should then be copied to the default firmware location in Linux, that is /lib/firmware .

Firmware is software that enables the communication between hardware and software. It gives the machine instructions that make the hardware function.

Run the following command to copy the compiled firmware:

sudo cp ./rtl8188fu/firmware/rtl8188fufw.bin /lib/firmware/rtlwifi/

Disable power saving and auto suspend modes on kernel

It is always a good idea to disable power saving and auto suspend modes for wifi drivers. So, you'll need to add this option by default on updating the kernel, too. You can add this configuration in the .conf file in /etc/modprobe.d/ directory.

We are creating this conf file in /etc/modprobe.d directory, because we need to load this customized module with the persistent changes.

You use the rtw_power_mgnt flag to control power saving mode:

• 0 - Disables power saving
• 1 - Power saving on with minPS
• 2 - Power saving on with maxPS

You use the rtw_enusbss flag to control auto-suspend mode:

• 0 - Disables auto suspend
• 1 - Enables auto suspend

Run the following commands to create a .conf file and store the options:

sudo mkdir -p /etc/modprobe.d/
sudo touch /etc/modprobe.d/rtl8188fu.conf
echo "options rtl8188fu rtw_power_mgnt=0 rtw_enusbss=0" | sudo tee /etc/modprobe.d/rtl8188fu.conf

Blacklist the existing module

You have to blacklist the module which you tried to install before.

Note: Blacklisting a module will not allow it to be loaded automatically, but the module may be loaded if another non-blacklisted module depends on it or if it is loaded manually.

Let's assume you have added a module named rtl8188au. Then, you need to blacklist it by adding the following line at the end of the /etc/modprobe.d/blacklist.conf file.

blacklist rtl8188au

If you haven't added any such module, you can ignore the blacklisting part.

Reload the module

You need to reload the module to make it start working.

Here's the command to reload the module we added now:

sudo modprobe -rv rtl8188fu && sudo modprobe -v rtl8188fu

And you're done! You should be able to see the wifi enabled on your Le Potato running Ubuntu OS. If you cannot see it, reboot your system and everything should be alright.

Trying to connect to a network after installing the driver

Connected to my wifi network

Conclusion

In this article, we have gone through the steps to install the driver for our external wifi adapter.

These are the exact (basic) steps you need to follow to add any external module into your kernel.

Subscribe to my newsletter to receive more such insightful articles that get delivered straight to your inbox.

Starting with an embedded "Hello World" equivalent, and advancing to a text-to-morse-code translator, this article will walk you through the process.

• How to Set Up the Pi
  • Format the SD Card
  • Flash the Distribution
    • Configure Wifi and SSH
  • Complete the Circuit
• How to Set Up Cross Compilation
  • Install the Target
  • Specify the Linker
• How to Program an Embedded "Hello World"
  • Successfully Exit the Program
• How to Cross Compile the Program
• How to Transfer the Binary to the Pi
• How to SSH into the Pi
  • Run the Program
• How to Code a Text-to-Morse-Code Translator
• Appendix
  • Targets

How to Set Up the Pi

Format the SD Card

Use the Raspberry Pi Imager which can be downloaded from the Raspberry Pi Software Webpage.

Flash the Distribution

A distribution I'd suggest is Raspberry Pi OS Lite. This is a headless distribution, which means it does not come with a GUI.

Configure Wifi and SSH

Once that is done, you can insert the SD card into the Raspberry Pi, and power it up.

Complete the Circuit

Circuit Diagram

Pi Pinout

Connect negative to ground, and positive to BCM pin 17 as shown below:

The pinout can be seen here: https://pinout.xyz/

How to Set Up Cross Compilation

Install the Target

Use rustup to install the necessary target for your Raspberry Pi:

my-pc$ rustup add target arm-unknown-linux-gnueabihf

Appendix for more information about targets in Rust.

Specify the Linker

Download the raspberrypi/tools repository into a directory named rpi_tools:

my-pc:~ $ git clone https://github.com/raspberrypi/tools $HOME/rpi_tools

Edit the ~/.cargo/config file using your favourite text editor:

my_pc:~ $ sudo nano ~/.cargo/config

Tell Cargo to use a specific linker version for your target:

[target.arm-unknown-linux-gnueabihf]
linker = "/rpi_tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc"

How to Program an Embedded "Hello World"

Start by creating a new Rust project, and opening the main.rs file in your favourite text editor:

my-pc:~ $ cargo new blink
my-pc:~ $ cd blink
my-pc:~/blink $ nano src/main.rs

Import the rust_gpiozero crate, and program an LED to alternate between on and off every second:

use rust_gpiozero::*;

fn main() {
    // Create a new LED attached to Pin 17
    let mut led = LED::new(17);

    led.blink(1.0, 1.0);

    led.wait();
}

Be sure to add the dependency to the Cargo.toml file:

[dependencies]
rust-gpiozero = "0.2.1"

Successfully Exit the Program

Since rustc 1.61.0 ^[1], you can use the std::process::ExitCode struct to specify the status code returned to the process' parent:

use std::process::ExitCode;
fn main() -> ExitCode {
    // ...
    if error {
      return ExitCode::from(1);
    }
    ExitCode::SUCCESS
}

Otherwise, you can simply return a Result:

fn main() -> Result<(), std::io::Error> {
  // ...
  Ok(())
}

How to Cross Compile the Program

Build a release of your program, targeting the required architecture:

my-pc:~/blink $ cargo build --release --target=arm-unknown-linux-gnueabihf

How to Transfer the Binary to the Pi

Use scp to transfer the compiled binary from your host computer to the Raspberry Pi over SSH:

my-pc:~/blink $ scp target/arm-unknown-linux-gnueabihf/release/blink pi@192.168.1.199:~/blink

Note: The local IP of your Pi will likely be different.

How to SSH into the Pi

SSH and log in to the Raspberry Pi via its local IP address:

my-pc:~ $ ssh pi@192.168.1.199

Run the Program

From the Raspberry Pi, run the compiled binary:

pi:~ $ ./blink

How to Code a Text-to-Morse-Code Translator

Here is an example of an application that reads the stdin line by line, translates the input into Morse Code, and toggles the LED on and off based on the Morse Code for the characters.

use rust_gpiozero::*;
use std::io::{BufRead, self};
use std::collections::HashMap;
use std::thread::sleep;
use std::time::Duration;

fn main() -> Result<(), std::io::Error> {
    println!("Starting...\n- Type in text to turn into Morse Code\n- Type `quit()` to quit\n");
    // Create a new LED attached to Pin 17
    let led = LED::new(17);

    /// Length of a dot in milliseconds
    const DOT_DELAY: u64 = 80;
    /// Length of a dash in milliseconds
    const DASH_DELAY: u64 = DOT_DELAY * 3;
    /// Delay between inputs in milliseconds
    const PUSH_DELAY: u64 = DOT_DELAY;
    /// Delay between letters in milliseconds
    const LETTER_DELAY: u64 = DOT_DELAY * 3;
    /// Delay between words in milliseconds
    const WORD_DELAY: u64 = DOT_DELAY * 7;

    let morse_code_alphabet: HashMap<char, &'static str> =
    [
        ('a', ".-"),
        ('b', "-..."),
        ('c', "-.-."),
        ('d', "-.."),
        ('e', "."),
        ('f', "..-."),
        ('g', "--."),
        ('h', "...."),
        ('i', ".."),
        ('j', ".---"),
        ('k', "-.-"),
        ('l', ".-.."),
        ('m', "--"),
        ('n', "-."),
        ('o', "---"),
        ('p', ".--."),
        ('q', "--.-"),
        ('r', ".-."),
        ('s', "..."),
        ('t', "-"),
        ('u', "..-"),
        ('v', "...-"),
        ('w', ".--"),
        ('x', "-..-"),
        ('y', "-.--"),
        ('z', "--.."),
        ('1', ".----"),
        ('2', "..---"),
        ('3', "...--"),
        ('4', "....-"),
        ('5', "....."),
        ('6', "-...."),
        ('7', "--..."),
        ('8', "---.."),
        ('9', "----."),
        ('0', "-----"),
        ('.', ".-.-.-"),
        (',', "--..--"),
        ('?', "..--.."),
        ('\'', ".----."),
        ('!', "-.-.--"),
        ('/', "-..-."),
        ('(', "-.--."),
        (')', "-.--.-"),
        ('&', ".-..."),
        (':', "---..."),
        (';', "-.-.-."),
        ('=', "-...-"),
        ('+', ".-.-."),
        ('-', "-....-"),
        ('_', "..--.-"),
        ('"', ".-..-."),
        ('$', "...-..-"),
        ('@', ".--.-."),
        (' ', " "),
    ].iter().cloned().collect();

    // Read standard input per line
    for line_res in io::stdin().lock().lines() {
        let line = line_res?;
        if line == "quit()" {
            break;
        }
        // Turn line into morse code
        let mut morse = String::new();
        for c in line.chars() {
            if let Some(morse_code_char) = morse_code_alphabet.get(&c) {
                morse.push_str(morse_code_char);
                // Separate characters with a comma
                morse.push_str(",");
            }
        }
        // Blink LED based on characters
        for c in morse.chars() {
            match c {
                '.' => {
                    led.on();
                    sleep(Duration::from_millis(DOT_DELAY));
                    led.off();
                    sleep(Duration::from_millis(PUSH_DELAY));
                },
                '-' => {
                    led.on();
                    sleep(Duration::from_millis(DASH_DELAY));
                    led.off();
                    sleep(Duration::from_millis(PUSH_DELAY));
                },
                ',' => {
                    sleep(Duration::from_millis(LETTER_DELAY));
                },
                ' ' => {
                    sleep(Duration::from_millis(WORD_DELAY));
                },
                _ => {
                    println!("Unknown character: {}", c);
                    break;
                }
            }
        }
        sleep(Duration::from_millis(WORD_DELAY));
    }

    // Free the variable and associated resources
    led.close();

    Ok(())
}

Appendix

Targets

In Rust, the target is the platform (architecture) the program is compiled for. Cargo automatically detects the target, based on the file system layout ^[2].

You can see the list of built-in targets, by running:

rustc --print target-list
# OR
rustup target list

From here you can add a new target to your project, by running:

rustup target add <target>

The given target is often in the form of a triple ^[3]:

• The architecture
• The vendor
• The operating system type
• The environment type

This is refered to as a 'target triple', because the fourth part is optional.

Kismet is perfect for detecting anomalies and certain types of attack – but what if I want to analyze the traffic and look for abnormal patterns or patterns that could indicate an attack?

And Intrusion Detection System (IDS) is:

...a device or software application that monitors a network or systems for malicious activity or policy violations.

I used a good IDS in the past called Snort V2, I'm aware than Snort 3 is out. But there is a pretty clear warning about running it on a machine without much memory:

While Snort can compile on almost all *nix based machines, it is not recommended that you compile Snort on a low power or low RAM machine. Snort requires memory to run and to properly analyze as much traffic as possible.

And

Snort does not officially support any particular OS.

Not exactly a reason to dislike it, but I feel more confident when a vendor tells me than my OS is in their supported platform list. I do also have more recent experience setting up with the open source tool Suricata, so I decided to give it a more serious try to keep tabs on my local network and alert me if any suspicious activity was detected.

Poking around I found than for my local network, 8 GB of RAM will be sufficient along with my Linux distribution:

josevnz@raspberrypi:~$ lsb_release --release
Release:    20.04

My version of Ubuntu is supported out of the box.

The choice is yours. In my case it felt better to use Suricata than Snort. As usual, you need to plan around your hardware, your use cases, and the features offered by the tools (including commercial support).

Table of Contents

• Quick Installation

• Where you should connect your Raspberry Pi 4 with Suricata

• How to Set Up Suricata

• How to Tune Up Suricata

• Making Sense of All the Alerts

• What Did We Learn and What is Next?

Quick Installation

Installation is explained in detail here, so I will only put here the quick installation steps I used on my machine:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

Suricata is a Complex Beast

You can use Suricata to detect and alert you about anomalies in your network traffic (IDS) or you can proactively drop suspicious connections when working in Intrusion Prevention System (IPS).

It can also capture network traffic and store it in PCAP format for later analysis (be careful as you can eat your disk space pretty fast).

We will keep things simple, and for now will take a more passive approach and get alerts when an intrusion is detected (sticking to IDS mode) in this tutorial.

Where you should connect your RaspBerryPI 4 with Suricata?

Ideally you want to put your Suricata sensor close to your home router. One way to do it is to connect all the devices (including your home router) to a common switch, and then mirror the traffic that goes into/out from the home router into a port on the switch. Suricata will be connected to that port, listening to all the traffic.

If you wanted to run Suricata as an IPS then the connectivity would have to be different, but this is not the intended use in this tutorial.

How to Set Up Suricata

Ideally the best place to put Suricata is between a firewall and the rest of the servers in your home network.

In this scenario let's assume than it is not possible because there is no firewall (OK, that will be your ISP router, but you cannot run Suricata there). So the next best thing is the wired network interface connected to it (in my case eth0).

The /etc/suricata/suricata.yaml file contains the defaults. I'll show here what I overrode:

root@raspberrypi:~# grep -in1 af-p /etc/suricata/suricata.yaml 
580-# Linux high speed capture support
581:af-packet:
582-  - interface: eth0
root@raspberrypi:~# grep -in 'HOME_NET: "' /etc/suricata/suricata.yaml |grep -v '#'
15:    HOME_NET: "[192.168.1.0/24]"

Start Suricata:

root@raspberrypi:~# systemctl start suricata.service
root@raspberrypi:~# systemctl status suricata.service
● suricata.service - LSB: Next Generation IDS/IPS
     Loaded: loaded (/etc/init.d/suricata; generated)
     Active: active (running) since Sun 2022-04-10 23:49:00 UTC; 24h ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 10 (limit: 9257)
     CGroup: /system.slice/suricata.service
             └─1834983 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -vvv

Apr 10 23:49:00 raspberrypi systemd[1]: Starting LSB: Next Generation IDS/IPS...
Apr 10 23:49:00 raspberrypi suricata[1834973]: Starting suricata in IDS (af-packet) mode... done.
Apr 10 23:49:00 raspberrypi systemd[1]: Started LSB: Next Generation IDS/IPS.

The important details go into the file '/var/log/suricata/eve.json'. Mine started to grow surprisingly fast after starting Suricata:

{"timestamp":"2022-04-10T23:49:32.527488+0000","event_type":"stats","stats":{"uptime":32,"capture":{"kernel_packets":113,"kernel_drops":0,"errors":0},"decoder":{"pkts":126,"bytes":17986,"invalid":0,"ipv4":30,"ipv6":74,"ethernet":126,"chdlc":0,"raw":0,"null":0,"sll":0,"tcp":4,"udp":30,"sctp":0,"icmpv4":0,"icmpv6":70,"ppp":0,"pppoe":0,"geneve":0,"gre":0,"vlan":0,"vlan_qinq":0,"vxlan":0,"vntag":0,"ieee8021ah":0,"teredo":0,"ipv4_in_ipv6":0,"ipv6_in_ipv6":0,"mpls":0,"avg_pkt_size":142,"max_pkt_size":392,"max_mac_addrs_src":0,"max_mac_addrs_dst":0,"erspan":0,"event":{"ipv4":{"pkt_too_small":0,"hlen_too_small":0,"iplen_smaller_than_hlen":0,"trunc_pkt":0,"opt_invalid":0,"opt_invalid_len":0,"opt_malformed":0,"opt_pad_required":0,"opt_eol_required":0,"opt_duplicate":0,"opt_unknown":0,"wrong_ip_version":0,"icmpv6":0,"frag_pkt_too_large":0,"frag_overlap":0,"frag_ignored":0},"icmpv4":{"pkt_too_small":0,"unknown_type":0,"unknown_code":0,"ipv4_trunc_pkt":0,"ipv4_unknown_ver":0},"icmpv6":{"unknown_type":0,"unknown_code":0,"pkt_too_small":0,"ipv6_unknown_version":0,"ipv6_trunc_pkt":0,"mld_message_with_invalid_hl":0,"unassigned_type":0,"experimentation_type":0},"ipv6":{"pkt_too_small":0,"trunc_pkt":0,"trunc_exthdr":0,"exthdr_dupl_fh":0,"exthdr_useless_fh":0,"exthdr_dupl_rh":0,"exthdr_dupl_hh":0,"exthdr_dupl_dh":0,"exthdr_dupl_ah":0,"exthdr_dupl_eh":0,"exthdr_invalid_optlen":0,"wrong_ip_version":0,"exthdr_ah_res_not_null":0,"hopopts_unknown_opt":0,"hopopts_only_padding":0,"dstopts_unknown_opt":0,"dstopts_only_padding":0,"rh_type_0":0,"zero_len_padn":21,"fh_non_zero_reserved_field":0,"data_after_none_header":0,"unknown_next_header":0,"icmpv4":0,"frag_pkt_too_large":0,"frag_overlap":0,"frag_invalid_length":0,"frag_ignored":0,"ipv4_in_ipv6_too_small":0,"ipv4_in_ipv6_wrong_version":0,"ipv6_in_ipv6_too_small":0,"ipv6_in_ipv6_wrong_version":0},"tcp":{"pkt_too_small":0,"hlen_too_small":0,"invalid_optlen":0,"opt_invalid_len":0,"opt_duplicate":0},"udp":{"pkt_too_small":0,"hlen_too_small":0,"hlen_invalid":0},"sll":{"pkt_too_small":0},"ethernet":{"pkt_too_small":0},"ppp":{"pkt_too_small":0,"vju_pkt_too_small":0,"ip4_pkt_too_small":0,"ip6_pkt_too_small":0,"wrong_type":0,"unsup_proto":0},"pppoe":{"pkt_too_small":0,"wrong_code":0,"malformed_tags":0},"gre":{"pkt_too_small":0,"wrong_version":0,"version0_recur":0,"version0_flags":0,"version0_hdr_too_big":0,"version0_malformed_sre_hdr":0,"version1_chksum":0,"version1_route":0,"version1_ssr":0,"version1_recur":0,"version1_flags":0,"version1_no_key":0,"version1_wrong_protocol":0,"version1_malformed_sre_hdr":0,"version1_hdr_too_big":0},"vlan":{"header_too_small":0,"unknown_type":0,"too_many_layers":0},"ieee8021ah":{"header_too_small":0},"vntag":{"header_too_small":0,"unknown_type":0},"ipraw":{"invalid_ip_version":0},"ltnull":{"pkt_too_small":0,"unsupported_type":0},"sctp":{"pkt_too_small":0},"mpls":{"header_too_small":0,"pkt_too_small":0,"bad_label_router_alert":0,"bad_label_implicit_null":0,"bad_label_reserved":0,"unknown_payload_type":0},"vxlan":{"unknown_payload_type":0},"geneve":{"unknown_payload_type":0},"erspan":{"header_too_small":0,"unsupported_version":0,"too_many_vlan_layers":0},"dce":{"pkt_too_small":0},"chdlc":{"pkt_too_small":0}},"too_many_layers":0},"flow":{"memcap":0,"tcp":1,"udp":20,"icmpv4":0,"icmpv6":15,"tcp_reuse":0,"get_used":0,"get_used_eval":0,"get_used_eval_reject":0,"get_used_eval_busy":0,"get_used_failed":0,"wrk":{"spare_sync_avg":100,"spare_sync":4,"spare_sync_incomplete":0,"spare_sync_empty":0,"flows_evicted_needs_work":0,"flows_evicted_pkt_inject":0,"flows_evicted":0,"flows_injected":0},"mgr":{"full_hash_pass":1,"closed_pruned":0,"new_pruned":0,"est_pruned":0,"bypassed_pruned":0,"rows_maxlen":1,"flows_checked":4,"flows_notimeout":4,"flows_timeout":0,"flows_timeout_inuse":0,"flows_evicted":0,"flows_evicted_needs_work":0},"spare":9600,"emerg_mode_entered":0,"emerg_mode_over":0,"memuse":11668608},"defrag":{"ipv4":{"fragments":0,"reassembled":0,"timeouts":0},"ipv6":{"fragments":0,"reassembled":0,"timeouts":0},"max_frag_hits":0},"flow_bypassed":{"local_pkts":0,"local_bytes":0,"local_capture_pkts":0,"local_capture_bytes":0,"closed":0,"pkts":0,"bytes":0},"tcp":{"sessions":0,"ssn_memcap_drop":0,"pseudo":0,"pseudo_failed":0,"invalid_checksum":0,"no_flow":0,"syn":0,"synack":0,"rst":0,"midstream_pickups":0,"pkt_on_wrong_thread":0,"segment_memcap_drop":0,"stream_depth_reached":0,"reassembly_gap":0,"overlap":0,"overlap_diff_data":0,"insert_data_normal_fail":0,"insert_data_overlap_fail":0,"insert_list_fail":0,"memuse":2424832,"reassembly_memuse":393216},"detect":{"engines":[{"id":0,"last_reload":"2022-04-10T23:49:00.377030+0000","rules_loaded":0,"rules_failed":0}],"alert":0},"app_layer":{"flow":{"http":0,"ftp":0,"smtp":0,"tls":0,"ssh":0,"imap":0,"smb":0,"dcerpc_tcp":0,"dns_tcp":0,"nfs_tcp":0,"ntp":1,"ftp-data":0,"tftp":0,"ikev2":0,"krb5_tcp":0,"dhcp":0,"snmp":0,"sip":0,"rfb":0,"mqtt":0,"rdp":0,"failed_tcp":0,"dcerpc_udp":0,"dns_udp":0,"nfs_udp":0,"krb5_udp":0,"failed_udp":19},"tx":{"http":0,"ftp":0,"smtp":0,"tls":0,"ssh":0,"imap":0,"smb":0,"dcerpc_tcp":0,"dns_tcp":0,"nfs_tcp":0,"ntp":1,"ftp-data":0,"tftp":0,"ikev2":0,"krb5_tcp":0,"dhcp":0,"snmp":0,"sip":0,"rfb":0,"mqtt":0,"rdp":0,"dcerpc_udp":0,"dns_udp":0,"nfs_udp":0,"krb5_udp":0},"expectations":0},"http":{"memuse":0,"memcap":0},"ftp":{"memuse":0,"memcap":0},"file_store":{"open_files":0}}}

Holy Priceless Collection of Etruscan Snoods!, Batman. How do we tune Suricata to avoid this overwhelming amount of information?

For now let's stop it while we figure it out.

How to Tune Up Suricata

Make sure the settings of suricata.yaml make sense for a home network:

sudo -i
# And a YAML linter so we can make sure our Suricata configuration files are good
apt-get install yamllint
cp -v -p  /etc/suricata/suricata.yaml /etc/suricata/suricata.yaml.orig

Note that I provide here a linted and clean version of my [suricata.yaml](file:///home/josevnz/SuricataLog/etc/suricata/suricata.yaml) file.

How to tame the /var/log/suricata/eve.json file

This is the file were we can learn in detail what triggered an alert. But it can grow VERY fast, depending on your traffic and event rules configuration.

So using logrotate (comes installed as part of Ubuntu), do this:

# Keep a week of logs, 1 GB of size.
# Always test your config: logrotate -vdf /etc/logrotate.d/suricata
/var/log/suricata/*.log /var/log/suricata/*.json {
    daily
    maxsize 1G
    rotate 7
    missingok
    nocompress
    create
    sharedscripts
    postrotate
        systemctl restart suricata.service
    endscript
}

How to help Suricata to do its job using emerging threats rules

We can tune Suricata using the ET OPEN Ruleset. Because threats change all the time, you need to automate their download and updating.

So install it first:

sudo -i
python3 -m venv ~/virtualenv/suricata
. ~/virtualenv/suricata/bin/activate
pip install --upgrade pip
pip install --upgrade suricata-update
suricata-update
# Also, install jq so we can see the contents of the eve.json file nicely formatted
apt-get install jq

Let's run it by hand and see how the rules are updated by the tool:

For our home network, we will download these rules once a day. A simple Cron job will do the trick:

crontab -e
# Run Suricata update once a day, 
# per https://rules.emergingthreats.net/OPEN_download_instructions.html
# Also will update at a different time than the log rotation, to avoid a race condition
# while rotating the logs. Note than we do not need to restart suricata
0 30 * * * . ~/virtualenv/suricata/bin/activate && suricata-update && suricatasc -c reload-rules

Let's start Suricata again, so we can test some rules:

What is inside the /var/log/suricata/eve.json file?

The file packs quite a bit of information, which is described in detail here:

{"timestamp":"2022-04-15T20:52:05.026189+0000","flow_id":1378250082748552,"in_iface":"eth0","event_type":"flow","src_ip":"192.168.1.1","src_port":59317,"dest_ip":"239.255.255.250","dest_port":1900,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":378,"bytes_toclient":0,"start":"2022-04-15T20:50:32.264328+0000","end":"2022-04-15T20:50:32.264328+0000","age":0,"state":"new","reason":"timeout","alerted":false}}
{"timestamp":"2022-04-15T20:52:05.026418+0000","flow_id":2222739437411106,"in_iface":"eth0","event_type":"flow","src_ip":"192.168.1.1","src_port":60890,"dest_ip":"239.255.255.250","dest_port":1900,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":376,"bytes_toclient":0,"start":"2022-04-15T20:50:32.482082+0000","end":"2022-04-15T20:50:32.482082+0000","age":0,"state":"new","reason":"timeout","alerted":false}}

If you are casually inspecting the contents of the file in real time, I suggest you use jq (test your filters on jqplay.org) and show a few fields of interest:

Going forward we will focus on the alerts, so we can just filter out by that type of event:

jq 'select(.event_type=="alert")' /var/log/suricata/eve.json

The Suricata folks have put together a nice page with examples that you should check out.

How to test Suricata installation

Tools of the trade: Wireshark, tcpreplay, and PCAP files

We will use some traffic capture files, in PCAP format. So what is a PCAP file?

In the late 1980's, Van Jacobson, Steve McCanne, and others at the Network Research Group at Lawrence Berkeley National Laboratory developed the tcpdump program to capture and dissect network traces.

The code to capture traffic, using low-level mechanisms in various operating systems, and to read and write network traces to a file was later put into a library named libpcap.

And we will use a tool to inspect the contents of the PCAP file. Wireshark is a powerful traffic analysis tool, and we will use tcpreplay to trigger the Suricata alerts by playing a PCAP file with suspicious activity:

# On Ubuntu, Debian: sudo apt-get install wireshark tcpreplay
sudo dnf install -y wireshark tcpreplay

The best way to learn how the bad actors operate is to see their footprints. You should definitely head to https://www.malware-traffic-analysis.net/ and download some samples, an even better practice with their PCAP analysis exercises.

WARNING: You will be downloading files that are dangerous:

Use this website at your own risk! If you download or use of any information from this website, you assume complete responsibility for any resulting loss or damage.

So be careful and responsible when using this network traffic capturer.

No rules are enabled by default?

How we can check if that is the case? I'll show you next:

Once you enable the rules (suricata-update list-sources --free; uricata-update enable-source source; suricata-update list-enabled-sources) you can tell Suricata to reload the rules without a reboot:

root@raspberrypi:~# suricatasc -c reload-rules
{"message": "done", "return": "OK"}

2022-02-23 - TRAFFIC ANALYSIS EXERCISE - SUNNYSTATION

Let's see if we can trigger Suricata using this specific threat (it is relative new).

Start by downloading 2022-02-23-traffic-analysis-exercise.pcap.zip (the password is on the [about page](file:///home/josevnz/SuricataLog/)).

insta_dir="$HOME/Downloads/malware/"
mkdir --parent --verbose "$insta_dir"
url="https://www.malware-traffic-analysis.net/2022/02/23/2022-02-23-traffic-analysis-exercise.pcap.zip"
exercise=$(basename $url)
curl --fail --location --output "$insta_dir/$exercise" $url
# Be ready to put the password :-)
cd $insta_dir && unzip $exercise

What is inside? We can check with capinfos to get some insight on the file we just downloaded:

[josevnz@dmaf5 malware]$ capinfos 2022-02-23-traffic-analysis-exercise.pcap
File name:           2022-02-23-traffic-analysis-exercise.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
File timestamp precision:  microseconds (6)
Packet size limit:   file hdr: 65535 bytes
Number of packets:   30k
File size:           19MB
Data size:           19MB
Capture duration:    2680.736661 seconds
First packet time:   2022-02-23 13:22:24.405139
Last packet time:    2022-02-23 14:07:05.141800
Data byte rate:      7,191 bytes/s
Data bit rate:       57kbps
Average packet size: 642.09 bytes
Average packet rate: 11 packets/s
SHA256:              eefc7e61b50e7846f5a3282d7645539d7b2b4b85aa08a09d0b823896c9449d1f
RIPEMD160:           a8d84d262e37563c179e9ca52cdc6aae271efd9c
SHA1:                fdfa0d0edfe0cbcc0c1400fbe6ac61ff40942755
Strict time order:   True
Number of interfaces in file: 1
Interface #0 info:
                     Encapsulation = Ethernet (1 - ether)
                     Capture length = 65535
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Number of stat entries = 0
                     Number of packets = 30023

Will use a [small wrapper](file:///home/josevnz/SuricataLog/scripts/replay_pcap_file.sh) around tcpreplay to replay our PCAP file:

#!/bin/bash
:<<DOC
Script to replay a PCAP file at accelerated pace on the default network interface
Author: Jose Vicente Nunez (kodegeek.com@protonmail.com)
DOC
default_dev=$(ip route show| grep default| sort -n -k 9| head -n 1| cut -f5 -d' ')|| exit 100
if [ "$(id --name --user)" != "root" ]; then
  echo "ERROR: I need to be root to inject the PCAP contents into '$default_dev'"
  echo "Maybe 'sudo $0 $*'?"
  exit 100
fi
for util in tcpreplay ip; do
  if ! type -p $util > /dev/null 2>&1; then
    echo "Please put $util on the PATH and try again!"
    exit 100
  fi
done
:<<DOC
We may have more than one 'default' route, so we sort by priority and pick the one with the
preferred metric:
default via 192.168.1.1 dev eno1 proto dhcp metric 100 <----- PICK ME!!!
default via 192.168.1.1 dev wlp4s0 proto dhcp metric 600
DOC
for pcap in "$@"; do
  if [ -f "$pcap" ]; then
    if ! tcpreplay --stats 5 --intf1 "$default_dev" --multiplier 24 "$pcap"; then
      echo "ERROR: Will not try to replay any pending PCAP files due previous errors"
      exit 100
    fi
  fi
done

Let it replay until it reaches the end of the file:

root@raspberrypi:~# tcpreplay --stats 5 --intf1 eth0 --multiplier 24 ~josevnz/Downloads/malware/2022-02-23-traffic-analysis-exercise.pcap 
Test start: 2022-04-16 17:51:40.673394 ...
Actual: 3783 packets (1075843 bytes) sent in 5.03 seconds
Rated: 213624.5 Bps, 1.70 Mbps, 751.17 pps
Actual: 6959 packets (3325918 bytes) sent in 10.04 seconds
Rated: 331191.4 Bps, 2.64 Mbps, 692.96 pps
Actual: 8627 packets (4464002 bytes) sent in 15.14 seconds
Rated: 294744.2 Bps, 2.35 Mbps, 569.61 pps
Actual: 10975 packets (6331901 bytes) sent in 20.21 seconds
Rated: 313180.5 Bps, 2.50 Mbps, 542.83 pps
Actual: 13148 packets (7870783 bytes) sent in 25.26 seconds
Rated: 311561.9 Bps, 2.49 Mbps, 520.45 pps
Actual: 14500 packets (8612630 bytes) sent in 30.43 seconds
...
Actual: 24467 packets (14960314 bytes) sent in 110.83 seconds
Rated: 134978.5 Bps, 1.07 Mbps, 220.75 pps
Test complete: 2022-04-16 17:53:33.735188
Actual: 30023 packets (19277433 bytes) sent in 113.06 seconds
Rated: 170503.5 Bps, 1.36 Mbps, 265.54 pps
Statistics for network device: eth0
    Successful packets:        30023
    Failed packets:            0
    Truncated packets:         0
    Retried packets (ENOBUFS): 0
    Retried packets (EAGAIN):  0

And eventually we get a few alerts:

"2022-04-16T17:52:20.134763+0000,dns,1296231906414153,172.16.0.170:53806,172.16.0.52:53"
"2022-04-16T17:52:20.286785+0000,dns,293726410006593,172.16.0.170:50935,172.16.0.52:53"
"2022-04-16T17:52:20.290084+0000,dns,293726410006593,172.16.0.170:50935,172.16.0.52:53"
"2022-04-16T17:52:20.520858+0000,alert,1626224981242326,172.16.0.149:49795,172.16.0.52:139"
"2022-04-16T17:52:21.784804+0000,alert,1992149752477936,172.16.0.149:49796,172.16.0.52:139"
"2022-04-16T17:52:22.142041+0000,flow,1739064507071469,172.16.0.149:5353,224.0.0.251:5353"
"2022-04-16T17:52:22.351091+0000,dns,2078727703255923,172.16.0.149:51367,172.16.0.52:53"
"2022-04-16T17:52:22.351260+0000,dns,181632058678300,172.16.0.149:64943,172.16.0.52:53"
"2022-04-16T17:52:22.351129+0000,dns,2078727703255923,172.16.0.149:51367,172.16.0.52:53"
"2022-04-16T17:52:23.037637+0000,alert,282956779721256,172.16.0.149:49798,172.16.0.52:139"
"2022-04-16T17:52:23.901721+0000,dns,556717995180633,172.16.0.170:51164,172.16.0.52:53"
"2022-04-16T17:52:23.904764+0000,dns,556717995180633,172.16.0.170:51164,172.16.0.52:53"
"2022-04-16T17:52:24.293356+0000,alert,2006941620009246,172.16.0.149:49799,172.16.0.52:139"
"2022-04-16T17:52:25.322102+0000,dns,1671081620007478,172.16.0.170:51909,172.16.0.52:53"

For sake of example, zoom in alert id '282956779721256':

// root@raspberrypi:~# grep 282956779721256 /var/log/suricata/eve.json|jq
{
  "timestamp": "2022-04-16T17:52:23.037637+0000",
  "flow_id": 282956779721256,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "172.16.0.149",
  "src_port": 49798,
  "dest_ip": "172.16.0.52",
  "dest_port": 139,
  "proto": "TCP",
  "metadata": {
    "flowints": {
      "applayer.anomaly.count": 1
    }
  },
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2260002,
    "rev": 1,
    "signature": "SURICATA Applayer Detect protocol only one direction",
    "category": "Generic Protocol Command Decode",
    "severity": 3
  },
  "smb": {
    "id": 1,
    "dialect": "NT LM 0.12",
    "command": "SMB1_COMMAND_NEGOTIATE_PROTOCOL",
    "status": "STATUS_SUCCESS",
    "status_code": "0x0",
    "session_id": 0,
    "tree_id": 0,
    "client_dialects": [
      "PC NETWORK PROGRAM 1.0",
      "LANMAN1.0",
      "Windows for Workgroups 3.1a",
      "LM1.2X002",
      "LANMAN2.1",
      "NT LM 0.12"
    ],
    "server_guid": "a21b9552-a4a0-48cd-8abb-ea111498253d"
  },
  "app_proto": "smb",
  "app_proto_ts": "failed",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 3,
    "bytes_toserver": 579,
    "bytes_toclient": 387,
    "start": "2022-04-16T17:52:23.037416+0000"
  },
  "payload": "AAAAiv9TTUJzAAAAABgHyAAAQlNSU1BZTCAAAP////4AAEAADP8AAAAEQTIAAAAAAAAASgAAAAAA1AAAoE8AYEgGBisGAQUFAqA+MDygDjAMBgorBgEEAYI3AgIKoioEKE5UTE1TU1AAAQAAAJeCCOIAAAAAAAAAAAAAAAAAAAAACgBhSgAAAA8AAAAAAA==",
  "payload_printable": ".....SMBs.........BSRSPYL ........@.......A2.......J.........O.`H..+......>0<..0..\n+.....7..\n.*.(NTLMSSP.........................\n.aJ.........",
  "stream": 0,
  "packet": "AB5PDqh0ABv8e9HACABFAAC2t+tAAIAG6WysEACVrBAANMKGAIthfGQf7GIEdVAYIBP6YwAAAAAAiv9TTUJzAAAAABgHyAAAQlNSU1BZTCAAAP////4AAEAADP8AAAAEQTIAAAAAAAAASgAAAAAA1AAAoE8AYEgGBisGAQUFAqA+MDygDjAMBgorBgEEAYI3AgIKoioEKE5UTE1TU1AAAQAAAJeCCOIAAAAAAAAAAAAAAAAAAAAACgBhSgAAAA8AAAAAAA==",
  "packet_info": {
    "linktype": 1
  },
  "host": "ras[berripi"
}
{
  "timestamp": "2022-04-16T17:55:42.050329+0000",
  "flow_id": 282956779721256,
  "in_iface": "eth0",
  "event_type": "flow",
  "src_ip": "172.16.0.149",
  "src_port": 49798,
  "dest_ip": "172.16.0.52",
  "dest_port": 139,
  "proto": "TCP",
  "app_proto": "smb",
  "app_proto_ts": "failed",
  "flow": {
    "pkts_toserver": 13,
    "pkts_toclient": 12,
    "bytes_toserver": 1743,
    "bytes_toclient": 1963,
    "start": "2022-04-16T17:52:23.037416+0000",
    "end": "2022-04-16T17:52:23.488633+0000",
    "age": 0,
    "state": "closed",
    "reason": "timeout",
    "alerted": true
  },
  "metadata": {
    "flowbits": [
      "smb.tree.connect.ipc"
    ],
    "flowints": {
      "applayer.anomaly.count": 1
    }
  },
  "tcp": {
    "tcp_flags": "1b",
    "tcp_flags_ts": "1b",
    "tcp_flags_tc": "1b",
    "syn": true,
    "fin": true,
    "psh": true,
    "ack": true,
    "state": "closed"
  },
  "host": "raspberrypi"
}

That's quite a bit to process. Keep in mind that while we are tuning Suricata, we can also ask it to replay one or more PCAP file directly.

Ask Suricata to run in offline mode using PCAP file for SUNNYSTATION

It is a very convenient way to test Suricata, as we do not inject any traffic in our network and instead let Suricata 'ingest' the contents of the PCAP file directly, to test the rules.

Also, we redirect the logs to a separate location (by default the directory where you are running the 'offline' mode), so we don't pollute a live installation.

Another example: EMOTET WITH COBALT STRIKE

Let's try another malware capture, in this case 2022-02-08 (TUESDAY) - FILES FOR AN ISC DIARY (EMOTET WITH COBALT STRIKE):

cd ~/Downloads/malware/ && \
curl --remote-name https://www.malware-traffic-analysis.net/2022/02/08/2022-02-08-Emotet-epoch4-infection-start-and-spambot-traffic.pcap.zip && \
unzip 2022-02-08-Emotet-epoch4-infection-start-and-spambot-traffic.pcap.zip && \
sudo suricata -r ~josevnz/Downloads/malware/2022-02-08-Emotet-epoch4-infection-start-and-spambot-traffic.pcap -k none --runmode autofp -c /etc/suricata/suricata.yaml -l ~josevnz/Downloads/malware/

Here is a sample session:

Making Sense of All the Alerts

Suricata will save lots of details when it detects an anomaly. You can tell that using jq to go through the alerts may not be desirable.

For a bigger setup, you may want to use an Elastic Stack (Filebeat, Logstash, Elastic Search, Kibana):

• Get the logs

• Store historically and normalize the logs

• Visualize their contents

But that feels overkill for a home setup, so I will roll out a few scripts to help me with what I need.

Show me what happened in the last 10 minutes

This is a script that assumes most of the defaults, so I don't have to type a jq expression. If there are any alerts then I dive deeper into the eve.json file.

A simple Python 3 script will do the trick for us:

#!/usr/bin/env python
"""
Show Suricata alerts
Author: Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import argparse
import json
from datetime import datetime, timedelta
from json import JSONDecodeError
from pathlib import Path
from typing import Callable, Any, Dict

DEFAULT_EVE = [Path("/var/log/suricata/eve.json")]
DEFAULT_TIMESTAMP_10M_AGO = datetime = datetime.now() - timedelta(minutes=10)


def _parse_timestamp(candidate: str) -> datetime:
    """
    Expected something like 2022-02-08T16:32:14.900292+0000
    :param candidate:
    :return:
    """
    if isinstance(candidate, str):
        try:
            iso_candidate = candidate.split('+', 1)[0]
            return datetime.fromisoformat(iso_candidate)
        except ValueError:
            raise ValueError(f"Invalid date passed: {candidate}")
    elif isinstance(candidate, datetime):
        return candidate


def alert_filter(
        *,
        timestamp: datetime = DEFAULT_TIMESTAMP_10M_AGO,
        data: Dict[str, Any]
) -> bool:
    if 'event_type' not in data:
        return False
    if data['event_type'] != 'alert':
        return False
    try:
        event_timestamp = _parse_timestamp(data['timestamp'])
        if event_timestamp > timestamp:
            return False
    except ValueError:
        return False
    return True


def get_alerts(
        *,
        eve_files=None,
        row_filter: Callable = alert_filter,
        timestamp: datetime = DEFAULT_TIMESTAMP_10M_AGO
) -> str:
    if eve_files is None:
        eve_files = DEFAULT_EVE
    for eve_file in eve_files:
        with open(eve_file, 'rt') as eve:
            for line in eve:
                try:
                    data = json.loads(line)
                    if row_filter(data=data, timestamp=timestamp):
                        yield data
                except JSONDecodeError:
                    continue  # Try to read the next record


if __name__ == "__main__":
    PARSER = argparse.ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "--timestamp",
        type=_parse_timestamp,
        default=DEFAULT_TIMESTAMP_10M_AGO,
        help=f"Minimum timestamp in the past to use when filtering events ({DEFAULT_TIMESTAMP_10M_AGO})"
    )
    PARSER.add_argument(
        'eve',
        type=Path,
        nargs="+",
        help=f"Path to one or more {DEFAULT_EVE[0]} file to parse."
    )
    OPTIONS = PARSER.parse_args()
    try:
        for alert in get_alerts(eve_files=OPTIONS.eve, timestamp=OPTIONS.timestamp):
            print(json.dumps(alert, indent=6, sort_keys=True))
    except KeyboardInterrupt:
        pass

It is a big improvement over jq as at least we can filter by timestamp, but it would be nice if our script could do the following:

1. Support pagination

2. Colorize output

3. Let you show between a table format or raw JSON output

What Did We Learn and What is Next?

Suricata is a complex piece of software. It takes time to tame it and more time to make sense of the information it presents. But it is very rewarding to see how you can tackle a tool that will allow you to secure your network from threats.

• The OISF Suricata YouTube channel has many interesting resources about this tool and a thriving community.

• Want to learn how to analyze PCAP files for bad traffic? malware-traffic-analysis has perfect material for you.

• Writing complex software is hard. For example, older versions of Snort are vulnerable to an attack that can disable it, CVE-2022-20685. Suricata also had CVE-2019-1010279 .These issues were fixed but illustrates the need to keep your software current, specially the one you use to protect your network.

• I did not touch the IPS mode, or even hybrid modes for Suricata. Please read the official documentation to get up to speed.

• Finally, do yourself a favor and read this Suricata Tutorial from FloCon 2016. It is very complete and will have you looking for more.

You can leave your comments on the Git repository and report any bugs. But more important get Suricata, get the code of this tutorial, and start securing your home wireless infrastructure in no time.

[josevnz@dmaf5 ~]$ sudo nmap -v -n -p- -sT -sV -O --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24

So I started to wonder:

• Is my wireless network secure?

• How long would it take to an attacker to get in?

I have a Raspberry 4 with Ubuntu (focal) installed and decided to use the well-known Kismet to find out.

In this article you will learn:

• How to get a whole picture of the networks nearby you with Kismet

• How to customize Kismet using Python and the REST-API

If you are curious, this is my home Raspberry PI 4, tiny monitor and all

Table of contents

• The saying 'Ask for forgiveness, not permission' doesn't apply here

• Getting to know your hardware

• kismet

• REST-API

• What did we learn?

The saying 'Ask for forgiveness, not permission' doesn't apply here

And by that I mean that you should not be trying to eavesdrop or infiltrate a wireless network that is not yours. It is relatively easy to detect if a new unknown client joined your wireless network, and it is also illegal.

So do the right thing – use this tutorial to learn and not to break into someone else's network, OK?

Getting to know your hardware

I will jump a little ahead to show you a small issue with the Raspberry 4 integrated Wireless interface.

The Raspberry PI 4 onboard wireless card will not work out of the box as the firmware doesn't support monitor mode.

There are works to support this. Instead, I took the easy way out and ordered an external Wi-Fi dongle from CanaKit.

The CanaKit wireless card worked out of the box, and we'll see it shortly. But first let's install and play around with Kismet.

Make sure the interface is running in monitor mode

By default, the network interface will have monitor mode off:

root@raspberrypi:~# iwconfig wlan1
wlan1     IEEE 802.11  ESSID:off/any  
          Mode:Managed  Access Point: Not-Associated   Tx-Power=0 dBm   
          Retry short  long limit:2   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off

I know I will always set up my Ralink Technology, Corp. RT5370 Wireless Adapter in monitor mode, but I need to be careful as Ubuntu can swap wlan0 and wlan1 (The Broadcom adapter I want to skip is a PCI device).

The Ralink adapter is a USB adapter, so we can find out where it is:

josevnz@raspberrypi:/etc/netplan$ /bin/lsusb|grep Ralink
Bus 001 Device 004: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter

Now we need to find out what device was mapped to the Ralink adapter. With a little bit of help of the Ubuntu community I found than the Ralink adapter uses the rt2800usb driver 5370 Ralink Technology

The answer I seek is here:

josevnz@raspberrypi:~$ ls /sys/bus/usb/drivers/rt2800usb/*:1.0/net/
wlan1

So the code that does the wireless card detection looks like this:

root@raspberrypi:~#/bin/cat<<RC_LOCAL>/etc/rc.local
#!/bin/bash
usb_driver=rt2800usb
wlan=\$(/bin/ls /sys/bus/usb/drivers/\$usb_driver/*/net/)
if [ $? -eq 0 ]; then
        set -ex
        /usr/sbin/ifconfig "\$wlan" down
        /usr/sbin/iwconfig "\$wlan" mode monitor
        /usr/sbin/ifconfig "\$wlan" up
        set +ex
fi
RC_LOCAL
root@raspberrypi:~# chmod u+x /etc/rc.local && shutdown -r now "Enabling monitor mode"

Make sure the card is on monitor mode:

root@raspberrypi:~# iwconfig wlan1
iw        iwconfig  iwevent   iwgetid   iwlist    iwpriv    iwspy     
root@raspberrypi:~# iwconfig wlan1
wlan1     IEEE 802.11  Mode:Monitor  Frequency:2.412 GHz  Tx-Power=20 dBm   
          Retry short  long limit:2   RTS thr:off   Fragment thr:off
          Power Management:off

Good, let's move on with the tool setup

What is Kismet?

Kismet is:

a wireless network and device detector, sniffer, wardriving tool, and WIDS (wireless intrusion detection) framework.

Kismet installation and setup

The version that comes with the Ubuntu RaspberryPI by default is from 2016, way too old.

Instead, get an updated binary as explained here (I have Ubuntu focal, check with lsb_release --all).

wget -O - https://www.kismetwireless.net/repos/kismet-release.gpg.key | sudo apt-key add -
echo 'deb https://www.kismetwireless.net/repos/apt/release/focal focal main' | sudo tee /etc/apt/sources.list.d/kismet.list
sudo apt update
sudo apt install kismet

Do not run as root, use a SUID binary and a unix group access

Kismet needs elevated privileges to run. And deals with possibly hostile data. So running with minimized permissions is the safest approach.

The right way to set it up is by using a Unix group and set user id (SUID) binary. My user is 'josevnz' so I did this:

sudo apt-get install kismet
sudo usermod --append --groups kismet josevnz

Encrypt your access to Kismet with a self-signed certificate

I will enable SSL for my Kismet installation by using a self-signed certificate. I will use for that the Cloudflare CFSSL tools:

sudo apt-get update -y
sudo apt-get install -y golang-cfssl

Next step is to create the self-signed certificates. There is a lot of boilerplate steps here, so I will show you how you can jump through them (but please read the man pages to see what each command does):

Initial certificate

sudo /bin/mkdir --parents /etc/pki/raspberrypi
sudo /bin/cat<<CA>/etc/pki/raspberrypi/ca.json
{
   "CN": "Nunez Barrios family Root CA",
   "key": {
     "algo": "rsa",
     "size": 2048
   },
   "names": [
   {
     "C": "US",
     "L": "CT",
     "O": "Nunez Barrios",
     "OU": "Nunez Barrios Root CA",
     "ST": "United States"
   }
  ]
}
CA
cfssl gencert -initca ca.json | cfssljson -bare ca

SSL profile config

root@raspberrypi:/etc/pki/raspberrypi# /bin/cat<<PROFILE>/etc/pki/raspberrypi/cfssl.json
{
   "signing": {
     "default": {
       "expiry": "17532h"
     },
     "profiles": {
       "intermediate_ca": {
         "usages": [
             "signing",
             "digital signature",
             "key encipherment",
             "cert sign",
             "crl sign",
             "server auth",
             "client auth"
         ],
         "expiry": "17532h",
         "ca_constraint": {
             "is_ca": true,
             "max_path_len": 0, 
             "max_path_len_zero": true
         }
       },
       "peer": {
         "usages": [
             "signing",
             "digital signature",
             "key encipherment", 
             "client auth",
             "server auth"
         ],
         "expiry": "17532h"
       },
       "server": {
         "usages": [
           "signing",
           "digital signing",
           "key encipherment",
           "server auth"
         ],
         "expiry": "17532h"
       },
       "client": {
         "usages": [
           "signing",
           "digital signature",
           "key encipherment", 
           "client auth"
         ],
         "expiry": "17532h"
       }
     }
   }
}
PROFILE

Intermediate certificate

root@raspberrypi:/etc/pki/raspberrypi# /bin/cat<<INTERMEDIATE>/etc/pki/raspberrypi/intermediate-ca.json
{
  "CN": "Barrios Nunez Intermediate CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C":  "US",
      "L":  "CT",
      "O":  "Barrios Nunez",
      "OU": "Barrios Nunez Intermediate CA",
      "ST": "USA"
    }
  ],
  "ca": {
    "expiry": "43830h"
  }
}
INTERMEDIATE
cfssl gencert -initca intermediate-ca.json | cfssljson -bare intermediate_ca
cfssl sign -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile intermediate_ca intermediate_ca.csr | cfssljson -bare intermediate_ca

Configuration for the SSL certificate on the Raspberry PI 4 machine

Here we put the name and IP address of the machine that will run our Kismet web application:

/bin/cat<<RASPBERRYPI>/etc/pki/raspberrypi/raspberrypi.home.json
{
  "CN": "raspberrypi.home",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "L": "CT",
    "O": "Barrios Nunez",
    "OU": "Barrios Nunez Hosts",
    "ST": "USA"
  }
  ],
  "hosts": [
    "raspberrypi.home",
    "localhost",
    "raspberrypi",
    "192.168.1.11"
  ]               
}
RASPBERRYPI
cd /etc/pki/raspberrypi
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=peer raspberrypi.home.json| cfssljson -bare raspberry-peer
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=server raspberrypi.home.json| cfssljson -bare raspberry-server
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=client raspberrypi.home.json| cfssljson -bare raspberry-client

Adding SSL support is then as easy as adding the following overrides:

/bin/cat<<SSL>>/etc/kismet/kismet_site.conf
httpd_ssl=true
httpd_ssl_cert=/etc/pki/raspberrypi/raspberry-server.csr
httpd_ssl_key=/etc/pki/raspberrypi/raspberry-server-key.pem
SSL

Putting everything together, with a Kismet 'site' overrides file

Kismet has a really nice feature: it can use a file that overrides some defaults, without the need to edit multiple files. In this case my installation will override the SSL settings, Wifi interface, and log location. So time to update our /etc/rc.local file:

#!/bin/bash
# Kismet setup
usb_driver=rt2800usb
wlan=$(ls /sys/bus/usb/drivers/$usb_driver/*/net/)
if [ $? -eq 0 ]; then
    set -ex
    /usr/sbin/ifconfig "$wlan" down
    /usr/sbin/iwconfig "$wlan" mode monitor
    /usr/sbin/ifconfig "$wlan" up
    set +ex
    /bin/cat<<KISMETOVERR>/etc/kismet/kismet_site.conf
server_name=Nunez Barrios Kismet server
logprefix=/data/kismet
source=$wlan
httpd_ssl=true
httpd_ssl_cert=/etc/pki/raspberrypi/raspberry-server.csr
httpd_ssl_key=/etc/pki/raspberrypi/raspberry-server-key.pem
KISMETOVERR
fi

Finally, it is time to start Kismet (in my case as the non-root user josevnz):

# If you know which interface is the one in monitoring mode, then 
josevnz@raspberrypi:~$ kismet

Now let's log on for the first time to the web interface (In my case http://raspberripi.home:2501)

You will get a prompt the first time you try to log in your Kismet installation

In here you set up your admin user and password.

Example of the wireless networks detected

After a little time, Kismet will populate the main Dashboard with the list of wireless networks and devices it can detect. You will be surprised not just how many neighboring devices are out there but how many you have in your own house.

In my example, the wireless devices around me look pretty normal, except one that doesn't have a name:

A device with suspicious characteristics

The web interface provides all sorts of useful information, but is there an easy way to filter all the mac addresses on my networks?

Kismet has a REST API, so it is time to see what we can automate from there.

REST-API in Python

The developer documentation contains examples of how to extend Kismet, specifically the one related to the official Kismet REST-API in Python.

But it seems to be missing a feature to use API keys, instead of user/password. And the interaction with the end points doesn't seem to be complicated, so I will write my (less rich feature) wrapper.

You can download and install the code for a small application I wrote (kismet_home to illustrate how to work with Kismet (also has a copy of this tutorial) like this:

python3 -m venv ~/virtualenv/kismet_home
. ~/virtualenv/kismet_home/bin/activate
python -m pip install --upgrade pip
git clone git@github.com:josevnz/kismet_home.git
python setup.py bdist_wheel
pip install kismet_home-0.0.1-py3-none-any.whl

And then run the unit tests/ integration tests and even the third party vulnerability scanner:

. ~/virtualenv/kismet_home/bin/activate
# Unit/ integration tests
python -m unittest test/unit_test_config.py
python -m unittest /home/josevnz/kismet_home/test/test_integration_kismet.py
# Third party vulnerability scanner
pip-audit  --requirement requirements.txt

You will find more details on the README.md and DEVELOPER.md files.

Let's move on with the code.

How to Interact with Kismet using Python

First I'll write a generic HTTP client I can use to query or send commands to Kismet, that is the KismetWorker class:

import json
from datetime import datetime
from typing import Any, Dict, Set, List, Union
import requests


class KismetBase:

    def __init__(self, *, api_key: str, url: str):
        """
        Parametric constructor
        :param api_key: The Kismet generated API key
        :param url: URL where the Kismet server is running
        """
        self.api_key = api_key
        if url[-1] != '/':
            self.url = f"{url}/"
        else:
            self.url = url
        self.cookies = {'KISMET': self.api_key}

    def __str__(self):
        return f"url={self.url}, api_key=XXX"

class KismetWorker(KismetBase):

    def check_session(self) -> None:
        """
        Confirm if the session is valid for a given API key
        :return: None, throws an exception if the session is invalid
        """
        endpoint = f"{self.url}session/check_session"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()

    def check_system_status(self) -> Dict[str, Any]:
        """
        Overall status of the Kismet server
        :return: Nested dictionary describing different aspect of the Kismet system
        """
        endpoint = f"{self.url}system/status.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

    def get_all_alerts(self) -> Any:
        """
        You can get a description how the alert system is set up as shown here: /alerts/definitions.prettyjson
        This method returns the last N alerts registered by the system. Severity and meaning of the alert is explained
        here: https://www.kismetwireless.net/docs/devel/webui_rest/alerts/
        :return:
        """
        endpoint = f"{self.url}alerts/all_alerts.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

    def get_alert_by_hash(self, identifier: str) -> Dict[str, Any]:
        """
        Get details of a single alert by its identifier (hash)
        :return:
        """
        parsed = int(identifier)
        if parsed < 0:
            raise ValueError(f"Invalid ID provided: {identifier}")
        endpoint = f"{self.url}alerts/by-id/{identifier}/alert.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

    def get_alert_definitions(self) -> Dict[Union[str, int], Any]:
        """
        Get the defined alert types
        :return:
        """
        endpoint = f"{self.url}alerts/definitions.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

The way Kismet API works is that you make the API KEY part of the query, or you define it in the KISMET cookie. I choose to populate the cookie.

KismetWorker implements the following methods:

• check_session: It checks if your API KEY is valid. If not it will throw an exception.

• check_system_status: Validates if the administrator (you most likely) defined an administrator for the Kismet server. If not, then all the API queries will fail.

• get_all_alerts: Gets all the available alerts (if any) from your Kismet server.

• get_alert_by_hash: If you know the identifier (hash) of an alert, you can retrieve the details of that event only.

• get_alert_definitions: Get all the alert definitions. Kismet supports a wide range of alerts and a user will definitely be interested to find out what type of alerts they are.

You can see all the integration code here to see how the methods work in action.

I also wrote a class that requires admin privileges. I use it to define a custom alert type and to send alerts using that type to Kismet, as part of the integration tests. Right now I don't have much use of sending custom alerts to Kismet in real life, but that may change in the future, so here is the code:

class KismetAdmin(KismetBase):

    def define_alert(
            self,
            *,
            name: str,
            description: str,
            throttle: str = '10/min',
            burst: str = "1/sec",
            severity: int = 5,
            aclass: str = 'SYSTEM'

    ):
        """
        Define a new type of alert for Kismet
        :param aclass: Alert class
        :param severity: Alert severity
        :param throttle: Optional throttle
        :param name: Name of the new alert
        :param description: What does this mean
        :param burst: Optional burst
        :return:
        """
        endpoint = f"{self.url}alerts/definitions/define_alert.cmd"
        command = {
            'name': name,
            'description': description,
            'throttle': throttle,
            'burst': burst,
            'severity': severity,
            'class': aclass
        }
        r = requests.post(endpoint, json=command, cookies=self.cookies)
        r.raise_for_status()

    def raise_alert(
            self,
            *,
            name: str,
            message: str
    ) -> None:
        """
        Send an alert to Kismet
        :param name: A well-defined name or id for the alert. MUST exist
        :param message: Message to send
        :return: None. Will raise an error if the alert could not be sent
        """
        endpoint = f"{self.url}alerts/raise_alerts.cmd"
        command = {
            'name': name,
            'text': message
        }
        r = requests.post(endpoint, json=command, cookies=self.cookies)
        r.raise_for_status()

Getting the data is just part of the story. We need to normalize it, so it can be used by the final scripts.

How to Normalize the Kismet raw data

Kismet contains a lot of details about the alerts, but we do not require to show the user those details (think about the nice view you get with the web application). Instead we do a few transformations using the following class with static methods:

• parse_alert_definitions: Returns a simplified report of all the alert definitions

• process_alerts: Changes numeric alerts for more descriptive types and also returns dictionaries for the types and severity meaning of those alerts.

• pretty_timestamp: Converts the numeric timestamp into something we can use for comparisons and display

The code for the KismetResultsParser helper class:

class KismetResultsParser:
    SEVERITY = {
        0: {
            'name': 'INFO',
            'description': 'Informational alerts, such as datasource  errors, Kismet state changes, etc'
        },
        5: {
            'name': 'LOW',
            'description': 'Low - risk events such as probe fingerprints'
        },
        10: {
            'name': 'MEDIUM',
            'description': 'Medium - risk events such as denial of service attempts'
        },
        15: {
            'name': 'HIGH',
            'description': 'High - risk events such as fingerprinted watched devices, denial of service attacks, '
                           'and similar '
        },
        20: {
            'name': 'CRITICAL',
            'description': 'Critical errors such as fingerprinted known exploits'
        }
    }

    TYPES = {
        'DENIAL': 'Possible denial of service attack',
        'EXPLOIT': 'Known fingerprinted exploit attempt against a vulnerability',
        'OTHER': 'General category for alerts which don’t fit in any existing bucket',
        'PROBE': 'Probe by known tools',
        'SPOOF': 'Attempt to spoof an existing device',
        'SYSTEM': 'System events, such as log changes, datasource errors, etc.'
    }

    @staticmethod
    def parse_alert_definitions(
            *,
            alert_definitions: List[Dict[str, str]],
            keys_of_interest: Set[str] = None
    ) -> List[Dict[str, str]]:
        """
        Remove unwanted keys from full alert definition dump, to make it easier to read onscreen
        :param alert_definitions: Original Kismet alert definitions
        :param keys_of_interest: Kismet keys of interest
        :return: List of dictionaries with trimmed keys, description, severity and header for easy reading
        """
        if keys_of_interest is None:
            keys_of_interest = {
                'kismet.alert.definition.class',
                'kismet.alert.definition.description',
                'kismet.alert.definition.severity',
                'kismet.alert.definition.header'
            }
        parsed_alerts: List[Dict[str, str]] = []
        for definition in alert_definitions:
            new_definition = {}
            for def_key in definition:
                if def_key in keys_of_interest:
                    new_key = def_key.split('.')[-1]
                    new_definition[new_key] = definition[def_key]
            parsed_alerts.append(new_definition)
        return parsed_alerts

    @staticmethod
    def process_alerts(
            *,
            alerts: List[Dict[str, Union[str, int]]],

    ) -> Any:
        """
        Removed unwanted fields from alert details, also return extra data for severity and types of alerts
        :param alerts:
        :return:
        """
        processed_alerts = []
        found_types = {}
        found_severities = {}
        for alert in alerts:
            severity = alert['kismet.alert.severity']
            severity_name = KismetResultsParser.SEVERITY[severity]['name']
            severity_desc = KismetResultsParser.SEVERITY[severity]['description']
            found_severities[severity_name] = severity_desc
            text = alert['kismet.alert.text']
            aclass = alert['kismet.alert.class']
            found_types[aclass] = KismetResultsParser.TYPES[aclass]
            processed_alert = {
                'text': text,
                'class': aclass,
                'severity': severity_name,
                'hash': alert['kismet.alert.hash'],
                'dest_mac': alert['kismet.alert.dest_mac'],
                'source_mac': alert['kismet.alert.source_mac'],
                'timestamp': alert['kismet.alert.timestamp']
            }
            processed_alerts.append(processed_alert)
        return processed_alerts, found_severities, found_types

    @staticmethod
    def pretty_timestamp(timestamp: float) -> datetime:
        """
        Convert a Kismet timestamp (TIMESTAMP.UTIMESTAMP) into a pretty timestamp string
        :param timestamp:
        :return:
        """
        return datetime.fromtimestamp(timestamp)

If you run the integration tests with the admin role enabled, you will see than one or more (depending how many times you ran the test) alerts were added to the Web UI:

These alerts where generated using the Python client and the REST API

As a reminder, you can see how this is used by looking at the code here. Showing a sample run of all the integration tests against my installation (this one without publishing alerts, so some tests are skipped):

(kismet_home) [josevnz@dmaf5 kismet_home]$ python -m unittest /home/josevnz/kismet_home/test/test_integration_kismet.py 
[09:13:05] DEBUG    Starting new HTTP connection (1): raspberrypi.home:2501                                                                                                                                                        connectionpool.py:228
           DEBUG    http://raspberrypi.home:2501 "GET /session/check_session HTTP/1.1" 200 None                                                                                                                                    connectionpool.py:456
.           DEBUG    Starting new HTTP connection (1): raspberrypi.home:2501                                                                                                                                                        connectionpool.py:228
           DEBUG    http://raspberrypi.home:2501 "GET /system/status.json HTTP/1.1" 200 None                                                                                                                                       connectionpool.py:456
.           DEBUG    Starting new HTTP connection (1): raspberrypi.home:2501                                                                                                                                                        connectionpool.py:228
           DEBUG    http://raspberrypi.home:2501 "GET /alerts/definitions.json HTTP/1.1" 200 None                                                                                                                                  connectionpool.py:456
.[09:13:05] 'ADMIN_SESSION_API' environment variable not defined. Skipping this test                                                                                                                                       test_integration_kismet.py:105
....
----------------------------------------------------------------------
Ran 7 tests in 0.053s

OK

Where do we store our API key and other configuration details?

Details like this won't be hardcoded inside the scripts, but instead they will reside on an external configuration file:

(kismet_home) [josevnz@dmaf5 kismet_home]$ cat ~/.config/kodegeek/kismet_home/config.ini 
[server]
url = http://raspberrypi.home:2501
api_key = E41CAD466552810392D538FF8D43E2C5

The following classes handle all the access details (using a Reader and a Writer class for each type of operation):

"""
Simple configuration management for kismet_home settings
"""
import os.path
from configparser import ConfigParser
from pathlib import Path
from typing import Dict

from kismet_home import CONSOLE

DEFAULT_INI = os.path.expanduser('~/.config/kodegeek/kismet_home/config.ini')
VALID_KEYS = {'api_key', 'url'}


class Reader:

    def __init__(self, config_file: str = DEFAULT_INI):
        """
        Constructor
        :param config_file: Optional override of the ini configuration file
        """
        self.config = ConfigParser()
        if not self.config.read(config_file):
            raise ValueError(f"Could not read {config_file}")

    def get_api_key(self):
        """
        Get back the API key used to connect to Kismet
        :return:
        """
        return self.config.get('server', 'api_key')

    def get_url(self):
        """
        Get back URL of Kismet server
        :return:
        """
        return self.config.get('server', 'url')


class Writer:

    def __init__(
            self,
            *,
            server_keys: Dict[str, str]
    ):
        if not server_keys:
            raise ValueError("Configuration is incomplete!, aborting!")
        self.config = ConfigParser()
        self.config.add_section('server')
        valid_keys_cnt = 0
        for key in server_keys:
            value = server_keys[key]
            if key not in VALID_KEYS:
                CONSOLE.log(f"Ignoring invalid key: {key} = {value}")
                continue
            self.config.set('server', key, value)
            CONSOLE.log(f"Added: server: {key} = {value}")
        for valid_key in VALID_KEYS:
            if not self.config.get('server', valid_key):
                raise ValueError(f"Missing required key: {valid_key}")

    def save(
            self,
            *,
            config_file: str = DEFAULT_INI
    ):
        basedir = Path(config_file).parent
        basedir.mkdir(exist_ok=True, parents=True)
        with open(config_file, 'w') as config:
            self.config.write(config, space_around_delimiters=True)
        CONSOLE.log(f"Configuration file {config_file} written")

The first time you set up your kismet_home installation, you can create the configuration files like this:

[josevnz@dmaf5 kismet_home]$ python3 -m venv ~/virtualenv/kismet_home
[josevnz@dmaf5 kismet_home]$ . ~/virtualenv/kismet_home/bin/activate
(kismet_home) [josevnz@dmaf5 kismet_home]$ python -m pip install --upgrade pip
(kismet_home) [josevnz@dmaf5 kismet_home]$ git clone git@github.com:josevnz/kismet_home.git
(kismet_home) [josevnz@dmaf5 kismet_home]$ python setup.py bdist_wheel
(kismet_home) [josevnz@dmaf5 kismet_home]$ pip install kismet_home-0.0.1-py3-none-any.whl

(kismet_home) [josevnz@dmaf5 kismet_home]$ kismet_home_config.py 
Please enter the URL of your Kismet server: http://raspberrypi.home:2501/
Please enter your API key: E41CAD466552810392D538FF8D43E2C5
[13:02:35] Added: server: url = http://raspberrypi.home:2501/                                                                                 config.py:44
           Added: server: api_key = E41CAD466552810392D538FF8D43E2C5                                                                          config.py:44
           Configuration file /home/josevnz/.config/kodegeek/kismet_home/config.ini written

Please note the use of the virtual environment here. This will allow us to keep the application's libraries self-contained.

Putting everything together: How to Write our CLI for kismet_home

The kismet_home_alerts.py script will support two modes:

• Show the alert definitions

• Show all the alerts

Also, it will allow filtering alerts based on the level (INFO, MEDIUM, HIGH, ...).

Showing all the definitions, filtered by CRITICAL:

You can see here the alert definitions filtered by level

Or showing all the alerts received so far, with anonymous MAC address (great for screenshots like this):

Alerts for my local network, with anonymous MAC addresses and filtered

How you can generate these tables with ease? There is a dedicated class for the text user interface (TUI):

from typing import List, Dict, Any

from rich.layout import Layout
from rich.table import Table

from kismet_home.kismet import KismetResultsParser


def create_alert_definition_table(
        *,
        alert_definitions: List[Dict[str, Any]],
        level_filter: str = 0
) -> Table:
    """
    Create a table showing the alert definitions
    :param alert_definitions: Alert definitions from Kismet
    :param level_filter: User can override the level of the alerts shown. But default is 0 (INFO)
    :return: A Table with the alert definitions
    """
    definition_table = Table(title="Alert definitions")
    definition_table.add_column("Severity", justify="right", style="cyan", no_wrap=True)
    definition_table.add_column("Description", style="magenta")
    definition_table.add_column("Header", justify="right", style="yellow")
    definition_table.add_column("Class", justify="right", style="green")
    filter_level = KismetResultsParser.get_level_for_security(level_filter)
    filtered_definitions = 0
    for definition in alert_definitions:
        int_severity: int = definition['severity']
        if int_severity < filter_level:
            continue
        severity = KismetResultsParser.SEVERITY[int_severity]['name']
        if 0 <= int_severity < 5:
            severity = f"[bold blue]{severity}[/ bold blue]"
        if 5 <= int_severity < 10:
            severity = f"[bold yellow]{severity}[/ bold yellow]"
        if 10 <= int_severity < 15:
            severity = f"[bold orange]{severity}[/ bold orange]"
        else:
            severity = f"[bold red]{severity}[/ bold red]"
        filtered_definitions += 1
        definition_table.add_row(
            severity,
            definition['description'],
            definition['header'],
            definition['class']
        )
    definition_table.caption = f"Total definitions: {filtered_definitions}"
    return definition_table


def create_alert_layout(
        *,
        alerts: List[Dict[str, Any]],
        level_filter: str = 0,
        anonymize: bool = False,
        severities: Dict[str, str]
):
    """
    :param severities:
    :param alerts:
    :param level_filter:
    :param anonymize:
    :return:
    """
    alerts_table = Table(title="Alert definitions")
    alerts_table.add_column("Timestamp", no_wrap=True)
    alerts_table.add_column("Severity", justify="right", style="cyan", no_wrap=True)
    alerts_table.add_column("Text", style="magenta")
    alerts_table.add_column("Source MAC", justify="right", style="yellow", no_wrap=True)
    alerts_table.add_column("Destination MAC", justify="right", style="yellow", no_wrap=True)
    alerts_table.add_column("Class", justify="right", style="green", no_wrap=True)
    filter_level = KismetResultsParser.get_level_for_security(level_filter)

    filtered_definitions = 0
    for alert in alerts:
        int_severity: int = KismetResultsParser.get_level_for_security(alert['severity'])
        if int_severity < filter_level:
            continue
        severity = KismetResultsParser.SEVERITY[int_severity]['name']
        if 0 <= int_severity < 5:
            severity = f"[bold blue]{severity}[/ bold blue]"
        if 5 <= int_severity < 10:
            severity = f"[bold yellow]{severity}[/ bold yellow]"
        if 10 <= int_severity < 15:
            severity = f"[bold orange]{severity}[/ bold orange]"
        else:
            severity = f"[bold red]{severity}[/ bold red]"
        filtered_definitions += 1
        if anonymize:
            s_mac = KismetResultsParser.anonymize_mac(alert['source_mac'])
            d_mac = KismetResultsParser.anonymize_mac(alert['dest_mac'])
        else:
            s_mac = alert['source_mac']
            d_mac = alert['dest_mac']
        alerts_table.add_row(
            str(KismetResultsParser.pretty_timestamp(alert['timestamp'])),
            severity,
            alert['text'],
            s_mac,
            d_mac,
            alert['class']
        )
    alerts_table.caption = f"Total alerts: {filtered_definitions}"

    severities_table = Table(title="Severity legend")
    severities_table.add_column("Severity")
    severities_table.add_column("Explanation")
    for severity in severities:
        explanation = f"[green]{severities[severity]}[/green]"
        severities_table.add_row(f"[yellow]{severity}[/yellow]", explanation)

    layout = Layout()
    layout.split(
        Layout(ratio=2, name="alerts"),
        Layout(name="severities"),
    )
    layout["alerts"].update(alerts_table)
    layout["severities"].update(severities_table)
    return layout, filtered_definitions

And now with all the ingredients ready, we can see how the final script looks:

#!/usr/bin/env python
"""
# kismet_home_alerts.py
# Author
Jose Vicente Nunez Zuleta (kodegeek.com@protonmail.com)
"""
import logging
import sys

from requests import HTTPError
import argparse

from kismet_home import CONSOLE
from kismet_home.config import Reader
from kismet_home.kismet import KismetWorker, KismetResultsParser
from kismet_home.tui import create_alert_definition_table, create_alert_layout

if __name__ == '__main__':

    arg_parser = argparse.ArgumentParser(
        description="Display alerts generated by your local Kismet installation",
        prog=__file__
    )
    arg_parser.add_argument(
        '--debug',
        action='store_true',
        default=False,
        help="Enable debug mode"
    )
    arg_parser.add_argument(
        '--anonymize',
        action='store_true',
        default=False,
        help="Anonymize MAC addresses"
    )
    arg_parser.add_argument(
        '--level',
        action='store',
        default='INFO',
        help="Enable debug mode"
    )
    arg_parser.add_argument(
        'mode',
        action='store',
        choices=['alert_type', 'alerts'],
        help="Operation mode"
    )

    try:
        args = arg_parser.parse_args()
        conf_reader = Reader()
        kw = KismetWorker(
            api_key=conf_reader.get_api_key(),
            url=conf_reader.get_url()
        )
        if args.mode == 'alert_type':
            alert_definitions = KismetResultsParser.parse_alert_definitions(
                alert_definitions=kw.get_alert_definitions()
            )
            table = create_alert_definition_table(alert_definitions=alert_definitions, level_filter=args.level)
            if table.columns:
                CONSOLE.print(table)
            else:
                CONSOLE.print(f"[b]Could not get alert definitions![/b]")
        elif args.mode == 'alerts':
            alerts, severities, types = KismetResultsParser.process_alerts(
                alerts=kw.get_all_alerts()
            )
            layout, found = create_alert_layout(
                alerts=alerts,
                level_filter=args.level,
                anonymize=args.anonymize,
                severities=severities
            )
            if found:
                CONSOLE.print(layout)
            else:
                CONSOLE.print(f"[b]No alerts to show for level={args.level}[/b]")
    except (ValueError, HTTPError):
        logging.exception("There was an error")
        sys.exit(100)
    except KeyboardInterrupt:
        CONSOLE.log("Scan interrupted, exiting...")
    sys.exit(0)

A few things to note:

• This is not a long-running application. Instead, is a snapshot of all the alerts. If you wanted, for example, to forward these alerts by email or to a framework like grafana, you are better off using Websockets and one of the methods that retrieves only the last changes.

• The layout is crude, and there is plenty of room for improvement. But our little tui is displaying relevant information without too many distractions

• And if was fun to code!

What did we learn?

• How to install Kismet and secure it with a self-signed SSL certificate

• How to write a simple Bash script to set up the correct Wireless interface in monitor mode, after the RaspBerryPI reboots

• How to add an API KEY with read-only access to use it instead of the legacy user/ password schema for authentication and authorization

• How to write classes in Python that can communicate with Kismet using its REST-API

• How to add unit and integration tests to the code to make sure everything works and new code changes do not break existing functionality

Please leave your comments on the git repository and report any bugs. But more important get Kismet, get the code of this tutorial, and start securing your home wireless infrastructure in no time.

Introduction

Are you curious about the IoT (Internet of Things?) Have you always wanted to try to make your own robot, smart mirror, or bird feeder camera? What about building a computer for a fraction of the cost of a commercially available machine?

If you said yes to any of these questions, you might enjoy playing around with a Raspberry Pi.

In this article, I'll explain what a Raspberry Pi is (and what it’s not.) Then I'll show you some things you can use it for, and finally, I'll list all of the current models along with their specs.

What is Raspberry Pi?

A Raspberry Pi is a single board computer (SBC) created in the United Kingdom by the Raspberry Pi Foundation. It's a charity that "works to put the power of computing and digital making into the hands of people all over the world."

The first model of the Raspberry Pi was released in 2012, and as of 2021 there have been five generations of the boards. A microcontroller (more about that later), called the Pico was released in early 2021.

GPIO Pins

Something that sets the Pi apart from your average computer is the set of 40 GPIO (General Purpose Input Output) pins.

GPIO pins are pretty much exactly what they sound like. They are designed to input and output single bits. This means that you can use them to add all sorts of functionality to your Raspberry Pi using switches, buzzers, lights, sensors, and so on.

Raspberry Pi HATs

There are a number of add-on boards that you can attach to the Raspberry Pi using the GPIO pins.

Raspberry Pi HATs (Hardware Attached on Top) are add-on boards designed according to certain specifications. The Raspberry Pi can automatically detect and configure the HATs, making set-up easy.

There is a huge variety of HATs and other add-on boards that you can buy, but here are some notable ones:

• PoE+ HAT – Power over Ethernet HAT
• Sense HAT – Designed for the Astro Pi mission, the Sense HAT includes a gyroscope, an accelerometer, a magnetometer, and sensors for temperature, humidity, and Barometric pressure.
• Pimoroni Explorer HAT Pro – All-purpose board featuring touch pads, crocodile clip pads, and analog inputs
• Adafruit Capacitive Touch HAT – Similar to a Makey Makey, this HAT allows you to use any conductive object to trigger events using Python.

Raspberry Pi Operating Systems

The Raspberry Pi often runs some form of Linux, but there are a ton of operating systems that you can use.

On the official website, you will find a list of operating system images available for download. These include the official Raspberry Pi OS, Debian Buster, and Ubuntu (desktop, core, and server.)

You will also find RetroPie, a specialized gaming platform operating system, and LibreELEC, a lightweight Linux distribution specifically created for use with the open source media player Kodi.

Raspberry Pi VS Arduino

You may have heard of Arduino boards and wondered what the difference is between them and the Raspberry Pi.

The main difference is that Pis (with the exception of the Pico and the RP2040) are full computers with operating systems. You can connect your Pi to a keyboard, mouse, and monitor and use it like you would use a Mac or PC.

The Arduino, on the other hand, is a microcontroller. It cannot function independently as a computer, but is programmed using a computer and then used to control things like cameras, lights, robots, and so on.

As the official Arduino website puts it: “Arduino boards are able to read inputs… and turn it into an output.”

What is the Raspberry Pi Used For?

Search the internet and you will find a huge array of projects created using Raspberry Pis.

Common use cases include home automation, gaming consoles, servers, WiFi extenders, streaming devices, weather stations, and home computers. (Fun fact: much of this article was written on a Raspberry Pi.)

During the Covid-19 pandemic, Raspberry Pi's were even used to control ventilators in some hard-hit areas.

Raspberry Pi Models

All models of Raspberry Pi that are currently in production have 40 GPIO pins.

This list does not include the Raspberry Pi microcontrollers, the Pico and the RP2040.

You can find out where to purchase any of these boards on the Raspberry Pi website.

Conclusion

The Raspberry Pi is an affordable way to explore electronics, hardware, and computer programming. It can be used for a huge variety of projects, from the very silly (like a hamster powered hamster drawing machine) to the very important (like computer labs in developing nations.)

Now that you know the basics, why don't you go out and make something? Whether you hook a Capacitive Touch HAT up to your Raspberry Pi and turn it into a banana piano or install Linux on it and use it to do your homework, I hope you have a great time creating something cool and useful.

The Raspberry Pi is a very powerful computer in a tiny package. The cheapest option, the Raspberry Pi Zero, is capable of running a fully featured Linux distribution and driving a high definition display. It is the size of two coins (US Quarters) and costs $5.

At $10, the Raspberry Pi Zero W comes with integrated WiFi and Bluetooth.

The $10 Raspberry Pi Zero W has a powerful CPU, WiFi, Bluetooth, and all kinds of connectors

At the “high end”, you can purchase a Raspberry Pi 4 desktop kit for less than $100. It has a 4-core ARM CPU running at 1.5GHz, GPU, 2GB (up to 8 GB) of RAM, 16GB (up to 2TB) of storage on MicroSD cards, wifi and Ethernet connectors, USB ports, HDMI ports that can drive 4K displays, as well as a keyboard and mouse.

The Raspberry Pi is also more than a standard computer. It is fun and hackable. The Raspberry Pi exposes a row of GPIO (General Purpose Input Output) pins. You can attach simple sensors (eg. temperature, humidity, light) to those pins, and capture their data from your applications running on the Pi.

You could also attach LED lights and motors to those pins, and use your Pi application to drive those peripheral devices.

For more complex sensors or devices, such as camera modules, you can also connect to the Pi via USB or Wifi, and access them in software. The Pi is a great device for learning and hardware hacking. Because of this, it is widely used in educational settings.

However, fun and learning are not just for kids. With so much computing power and easy networking, the Raspberry Pi can easily become a personal application server for you.

For example, you can put a web application (for example, a collaborative note taking app, or just some documents / videos to share) on a Pi, bring it to a meeting, and make it accessible to everyone in the room. You do not even need the Internet. It is completely decentralized and censorship-resistant.

The personal server is especially useful for developers. You can have a separate environment to deploy and test your server-side applications without having to mess with your laptop. A personal dev server is like Docker on steroids. In this article, I will teach you how to set one up.

First, get a Raspberry Pi

If this is your first Raspberry Pi, the easiest (and most expensive) way to set up is just to buy a desktop kit for around $100. It comes with everything you need for a computer except for the display.

If you are using the Pi as a personal dev server, you would NOT need a display after the initial setup. You can just SSH into it from your laptop once it is turned on!

Learn how to get your Raspberry Pi starter kit for free when you participate in this high performance web application learning exercise.

Of course, if you have spare computer parts, such as MicroSD cards, USB power supply, a keyboard, and a mouse laying around, you could save money by purchasing only the boards. You could get a Raspberry Pi Zero board for $5 and a Raspberry Pi 4 board for $35.

But the thing missing from the board is a MicroSD card that acts as the “hard drive” for storing the operating system and data. You can purchase a 16GB MicroSD card for $10 online, a MicroSD card reader, and use the Raspberry Pi Imager to load an operating system onto the MicroSD card from your laptop.

The two popular choices are Raspberry Pi OS and Ubuntu Linux. Both are Debian-based Linux distributions. Most starter kits pre-install the Raspberry Pi OS on their MicroSD cards (it is called NOOBS).

In the next two sections, I will talk you through both operating systems.

How to set up Raspberry Pi OS

Once you put in the MicroSD card with NOOBS, and connect a display, a keyboard, and a mouse, you can turn on the power!

From there, just follow on screen instructions to install Raspberry Pi OS (previously known as the Raspbian OS). Then setup a password for the user pi, and setup the wifi connection.

After you are logged in, go to the Preferences → Raspberry Pi Configuration menu and enable SSH. That will allow you to log into the Pi from another computer.

Note: in order to use the Pi as a “headless” server, you could request a static IP address from your router. In the future, you can just power on the Pi, and connect to it via SSH from your other computers or phones.

The Raspberry Pi OS is derived from the Debian Linux distribution. It comes with a full desktop UI environment with a modern web browser, a command line terminal, and learning programs such as IDEs for Python, Java, and Scratch.

My Raspberry Pi 4 with Raspberry Pi OS setup. Notice how small the actual computer is.

For our purposes, we are mostly interested in installing dev and server software through the command line terminal.

At this point, you could also find out the IP address of the Pi on your local network by running the following command. Then you can SSH into the Pi using that local IP address, username pi, and the password you gave pi during setup.

$ hostname -I
192.168.2.108 172.17.0.1

You can find a complete list of software packages installed on the Raspberry Pi OS here. It is always a good idea to update and upgrade to the latest packages. Run the command below and be patient. It could take an hour.

$ sudo apt update && sudo apt upgrade

How to set up Ubuntu Server 20.04

The Raspberry Pi OS is primarily geared toward a desktop experience. For developers who just want to use the device as a server or IoT device, the Ubuntu Linux is a much better choice. It has the latest software packages and libraries, and could be far more efficient without the desktop windows, web browser, Java, games, and learning tools.

You can download Ubuntu Server images for Raspberry Pi from the web, and load it on a MicroSD card. But perhaps a much easier way is just to use the Raspberry Pi Imager, select Ubuntu Server 20.04 TLS from the menu, and write into an empty MicroSD card.

Once the MicroSD card is prepared, you should follow these instructions to put in your WiFi network name and password. This allows the Raspberry Pi device to connect to the network as soon as it boots.

Basically, you can just put the MicroSD card into the Raspberry Pi, connect USB power, then wait for it to come online. You can find the raspberrypi device IP from your WiFi router, and then SSH into from any computer on your network.

The initial username and password are ubuntu / ubuntu. There is no need to even connect a monitor or keyboard. That’s it for a completely headless setup!

Note: if, for some reason, your Raspberry Pi cannot connect to WiFi at startup, you can hook up an HDMI display and a USB keyboard to it. Then follow these instructions to debug and set up WiFi on the running system.

Next, let's install the developer tool stack on the Pi.

Install Git

I always install Git on all my development environments because a lot of software can be directly retrieved from Git repositories. It saves me the trouble of downloading and copying.

Git also allows me to save and backup my own work in private repositories. For a small computer like Raspberry Pi, I would recommend that you save work in Git in case you lose the device or MicroSD card.

The following command installs Git:

$ sudo apt install git

Install Node.js

To turn the Raspberry Pi into a personal dev server for web applications, you need to install a modern web application runtime.

For most developers today, the best starting point is Node.js, which allows you to write server-side applications in JavaScript. The following two commands install Node.js on your Pi.

$ curl -sL https://deb.nodesource.com/setup_10.x | sudo bash -
$ sudo apt install nodejs

You can verify the installation is done correctly by running the following two commands. Both node and npm are now available.

$ node -v
v10.19.0
$ npm -v
5.8.0

From here, you can use npm to install modules. For example, a commonly used npm module is the express framework for web applications.

$ npm install express

Now, you can go through the ExpressJS hello world example to create a web server on your Pi, and use web browsers from any computer on your network to access the application!

Install Rust

Rust is a fast growing programming language for writing both systems and web applications. It is close to the hardware, high performance, and memory safe. That makes Rust a great language for writing applications on resource constrained devices like the Raspberry Pi.

Also, Rust is the most beloved programming language by StackOverflow users for the past 5 years in a row. It is well worth your time to learn it!

An important use case of Rust is to compile Rust functions into WebAssembly and run them inside Node.js applications to achieve performance, safety, and code portability. It is a great choice for running computationally intensive web applications on a small Raspberry Pi device. In fact, you could get a free Raspberry Pi starter kit if you learn how to do that.

Note: strictly speaking, you do not need to install Rust tools on the Pi. You typically only need to run Rust programs in the Pi. You can compile your Rust program on any computer and then copy the compiled binaries to the Pi.

But still, with the powerful CPU, you can compile Rust programs on the Raspberry Pi. So why not?

The following command installs the Rust compiler toolchain on the Pi.

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Run the following command to set up the correct path without logging out and back in again.

$ source $HOME/.cargo/env

The above command also installs the Rust package manager called cargo. Most Rust developers use cargo to build and share their work.

$ cargo -V
cargo 1.44.1 (88ba85757 2020-06-11)

Next, you can clone our Rust learning repository, and learn from examples.

$ git clone https://github.com/second-state/wasm-learning.git

Here is the hello world example. Have fun!

$ cd wasm-learning/rust/hello
$ cargo build
   Compiling hello v0.1.0 (/home/pi/Dev/wasm-learning/rust/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 4.35s
$ target/debug/hello
Hello, world!

Check out the official Rust web site and the Rust by Example books for more learning resources.

Learn Docker

We have seen that the Raspberry Pi OS and Ubuntu Server are both very capable Linux distributions with lots of software packages.

But what if I want to test applications on other OSes? Do I need to wipe clean and reinstall a different OS on the MicroSD card? The answer is no. You can just use Docker! The following two commands install docker on the Raspberry Pi:

$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh

Run the following command so that you can use Docker as the pi user:

$ sudo usermod -aG docker pi

The Docker info command shows that Docker is now installed on an ARM system with Raspberry Pi OS.

$ docker info
... ...
 Kernel Version: 4.19.118-v7l+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 3.814GiB
 Name: raspberrypi
 ID: XERI:ZVVZ:XQVA:HXSH:KRPI:6GL2:5QRE:E7GZ:Z72Q:6SGF:CEI6:GKTC
 Docker Root Dir: /var/lib/docker
... ...

Next, you can pull a Docker image for the latest Ubuntu distribution, run it, and log into Ubuntu as a command line user.

$ docker pull ubuntu
... ...
$ docker run -it ubuntu bash
root# ... enter commands ...

What’s next?

In this article, we have touched on the basics and learned how to turn your Raspberry Pi 4 device into a personal dev server for software developers.

There is much to learn about Git, Node.js, Rust, WebAssembly and Docker. There are also many other developer stacks you can install on the Raspberry Pi.

Grab your free Raspberry Pi kit and let us know what you did with it!

Subscribe to our newsletter and stay in touch.

Over the last few years, smart home devices have gone from less than 300,000 back in 2015 up to almost 1.2 billion in 2020. And they’re expected to grow to 1.5 billion by 2021.

So it's likely you have at least some smart devices in your home, given that the average will reach 8.7 smart devices per home by 2021.

As developers, we have some advantage in this domain, since we can build our own smart home devices.

It’s not only the devices that have experienced rapid development. The development boards used for them have started to become more and more commercial and accessible.

In this article, we will see how we can use a Raspberry Pi, an LCD screen, and a few lines of code to monitor the weather outside or for a specific location.

Because this is a Do It Yourself (DIY) project, there are some prerequisites that we need for this device.

Prerequisites

• Raspberry Pi 3 (or higher)
• LCD Screen
• Connection Wires
• Potentiometer (Optional)
• Breadboard (Optional)

How to Build It

As soon as we have everything we need we can start. Let's take it step by step.

Step I - Base Configuration

The first step consists of the basic setup and a verification of all the components.

For this demo, we will use the ClimaCell Weather API as a weather data provider, as they have a large number of indicators, including air quality indicators, for us to use.

To use their API we must open an account on their platform and get an API key, which we’ll use to sign our requests.

ClimaCell API Limit

The account is free to open and it comes with a 100-hour limit of API calls, which is more than enough for our project.

As soon as we have this API key, we can move to the hardware configuration and connect the LCD screen to our Raspberry Pi. You should turn the Raspberry Pi off while you make the wire connection.

The pin layout for Raspberry Pi 3 can be seen in the next image.

Raspberry Pi 3 Pins

The wire connection between the LCD and the development board is the following:

Connection between Raspberry PI and LCD

This hardware connection will make the LCD screen be on full brightness and full contrast. The brightness level is not a problem, but contrast is because we won’t be able to see the characters on the screen.

That’s why we need to introduce at least one potentiometer with which to set the contrast level.

Schematic

At this point, we can turn on our Raspberry Pi and we should see the LCD screen alive. With the help of variable resistance we should be able to control the contrast.

Step II - Project Configuration

As a programming language, we’ll use NodeJS to write the code. If you don’t already have NodeJS installed on your Raspberry then you can follow these simple instructions.

In a new folder, run the command npm init -y to set up a new npm package, followed by the command npm install lcd node-fetch to install these 2 necessary dependencies.

• lcd will be used to communicate with the LCD Screen
• node-fetch will be used to make HTTP requests to ClimaCell API.

We said that we need an API key to communicate with the weather data provider. You place your secret API key directly in the main code, or you can create a config.json file in which you can place this key and any other code-related configuration you may have.

config.json

{  "cc_key": "<your_ClimaCell_API_key>"}

Lastly, let’s create the main file of our project and include all these things we talked about.

// * Dependencies
const Lcd = require("lcd");
const fs = require("fs");
const fetch = require("node-fetch");

// * Globals
const { cc_key } = JSON.parse(fs.readFileSync("./config.json"));

Step III - The LCD

Writing on the screen is a piece of cake using the lcd module. This library acts as a layer of abstraction over how we communicate with the device. In this way we don’t need to micro-manage each command individually.

The entire code for our LCD screen is the following:

[RAW](https://carbon.now.sh/?bg=rgba(171%2C%20184%2C%20195%2C%201)&t=seti&wt=none&l=javascript&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fl=1&fm=Hack&fs=14px&lh=133%25&si=false&es=2x&wm=false&code=const%2520lcd%2520%253D%2520new%2520Lcd(%257B%2520rs%253A%252026%252C%2520e%253A%252019%252C%2520data%253A%2520%255B13%252C%25206%252C%25205%252C%252011%255D%252C%2520cols%253A%252016%252C%2520rows%253A%25202%2520%257D)%253B%250A%250A%250Afunction%2520writeToLcd(col%252C%2520row%252C%2520data)%2520%257B%250A%2520%2520return%2520new%2520Promise((resolve%252C%2520reject)%2520%253D%253E%2520%257B%250A%2520%2520%2520%2520lcd.setCursor(col%252C%2520row)%253B%250A%2520%2520%2520%2520lcd.print(data%252C%2520(err)%2520%253D%253E%2520%257B%250A%2520%2520%2520%2520%2520%2520if%2520(err)%2520%257B%250A%2520%2520%2520%2520%2520%2520%2520%2520reject()%253B%250A%2520%2520%2520%2520%2520%2520%257D%250A%2520%2520%2520%2520%2520%2520resolve()%253B%250A%2520%2520%2520%2520%257D)%253B%250A%2520%2520%257D)%253B%250A%257D)

The first step was to create a new lcd object and to pass as argument the pins we’ve used.

The keys cols and rows represent the number of columns and rows of our LCD display. 16x2 is the one I used in this example. If your LCD has just 8 columns and 1 row, then replace 16 and 2 with your values.

To write something on the display we need to use these two methods successively:

• lcd.setCursor() - selecting the position from which to start writing
• lcd.print()

At the same time, we wrapped these two functions in a promise to make use of async/away keywords.

At this point, you can use this function and print something on your display. writeToLcd(0,0,'Hello World') should print the message Hello World on the first row starting from the first column.

Step IV - The Weather Data

The next step is to get the weather data and print it on the display.

ClimaCell provides a lot of weather data information, but also air quality and pollen, fire and other information. The data is vast, but keep in mind that your LCD screen only has 16 columns and 2 rows – that’s just 32 characters.

If you want to display more types of data and this limit is too small for you, then you can use a scroll effect.

For this demo we’ll keep it simple and we’ll print on the LCD screen the following data:

• current date (hour, minutes, seconds)
• temperature
• precipitation intensity

_[RAW](https://carbon.now.sh/?bg=rgba(171%2C%20184%2C%20195%2C%201)&t=seti&wt=none&l=javascript&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fl=1&fm=Hack&fs=14px&lh=133%25&si=false&es=2x&wm=false&code=async%2520function%2520getWeatherData(apiKey%252C%2520lat%252C%2520lon)%2520%257B%250A%2520%2520const%2520url%2520%253D%2520%2560https%253A%252F%252Fapi.climacell.co%252Fv3%252Fweather%252Frealtime%253Flat%253D%2524%257Blat%257D%2526lon%253D%2524%257Blon%257D%2526unit_system%253Dsi%2526fields%253Dtemp%2526fields%253Dprecipitation%2526apikey%253D%2524%257BapiKey%257D%2560%253B%250A%250A%2520%2520const%2520res%2520%253D%2520await%2520fetch(url)%253B%250A%2520%2520const%2520data%2520%253D%2520await%2520res.json()%253B%250A%2520%2520return%2520data%253B%250A%257D%250A%250Aasync%2520function%2520printWeatherData()%2520%257B%250A%2520%2520const%2520%257B%2520temp%252C%2520precipitation%2520%257D%2520%253D%2520await%2520getWeatherData(cckey%252C%252045.658%252C%252025.6012)%253B%250A%250A%2520%2520%252F%252F%2520%2520first%2520row%250A%2520%2520await%2520writeToLcd(0%252C%25200%252C%2520Math.round(temp.value)%2520%252B%2520temp.units)%253B%250A%250A%2520%2520%252F%252F%2520%2520second%2520row%250A%2520%2520const%2520precipitationMessage%2520%253D%250A%2520%2520%2520%2520%2522Precip.%253A%2520%2522%2520%252B%2520precipitation.value%2520%252B%2520precipitation.units%253B%250A%2520%2520await%2520writeToLcd(0%252C%25201%252C%2520precipitationMessage)%253B%250A%257D)

To get data from ClimaCell for a specific location, then you need to send its geographical coordinates, latitude and longitude.

To find your city’s coordinates, you can use a free tool like latlong.net and then you can save them in config.json file along with your API key, or you can write them directly in the code.

At this point the data format returned by the API call is the following:

{
  lat: 45.658,
  lon: 25.6012,
  temp: { value: 17.56, units: 'C' },
  precipitation: { value: 0.3478, units: 'mm/hr' },
  observation_time: { value: '2020-06-22T16:30:22.941Z' }
}

We can deconstruct this object and get the temp and the precipitation values and print them on the first and second row.

Step V - Wrap it Up

All we need to do now is to write the logic for our script, and update the LCD screen when new data arrives.

[RAW](https://carbon.now.sh/?bg=rgba(171%2C%20184%2C%20195%2C%201)&t=seti&wt=none&l=javascript&ds=true&dsyoff=20px&dsblur=68px&wc=true&wa=true&pv=56px&ph=56px&ln=false&fl=1&fm=Hack&fs=14px&lh=133%25&si=false&es=2x&wm=false&code=async%2520function%2520main()%2520%257B%250A%2520%2520await%2520printWeatherData()%253B%250A%250A%2520%2520setInterval(()%2520%253D%253E%2520%257B%250A%2520%2520%2520%2520printWeatherData()%253B%250A%2520%2520%257D%252C%25205%2520%252060%2520%25201000)%253B%250A%250A%2520%2520setInterval(async%2520()%2520%253D%253E%2520%257B%250A%2520%2520%2520%2520await%2520writeToLcd(8%252C%25200%252C%2520new%2520Date().toISOString().substring(11%252C%252019))%253B%250A%2520%2520%257D%252C%25201000)%253B%250A%257D%250A%250Alcd.on(%2522ready%2522%252C%2520main)%253B%250A%250A%252F%252F%2520*%2520If%2520ctrl%252Bc%2520is%2520hit%252C%2520free%2520resources%2520and%2520exit.%250Aprocess.on(%2522SIGINT%2522%252C%2520()%2520%253D%253E%2520%257B%250A%2520%2520lcd.close()%253B%250A%2520%2520process.exit()%253B%250A%257D)%253B)_

The weather data is updated every 5 minutes. But because we have a limit of 100 API Calls / Hour imposed by ClimaCell, we can go even further and update the weather data each minute.

For the current date, we have two options:

• we can use the property observation_time and display the date at which the data was received, or
• we can make a real clock and display the current time.

I chose the second option, but feel free to do it as you please.

To print the time in the upper right corner, we must first calculate the starting column so that the text fits snugly. For this we can use the next formula total columns number minus text to display length

The date has 8 characters and because he has 16 columns, we must start from column number 8.

The LCD setting is asynchronous, so we must use the method lcd.on() provided by the related library, so we know when the LCD has been initialized and is ready to be used.

Another best practice in embedded systems is to close and free the resources that you use. That’s why we use the SIGNINT event to close the LCD screen when the program is stopped. Other events like this one include:

• SIGUSR1 and SIGUSR2 - to catch "kill pid” like nodemon restart
• uncaughtException - to catch uncaught exceptions

Step VI - Run it Forever

The script is complete and at this point we can run our program. We just have one more thing we must do before we can finish.

At this point you’re probably connected to your Raspberry Pi using SSH or directly with an HDMI cable and a monitor. No matter what, when you close your terminal the program will stop.

At the same time if you power off your device and after some time or immediately power it on again, the script will not start and you’ll have to do it manually.

To solve this problem, we can use a process manager like pm2.

Here are the steps:

1. sudo npm install pm2 -g - install pm2
2. sudo pm2 startup - create a startup script for pm2 manager
3. pm2 start index.js - start an application
4. pm2 save - save your process list across server restart

Now you can reboot your board and the script will start automatically when the device is ready.

Conclusion

From this point you can customize your new device however you want. If you find this weather data important for you (or any other data from ClimaCell, like air pollution, pollen, fire index or road risk), you can create a custom case to put the Raspberry Pi and the LCD display in it. Then after you added a battery you can place the device in your house.

Raspberry Pi is like a personal computer, so you can do much more on it than you would normally do on a microcontroller like Arduino. Because of this, it's easy to combine it with other devices you have in your house.

A few days ago a Raspberry Pi I use for CI on my personal projects stopped working. The error was easily fixable, but since running anything on that PI was slow from day one, I decided not to proceed in that direction.

Also, because of the same issue I always wanted to switch to another OS – but I never had the proper motivation since everything worked on that Raspberry PI, right? Well the motivation came along with this article :)

Note: The original setup on the Raspberry PI was quite simple: Raspibian OS, VNC Server, and Jenkins.

For you that are new to Raspberries here is a little background at what they are, and what they can be used for.

Raspberry Pi's are basically small computers used most often used to learn programming skills, build hardware projects, do home automation, and some have even found usage in industrial applications.

There are a few different types of PI's on the market today. To learn more about them please visit the their Wikipedia page: https://en.wikipedia.org/wiki/Raspberry_Pi#Generations

Tools you'll needed to complete this tutorial:

• Raspberry PI
• Micro SD card
• Micro SD card adapter for your laptop
• Windows
• Mouse for Raspberry PI
• HDMI cable to connect your Raspberry PI to at TV or any other kind of screen
• Keyboard for Raspberry PI is good to have but not necessary as Ubuntu MATE offers on screen keyboard

Once we complete all the steps, this article will show you how to:

• Set up Ubuntu MATE on an SD card using Windows
• Install Ubuntu MATE to Raspberry PI
• Things to do after installing Ubuntu MATE

There are several OS options for Raspberry PI available, and Ubuntu Mate is just one option. For more info on other options you can check out this article:

Flashing Ubuntu MATE to Micro SD card

The SD card and USB sticks are called a Flash drives because they have a flash type memory in them. So flashing means creating a bootable drive with a specific operating system (OS) on it.

Navigate to the download page of Ubuntu MATE and download the recommended architecture image for Raspberry PI:

Download the Raspberry PI 32 bit version (recommended)

Setting Up the SD Card

After downloading the Ubuntu MATE image we need to write it to the SD card. To do that we will use the Balena Etcher tool. Etcher is a tool we will download from here:

Balena Ether download page

Install Etcher and launch it. Then select the Ubuntu MATE image file you previously downloaded, along with your SD card, and start the flashing process.

Note: Flashing Ubuntu MATE to the SD card takes some time, so feel free to get yourself a cup of coffee.

Balena Etcher decompressing Ubuntu MATE image

Balena Etcher validating Ubuntu MATE image

Once you are done with flashing, take out the SD card from the adapter and put it in the Raspberry PI. Also, this is the step where you will need to connect the PI with the mouse and a screen.

Installing Ubuntu MATE on Raspberry PI

Once you are done connecting all peripherals of the Raspberry PI, connect it to the power source and let it startup.

The Ubuntu MATE installation process is exactly the same as for ordinary Ubuntu. During this process you will be asked to to select your keyboard layout, timezone, username and password. Here is the step by step instalation guide for Ubuntu 18.04: Ubuntu installation.

After the installation is done you will be greeted by the desktop screen.

Ubuntu Mate desktop greeting screen

Things to do after installing Ubuntu MATE

Update the local database

This command updates/maps the local database of your local packages with updates, later allowing your system to fetch new versions of the packages.

sudo apt update

Upgrading the installed packages

This command fetches new versions of packages existing on the machine you previously mapped with sudo apt update command.

sudo apt upgrade

Note: Updating Ubuntu MATE takes a lot of time so feel free to get yourself another cup of coffee, and a pastry to go along with it. If you don't have any pastry around feel free to go to the nearest bakery, because by the time you get back you system will probably be 50% upgraded.

Installing the Ubuntu Software application

Software Boutique is the default software center on Ubuntu MATE. However, it has a very limited selection of applications, and one of the more interesting apps needed for my project is missing.

Luckily, since this is an Ubuntu distribution, there is a way to install the standard Ubuntu Software center which has more app choices.

Apparently there is a way of installing Ubuntu Software center via Software Bundle. But when selecting the option to doing that I was stuck and nothing seemed to happen for me so I just used the terminal command:

sudo apt install ubuntu-software

Bra gjort! You have just finished setting up the Ubuntu Mate OS on your Raspberry PI device!

Check out more articles like this on my freeCodeCamp profile, Medium profile, and other fun stuff I build on my GitHub page.

3 Frameworks for Machine Learning on the Raspberry Pi

The revolution of AI is reaching new heights through new mediums. We’re all enjoying new tools on the edge, but what are they? What products frameworks will fuel the inventions of tomorrow?

If you’re unfamiliar with why Machine Learning is changing our lives, have a read here.

If you’re already excited about Machine Learning and you’re interested in utilizing it on devices like the Raspberry Pi, enjoy!

Simple object detection on the Raspberry Pi

I’ve implemented three different tools for detection on the Pi camera. While it’s a modern miracle that all three work, it’s important for creators to know “how well” because of #perfmatters.

Our three contenders are as follows:

1. Vanilla Raspberry Pi 3 B+— No optimizations, but just using a TensorFlow framework on the device for simple recognition.
2. Intel’s Neural Compute Stick 2 — Intel’s latest USB interface device for Neural Networks, boasting 8x perf over the first stick! Around $80 USD.
3. Xnor.ai — A proprietary framework that reconfigures your model to run efficiently on smaller hardware. Xnor’s binary logic shrinks 32-bit floats to 1-bit operations, allowing you to optimize deep learning models for simple devices.

Let’s evaluate all three with simple object detection on a camera!

Vanilla Raspberry Pi 3 B+

A Raspberry Pi is like a small, wimpy, Linux machine for $40. It allows you to run high-level applications and code on devices like IoT made easy. Though it sounds like I can basically use laptop machine learning on the device, there’s one big gotcha. The RPi has an ARM processor, and that means we’ll need to recompile our framework, i.e. TensorFlow, to get everything running.

⚠️ While this is not hard, this is SLOW. Expect this to take a very… very… long time. This is pretty much the fate of anything compiled on the Raspberry Pi.

Setup

Here are all the steps I did, including setting up the Pi camera for object detection. I'm simply including this for posterity. Feel free to skip reading it.

Install pi, then camera, then edit the /boot/config.txt Add disable_camera_led=1 to the bottom of the file and rebooting.

Best to disable screensaver mode, as some follow-up commands may take hours

sudo apt-get install xscreensaver
xscreensaver

Then disable screen saver in the “Display Mode” tab.

Now get Tensorflow Installed

sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get update
sudo apt-get install libatlas-base-dev
sudo apt-get install libjasper-dev libqtgui4 python3-pyqt5
pip3 install tensorflow
sudo apt-get install libjpeg-dev zlib1g-dev libxml2-dev libxslt1-dev
pip3 install pillow jupyter matplotlib cython
pip3 install lxml # this one takes a long time
pip3 install python-tk

OpenCV

sudo apt-get install libtiff5-dev libjasper-dev libpng12-dev
Sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
sudo apt-get install libxvidcore-dev libx264-dev
sudo apt-get install qt4-dev-tools
pip3 install opencv-python

Install Protobuff

sudo apt-get install autoconf automake libtool curl

Then pull down protobuff and untar it.
https://github.com/protocolbuffers/protobuf/releases

Then cd in and then run the following command which might cause the computer to become unusable for the next 2+ hours. Use ctrl + alt + F1, to move to terminal only and release all UI RAM. Close x process with control + c if needed. You can then run the long-running command. Base username “pi” and password “raspberry”

make && make check

You can then install simply with

sudo make install
cd python
export LD_LIBRARY_PATH=../src/.libs
python3 setup.py build --cpp_implementation
python3 setup.py test --cpp_implementation
sudo python3 setup.py install --cpp_implementation
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=3
sudo ldconfig

Once this is done, you can clean up some install crud with sudo apt-get autoremove, delete the tar.gz download and then finally reboot with sudo reboot now which will return you to a windowed interface

Setup Tensorflow

mkdir tensorflow1 && cd tesorflow1
git clone --recurse-submodules \ https://github.com/tensorflow/models.git
modify ~/.bashrc to contain new env var named PYTHONPATH as such
export PYTHONPATH=$PYTHONPATH:/home/pi/tensorflow1/models/research:/home/pi/tensorflow1/models/research/slim

Now go to the zoo: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md We’ll take the ssdlite_mobilenet, which is the fastest! Wget the file and then tar -xzvf the tar.gz result and delete the archive once untarred. Do this in the object_detection folder in your local tensorflow1 folder. Now cd up to the research dir. Then run:

protoc object_detection/protos/*.proto --python_out=.

This converted the object detection protos files to python in the proto folder

Done Installing!!

Special thanks to Edje Electronics for sharing their wisdom on setup, an indispensable resource for my own setup and code.

Once I got Tensorflow running, I was able to run object recognition (with the provided sample code) on Mobilenet for 1 to 3 frames per second.

Vanilla Pi Results

For basic detection, 1 to 3 frames per second aren’t bad. Removing the GUI or lowering camera input quality speeds up detection. This means the tool could be an excellent detector for just simple detection. What a great baseline! Let’s see if we can make it better with the tools available.

Intel’s Neural Compute Stick 2

This concept excites me. For those of us without GPUs readily available, training on the edge instead of the cloud, and moving that intense speed to the Raspberry Pi is just exciting. I missed the original stick, the “Movidius”, but from this graph, it looks like I chose a great time to buy!

Setup

My Intel NCS2 arrived quickly and I enjoyed unboxing actual hardware for accelerating my training. That was probably the last moment I was excited.

Firstly, the USB takes a lot of space. You’ll want to get a cable to keep it away from the base.

That’s a little annoying but fine. The really annoying part was trying to get my NCS 2 working.

There are lots of tutorials for the NCS by third parties, and following them got me to a point where I thought the USB stick might be broken!

Everything I found on the NCS didn’t work (telling me the stick wasn’t plugged in!), and everything I found on NCS2 was pretty confusing. For a while, NCS2 didn’t even work on ARM processors!

?????????????????

After a lot of false-trails, I finally found and began compiling C++ examples (sorry Python) that only understood USB cameras (sorry PiCam). Compiling the examples was painful. Often the entire Raspberry Pi would become unusable, and I’d have to reboot.

locked up at 81% for 24 hours

The whole onboarding experience was more painful than recompiling Tensorflow on the raw Pi. Fortunately, I got everything working!

The result!? ??????????????????????

NC2 Stick Results

6 to 8 frames per second… ARE YOU SERIOUS!? After all that?

It must be a mistake, let me run the perfcheck project.

10 frames per second…

From videos on the original NCS on python I saw around 10fps.. where’s the 8x boost? Where’s the reason for $80 hardware attached to a $40 device? To say I was let down by Intel’s NCS2 is an understatement. The user experience and final results were frustrating, to put it lightly.

Xnor.ai

Xnor.ai is a self-contained software solution for deploying fast and accurate deep learning models to low-cost devices. As many discrete logic enthusiasts might have noticed, Xnor is the logical complement of the bitwise XOR operator. If that doesn’t mean anything to you, that’s fine. Just know that the people who created the YOLO algorithm are alluding to the use of the logical operator to compress complex 32-bit computations down to 1-bit by utilizing this inexpensive operation and keeping track of the CPU stack.

In theory, avoiding such complex calculations required by GPUs should speed up execution on edge devices. Let’s see if it works!

Setup

Setup was insanely easy. I had an object detection demo up and running in 5 minutes. 5 MINUTES!**

The trick with Xnor.ai is that, much like the NCS2 Stick, the model is modified and optimized for the underlying hardware fabric. Unlike Intel’s haphazard setup, everything is wrapped in friendly Python (or C) code.

model = xnornet.Model.load_built_in()

That’s nice and simple.

But it means nothing if the performance isn’t there. Let’s load their object detection model.

Again, no complexity, they have one with no overlay, and one with. Since the others (except for perfcheck on NCS2) were with overlays, let’s use that.

Xnor.ai Results

JAW… DROPPING… PERFORMANCE. I not only get a stat on how fast inference could work, but I also get an overall FPS with my overlay that blew everything else out of the water.

OVER 12FPS and an inference speed over 34FPS!?

This amazing throughput is achieved with no extra hardware purchase!? I’d call Xnor the winner at this point, but it seems a little too obvious.

I was able to heat up my device and open a browser in the background to get it down to 8+ FPS, but even then, it’s a clear winner!

Xnor hype is real

The only negative I can give you on Xnor.ai is that I have no idea how much it costs. The Evaluation model has a limit of 13,500 inferences per startup.

While emailing them to get pricing, they are just breaking into non-commercial use, so they haven’t created a pricing system yet. Fortunately, the evaluation model would be fine for most hobbyists and prototypes.

In Summary:

If you need to take a variety of models into account, you might be just fine getting your Raspberry Pi setup from scratch. This would make it a great resource for testing new models and really customize your experience.

When you’re ready to ship, it’s no doubt that both the NCS2 and the Xnor.ai frameworks speed things up. It’s also no doubt that Xnor.ai outperformed the NCS2 in both onboarding and performance. I’m not sure what Xnor.ai’s pricing model is, but that would be the final factor in what is clearly a superior framework.

Post Publish Updates:

This is an excellent blog post on setting up the NCS2

Getting Started with the Intel Neural Compute Stick 2 and the Raspberry Pi
_Getting started with Intel’s Movidius hardware_medium.com

Additionally, if you’re looking to play around with Xnor.ai, the link is www.xnor.ai/ai2go

Gant Laborde is Chief Technology Strategist at Infinite Red, a published author, adjunct professor, worldwide public speaker, and mad scientist in training. Clap/follow/tweet or visit him at a conference.

Expect more awesome edge blog posts coming soon!

Have a moment? Read more by Gant

Avoid Nightmares — NSFW JS
_Client-side indecent content checking for the soul_shift.infinite.red

With a Raspberry Pi, low-cost gas sensors, and a remote-controlled switch, you can control the air quality in your house.

The air we breathe indoors is not always healthier than the air outside.

According to a study of the E.U. Joint Research Centre, you can find a wide range of air pollutants indoors. Some of them are toxic, or can cause genetic mutations or cancer. Factors that influence indoor air quality are:

• ambient air, or the air outdoors
• air tightness and ventilation of the building
• indoor sources like tobacco smoke, heating gases, consumer products, etc.

Do you know how much time you spend indoors? According to the Environmental Protection Agency, Americans spend 87% of their time indoors. In Europe, this average percentage is 90%, according to the JRC. And the more time we spent indoors, the more pollutants we inhale.

So it would be interesting if we track the indoor air quality. In this article, I will explain how I did this with a Raspberry Pi, a GrovePi and some sensors. We will upload the sensor data to a Firebase database and visualize trends with Dash.

When pollutant levels reach an unhealthy level, we can send an alert notification to warn us.

Moreover, it would be great if we could automatically start up the ventilation to purify the air. This can be done with a remote-controlled switch from Energenie.

Raspberry Pi and GrovePi

Raspberry Pi model 2B

In this project, I will use a Raspberry Pi model 2B. The Raspberry Pi is a low-cost small computer. It enables programmers and makers to build anything they can imagine.

GrovePi (blue board) attached to a Raspberry Pi

The GrovePi is an electronics board that we attach to the Raspberry Pi. It makes connecting a wide range of sensors easy. As such, you don’t need to bother about a breadboard, resistors, soldering or jumper wires. You can plug in a connector and start working with it.

Data flow

In the illustration below you see how the sensor data will flow from the sensors to the charts on a web page. We’ll be using Python to do all that.

Data flow from a GrovePi and a Raspberry Pi to Firebase to a web page hosted on PythonAnywhere.

The first step is to pull in the data from the Grove sensors. Then we process the data on the Raspberry Pi and send them to a Firebase database. This data is also used to switch on or off a ventilation unit via a remote controlled switch.

We can get the stored data with a script running on PythonAnywhere.com. With the Dash package, we can build a dashboard to follow up the indoor air quality.

Gas sensors

We will use a set of three different gas sensors for this project. I use Grove gas sensors and connect them to the GrovePi.

It is not necessary to use all three sensors in your own project. Feel free to choose sensors that fit your needs. You can do that in the config.py file by keeping the sensors you need in the Python dictionary MQ_SENSORS.

Grove gas sensors

Each sensor detects a specific set of gases. There is quite some overlap between the sensors with regard to the gases they detect. But, the range in which the sensors detect a gas can differ. You can find the ranges on the website of Seeedstudio.

Gases detected per sensor type

Temperature and Humidity sensor

Temperature and humidity influence the readings from the gas sensors. So, it is interesting to measure temperature and humidity as well. We use the Grove BME680 sensor for that.

Grove BME680 to measure temperature and humidity

Working with sensor data

In short, a gas sensor will output a higher voltage when the concentration of a gas is higher. This is because the built-in resistor varies its resistance (Rs) according to the concentration of the gas.

The sensor value only reflects an approximation of a trend in the gas concentration. This means that you can use it to show whether the gas concentration increases or decreases. It does not give the exact gas concentration. If you want to measure the actual concentration, you would need a more expensive sensor.

For learning purposes, we will use the sensor value to approximate the gas concentration. You should keep that in mind. You should not use these scripts in real-life situations to prevent intoxication by these gases!

On the datasheets listed below, you find a graph with the relation between the gas concentration and sensor resistor values. The gas concentration is expressed in parts per million (ppm). The resistor values as the ratio Rs/R0.

• MQ2 datasheet
• MQ5 datasheet
• MQ9 datasheet

The curves below correspond to the standard conditions of:

• a temperature of 20°C
• a humidity of 65%
• an oxygen concentration of 21%
• a load resistance of 5 kilo-Ohm. The load resistance is the total resistance of an electronic circuit.

Curves displaying the relation between gas concentrations and sensor resistance values. Both the x-axis and y-axis are on a log scale.

We can assume that the load resistance and oxygen concentration are stable over time. Yet, the temperature and humidity indoors can vary. Both factors have an influence on the sensor readings, as you can see in the graph below.

Influence of different temperatures and humidities (RH) on sensor resistance values in the MQ2 sensor for a hydrogen (H2) concentration of 1.000 ppm.

To get accurate readings, you should have a graph or look-up table for various temperature-humidity combinations. Unfortunately, these graphs are not provided by the manufacturer.

Another approach is to correct the sensor readings for the temperature-humidity influence.

We can do this with artificial neural networks, as proposed in the paper by Nenova & Dimchev. That approach requires ground-truth measurements of the gas concentrations. This is out of scope for this article.

Defining the R0 value

First, we need to compute the R0 value. R0 stands for the sensor resistance value of 1.000 ppm of hydrogen (H2)in clean air. The ratio for clean air (black line with blue crosses) is constant. It remains at 9.8 regardless of the concentration of clean air. So, we can compute R0 by reading the sensor value (Rs) in clean air and dividing it by 9.8.

The value we get from the sensor is a value between 0 and 1.023 and has no measurement unit. So, to get the output voltage we divide the sensor value by 1.023. We multiply this value with the circuit voltage, which is usually 5V.

From the sensor voltage, we can derive the sensor resistance by applying Ohm’s law. The sensor resistance is equal to

Vc is the circuit voltage and Vs is the sensor voltage

Let’s see how we do that with Python. The complete code and more documentation can be found on Github in the script get_R0_values.py.

import config as cfg
import grovepi
import time

First, we import some packages. The config package is a Python script I made to store all configuration parameters.

Throughout the code, you’ll notice that I sometimes refer to cfg.PARAMETER_NAME. We set thePARAMETER_NAME value in that config.py file. It also contains some passwords and API tokens.

For security reasons, I will not save that file on Github. Instead, I’ll provide a clean template config_template.py that you can use for your own project.

Next, we import the grovepi package. You can install it from the Github page of Dexter Industries on your Raspberry Pi. It allows us to work with the GrovePi and the connected sensors.

Finally, we use the time package to pause the program during sensor readings.

mq_values = {}

for sensor, data in cfg.MQ_SENSORS.items():
    grovepi.pinMode(data['pin'],"INPUT")
    mq_values[sensor] = 0

We will store the sensor values for the different sensors in the dictionary mq_values. But first, we initialize the values to zero in a loop over the sensors defined in MQ_SENSORS.

With the pinMode method, you can define the pin as INPUT or OUTPUT. In our case, we’ll use it as INPUT.

The pin tells us to which pin on the GrovePi the sensor is connected. We use the analog pins (or ports) which are labeled as A0, A1 and A2 on the GrovePi. Make sure you connect the right sensor to the port described in the config.py file.

for i in range(cfg.NB_R0_READ):
    for sensor, value in mq_values.items():
        mq_values[sensor] += grovepi.analogRead(cfg.MQ_SENSORS[sensor]['pin'])
    time.sleep(cfg.R0_INTERVAL)

We then read the sensor value in a loop for cfg.NB_R0_READ times and sum it up in mq_value[sensor]. We read the sensor value with the analogRead method of the grovepi package.

As described in the documentation, this will return a value between 0 and 1.023. In fact, it converts an analog sensor value to a digital value.

After one reading for all sensors, we pause the program at cfg.R0_INTERVAL seconds. To get the average value, we divide the cumulated value by cfg.NB_RO_READ.

for sensor, value in mq_values.items():
    mq_values[sensor] = mq_values[sensor]/cfg.NB_R0_READ
    mq_values[sensor] = mq_values[sensor]/cfg.AR_MAX * cfg.VC
    mq_values[sensor] = (cfg.VC - mq_values[sensor])/mq_values[sensor]
    mq_values[sensor] = mq_values[sensor]/cfg.MQ_SENSORS[sensor]['r0_rs_air']

We compute the sensor voltage by dividing the averaged sensor value by cfg.AR_MAX. Then we multiply it by the circuit voltage cfg.VC.

From that voltage, we can apply Ohm’s law and compute the sensor resistance value. Dividing that by the ratio for clean air cfg.MQ_SENSORS[sensor]['r0_rs_air'] gives us R0.

It is better to have the sensor working at least for 24 hours before you measure the R0 value. This will give more stable sensor readings.

Linear interpolation of gas concentration

Now that we know the R0 value, we can compute the Rs/R0 ratio with the sensor value. With that ratio, we can derive the gas concentration with linear interpolation.

For that, we assume that we are working in the standard conditions described for the first graph. In that case, the curves are nearly linear.

For linear interpolation, we need two known points of each curve to calculate its slope. Suppose we have two points with the coordinates (x1, y1) and (x2, y2). The y-values stand for the Rs/R0 values and the x-values for the gas concentrations. For a linear curve, the slope is then calculated as:

When we know the slope, we can find the gas concentration(x) for any given Rs/R0 value (y). The formula for this is:

The code snippets below come from the script get_sensor_values.py which you can find on Github.

We put the formula in a function get_ppm. curve['y'] and curve['x'] are the known points on the curve for a gas. curve['slope'] is what we calculated with the previous formula.

You can find these values in the config_template.py file. I derived these values with Webplotdigitizer from the graphs on the data sheets. As a result, these values are not completely accurate. Use them with caution.

Note the use of np.log10 and np.power. This reason for this is that the axes on the graph are in a log-scale.

def get_ppm(Rs_R0_ratio, curve):
    x_val = (np.log10(Rs_R0_ratio) - curve['y'])/curve['slope'] + curve['x']
    ppm_val = np.power(x_val, 10)
    return ppm_val

We calculate the Rs_R0_ratio in the same manner as when calculating the R0 value. So I will not repeat this. To calculate this ratio, we loop over all gases and sensors and store this in ppm_values[mq_sensor][gas].

for gas, curve in cfg.CURVES[mq_sensor].items():
                ppm_values[mq_sensor][gas] = get_ppm(mq_values[mq_sensor], curve)

Temperature, humidity and pressure

Besides the gas sensors, we read the temperature, humidity and pressure with the BME680 sensor. The BME680 sensor is connected to the GrovePi via an I2C port. To use the sensor, we import the package which can be installed from the Pimoroni repo on Github.

import bme680

The set_..._oversample methods specify how many samples we take to calculate the average value. We also did that for the gases. With get_sensor_data we read the sensor values.

bme680_sensor = bme680.BME680(bme680.I2C_ADDR_PRIMARY)
bme680_sensor.set_humidity_oversample(bme680.OS_2X)
bme680_sensor.set_pressure_oversample(bme680.OS_4X)
bme680_sensor.set_temperature_oversample(bme680.OS_8X)
bme680_sensor.get_sensor_data()

bme680_sensor.get_sensor_data()

Storing and retrieving sensor data in Cloud Firestore

Some sensors provide new readings very fast. Other (less expensive) sensors will take more time. For this project, we will read the data every minute. This is set in the config file with FIREBASE_INTERVAL = 60.

In the free Spark plan of Firebase, the Cloud Firestore quota allow for 20K writes per day. With a one-minute interval, we will be well below that quota. The limit for reading documents in the Firestore is 50K per day.

You’ll need to create a Firebase project and create a Cloud Firestore. After that, make sure to generate the credentials to authenticate your application. Save the credentials file in a secure location.

To work with Firebase via Python, we need to import the firebase_admin package. This package needs to be installed on your Raspberry Pi first, if needed.

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

After that, we can initialize the Firebase app with credentials. I store the location to the credentials file in cfg.FIREBASE_CREDS_JSON. When the app is initialized, we create a Firestore object db.

firebase_path = Path.cwd() / cfg.FIREBASE_CREDS_JSON
cred = credentials.Certificate(str(firebase_path))
firebase_admin.initialize_app(cred)

After processing the gas values, it is time to store them in the Cloud Firestore. We will create a dictionary firebase_values to gather all the data. With a dict comprehension, we add the values for all gases for all MQ sensors. The BME680 values and timestamp are also added.

With the add method of the Firestore object db, it is easy to store the data in the Firestore.

The name of the collection in the Firestore is cfg.FIREBASE_DB_NAME. Learn more about the data model of Firestore on the Firebase website.

firebase_values = {mq_sensor + '_' + gas + '_ppm': ppm
                            for mq_sensor, gases in ppm_values.items()
                            for gas, ppm in gases.items()
                          }
firebase_values['temperature'] = bme680_sensor.data.temperature
firebase_values['pressure'] = bme680_sensor.data.pressure
firebase_values['humidity'] = bme680_sensor.data.humidity
firebase_values['date'] = datetime.now()
db.collection(cfg.FIREBASE_DB_NAME).add(firebase_values)

After storing the data, we wait a minute to start over with the following line of code.

time.sleep(cfg.FIREBASE_INTERVAL)

Improving the air quality

If the air quality indoors is not good we should take measures to improve it. Before we can do that, we need to be notified about critical gas concentrations.

One possibility is to send an alert notification by email, which we’ll discuss below.

Sometimes the source of indoor air pollution comes from outdoor air. For example, when your neighbors have wood-burning stoves or when you live near a factory.

In that case, you could install your measurement station outside and turn off the ventilation unit in your house if outdoor air quality is bad. With a remote-controlled switch, this can be done easily.

All code for this section is in improve_air_quality.py on Github.

Sending notifications when air quality reaches a critical level

We do not want to send an email each time the sensor outputs critical values (here, each minute).

Let’s say we want to check each hour whether there were critical values in the last hour. For that, we need to keep track of a reference_time. We initialize this when the program for readings sensor values starts. The interval at which we check again for critical gas concentrations is defined in cfg.ALERT_INTERVAL.

reference_time = datetime.now()

When an hour has passed since reference_time, we can start to check if there were critical air pollutant values. We update reference_time with the current time.

With the pytz package, we can take into account our timezone. In my case, that is Europe/Brussels. We compute one_hour_ago by subtracting 60 minutes from the current time.

if datetime.now() > reference_time + timedelta(minutes=cfg.ALERT_INTERVAL):
    reference_time = datetime.now()
    brussels_tz = pytz.timezone('Europe/Brussels')
    prev_check_time = brussels_tz.localize(datetime.now()) - timedelta(minutes=cfg.ALERT_INTERVAL)

With prev_check_time we extract the sensor readings from Firebase of the last hour. We do that by applying a where clause to the data that we get from the Cloud Firestore.

In this script, we will only use one gas sensor and a limited set of gases. The sensor is selected in cfg.ALERT_SENSOR. The gases are selected in cfg.ALERT_GASES. The data per gas is appended to ppm_vals as well as the timestamps.

timestamps = []
ppm_vals = {}
for gas in cfg.ALERT_GASES:
    ppm_vals[gas] = []

docs = db.collection(cfg.FIREBASE_DB_NAME).order_by(u'date').where(u'date', '>=', one_hour_ago).get()

for doc in docs:
    data = doc.to_dict()
    for gas in cfg.ALERT_GASES:
        ppm_vals[gas].append(data[cfg.ALERT_SENSOR + gas + '_ppm'])

    timestamps.append(data['date'].strftime('%H:%M:%S'))

We look for critical values with the function find_crit_val. We will only check if the value surpassed an upper bound ubound. These upper bounds need to be specified in the config file.

The data are in ascending chronological order. Thus, we can use the next method to find the first timestamp for which v > ubound. We return a tuple containing the timestamp of the critical value and the critical value itself.

If there is no critical value, we return the tuple (None, None).

def find_crit_val(timestamps, val_list, ubound):
    try:
        (crit_time, crit_value) = next(((i,v) for i, v in zip(timestamps, val_list) if v > ubound))          
    except:
        (crit_time, crit_value) = (None,None)
    return (crit_time, crit_value)

The critical tuples are stored in a dictionary crit_dict. As the key, we use the gas name. We then check for gas-sensor combinations with a critical timestamp and critical value. In that case, we add an alert message to critical_msg.

crit_dict = {}
for gas in cfg.ALERT_GASES:
    (crit_time, crit_value) = find_crit_val(timestamps, ppm_vals[gas], cfg.UPPERBOUNDS[gas])
    crit_dict[gas] = (crit_time, crit_value)
critical_msg = ''
    for k, v in crit_dict.items():
        if v[0] is not None and v[1] is not None:
            critical_msg += '\nCritical value for ' + k + ' of ' + str(v[1]) + cfg.UNITS[k] + ' at ' + str(v[0])

If critical_msg is not empty, we send an email. Sending an email is done with the smtplib package. How to send an email with Python is explained on AutomateThe BoringStuff.com.

You need to generate an application-specific password for your email if you are using Google’s two-factor authentication.

if critical_msg != '':
    try:
        msg = MIMEText(critical_msg, _charset='utf-8')
        msg['Subject'] = Header('Air Quality Alert', 'utf-8')
        smtpObj = smtplib.SMTP('smtp.gmail.com', 587)
        smtpObj.ehlo()
        smtpObj.starttls()
        smtpObj.login(cfg.EMAIL, cfg.EMAIL_PW)
        smtpObj.sendmail(cfg.EMAIL, cfg.EMAIL, msg.as_string())
        smtpObj.quit()  
    except smtplib.SMTPException:
        print('Something went wrong while sending the email')

Automatically control your ventilation unit

With a remote-controlled switch, we can turn on or off any device that is connected to it. Thus, also a ventilation unit. Energenie creates the PiMote add-on specifically for the Raspberry Pi. To control the Energenie switch, you need to install the energenie package and import it.

Note that you can not attach the PiMote on top of the GrovePi. So you’ll need a second Raspberry Pi.

When starting the script, the first thing we do is define a boolean variable ventilation_on. We set it to False because this is the first time we run the script.

import energenie
ventilation_on = False

If critical_msg is not empty, there was a critical gas concentration in the last alert interval. In that case, we turn on the ventilation with the switch_on method of the energenie package.

If there was no critical gas concentration and the ventilation was switched on in the last alert interval, we can switch it off.

You might need to set another interval before switching your ventilation off. That depends on the flow rate of your ventilation unit, the gases measured and whether the pollution source has been disabled.

if critical_msg != '':
    if not ventilation_on:
        energenie.switch_on(1)
        ventilation_on = True
else:
    if ventilation_on:
        energenie.switch_off(1)
        ventilation_on = False

Visualization of air quality with Dash

A notification with critical gas concentrations can help to take immediate action. But it is also interesting to track the gas concentrations over time. By visualizing the sensor values in a dashboard, we can look at the trend of the gas concentrations. This can be done with Dash. On the website of Dash, you can find a great tutorial on how to get started.

In this project, we will build a dashboard and host it on PythonAnyWhere.com. To use Dash on PythonAnywhere, you need to create a virtual environment. You can follow the steps of this demo on how to set-up a Dash app on PythonAnyWhere.

Below I will show how I built the dashboard for our air quality station. The full script can be found in plot_sensor_values.py on Github.

First of all, you need to import the dash package.

import dash
import dash_core_components as dcc
import dash_html_components as html

In the demo on the Dash website, they use a link to a Cascading Style Sheet (CSS) to provide a nice page design. If you want to use a local CSS on your laptop or web server, you can add an assets folder. In that folder you can add your CSS and Dash will pick it up from there.

Then you’ll need to get the data from Firebase. This can be done similarly as we did for sending the alert notifications. So we will not go over that again.

With the data collected from Firebase, we can fill the graphs in our dashboard. We first create a Dash object app and give it a title.

app = dash.Dash(__name__)
app.title = 'Indoor Air Quality Dashboard'

Then we create the layout of the dashboard. A H1 heading component, a container div and a div containing the graphs.

app.layout = html.Div([
    html.H1(style={'textAlign':'center'}, children='Indoor Air Quality Dashboard'),
    html.Div(id='container'),
    html.Div(graphs)
])

graphs is a list that contains the info per graph. Below you can see how the graph for temperature is set up. You can add the dcc.Graph for humidity and pressure as well by appending them to graphs.

graphs = [
    dcc.Graph(
        id='temperature',
        figure={
            'data':[{
                'x':timestamps,
                'y':temperatures,
                'type':'line',
                'name': 'Temperature',
                'line': {'width':2, 'color': '#542788'}
                }],
            'layout':{
                'title': 'Temperature',
                'yaxis': {'title': 'Celsius'},
                'xaxis': {'title': 'Timestamp', 'tickvals':timestamps}
            }
        }
    )
]

The graphs for the MQ sensors can be appended in a for loop.

for mq_sensor in cfg.MQ_SENSORS.keys():
    for gas in cfg.CURVES[mq_sensor].keys():
        sensor_gas_key = mq_sensor + '_' + gas + '_ppm'
        title = gas + ' concentration on '+ mq_sensor + ' sensor'
        data = ppm_values[mq_sensor][gas]
        data.reverse()

        graphs.append(dcc.Graph(
            id=sensor_gas_key,
            figure={
                'data': [{
                    'x': timestamps,
                    'y': data,
                    'type':'line',
                    'name': title,
                    'line': {'width':2}
                }],
                'layout': {
                    'title': title,
                    'yaxis': {'title': 'ppm'},
                    'xaxis': {'title': 'Timestamp', 'tickvals':timestamps}
                }
            }
        ))

As a result, you will have a dashboard with graphs like the one below.

Graph with MQ2 sensor data for methane (ch4)

Conclusion

With a few low-cost gas sensors and a Raspberry Pi (and GrovePi) it is easy to build an air quality measurement station. You can then act on the data by sending alert notifications when air quality is bad or switch on the ventilation. With Dash you can build beautiful visualizations to monitor the air quality over time.

Below I noted some ideas to take this project further.

• make a mobile app for the visualizations and receiving notifications
• add LEDs, a buzzer and an OLED display to the Raspberry Pi to get instant feedback on air quality
• install measurement stations at your family and friends' place and visualize it on an interactive map.
• once you have enough data, build a time-series model to predict the indoor air quality

Hopefully, this story motivates you to start building your own measurement station. If you have questions or suggestions let me know.

In this article, I’ll discuss how I built a Smart Home Automation System with Angular and Node.js on a Raspberry Pi without relying on any external cloud services.

Intro

Over the last few days, I spent some nights designing and developing a home automation system based on JavaScript, with the use of Angular and Node.js. And, like with any other project, the planning involved some deep research on the internet.

It turned out that there are plenty of fish in the sea —plenty of solutions on how to implement a home automation system. Some offer paid services in “the cloud” and others explain how to build your own using a technology called MQTT.

None of the solutions made any sense to me. All options were either expensive, or had inconvenient implementations or even security flaws.

But, before we go any further, let’s explain what MQTT is. MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight messaging protocol. MQTT is designed for constrained devices and low-bandwidth, high-latency, or unreliable networks.

The design principles are to minimise network bandwidth and device resource requirements whilst attempting to ensure reliability and some degree of assurance of delivery. These principles also turn out to make the protocol ideal for the emerging “machine-to-machine” (M2M) or “Internet of Things” world of connected devices, and for mobile applications where bandwidth and battery power are at a premium.

MQTT Publish/Subscribe Architecture (Image source: HiveMQ.com)

Why wasn’t I convinced about using MQTT, or by any of the solutions I found on the internet? Two reasons:

1. While the MQTT technology seems very convenient for IoT devices, I still thought it was unnecessary. The system which I will be showing in the following tutorial operates in the same medium where the IoT devices live. All benefits that MQTT has for being “fast” and having “low-bandwidth” become irrelevant. Plus there’s all the hassle that is involved in its implementation and all the extra overhead with the additional npm packages that are required for it to work in a JavaScript environment. Instead, I will be using generic JavaScript and Node.js libraries only, nothing more!
2. What about the security part? Well, I’m not a big fan of “the cloud” or cloud computing in general. In some cases it can be very beneficial, but in most cases it’s just unnecessary. Think about it: why would you have a service that is required for controlling your home appliances to be hosted somewhere else in “the cloud” and not in your own network?

_Comic by [Geek and Poke](http://geekandpoke.typepad.com/geekandpoke/2009/11/good-consultants.html" rel="noopener" target="blank" title=")

One might think that “the cloud” provides the ability to access your home appliances from anywhere in the world via the internet.

But think about this: when your home network doesn’t have internet connection, “the cloud” becomes redundant. More importantly, you can still make your home automation system accessible from the internet using port-forwarding even if it’s hosted on your local network.

This is when it “clicked” for me, and I thought about hosting the whole system on a Raspberry Pi and keeping it in my local network.

A Raspberry Pi 3 Model B

The technology

1. Software: The reason why I chose Angular and Node.js is because they’re based on JavaScript and I’m already familiar with it. After all, I wanted to design and develop a progressive web app that communicates with my IoT devices via HTTP — and JavaScript offered all the functionality that I needed.
2. Hardware: The system works with microcontrollers like the Arduino Uno/Mega/Du/MKR1000, Adafruit HUZZAH CC3000, and any other microcontroller with a WiFi connection. I am using the ESP8266 as a base component for my home automation system. It’s a low-cost WiFi microchip with microcontroller capability. It has everything I need and for a cheap price! Lastly, we need to host the system somewhere on our local network — so what’s better than the Raspberry Pi?

This won’t be a coding tutorial where I dive deep into coding, since this project is open-source and I will be publishing everything on GitHub. I will only demonstrate how to implement your own home automation system and will be going through each step. If you’re a developer please fork the repository and get involved in improving it.

The setup

I estimate that it will take about 40 minutes to finish this whole setup plus any time spent online searching for solutions for installation errors.

What is needed?

A Raspberry Pi is required. In my example I’m using a Raspberry Pi 3, but it should work with most versions. The components needed are:

1. Raspberry Pi board
2. MicroSD card (A class 10 with 16 GB or higher is recommended)
3. A USB MicroSD card reader or SD card adapter
4. HDMI monitor and a USB keyboard (only required temporarily for the first boot of the Raspberry Pi)
5. Ethernet cable (not needed for Raspberry Pi 3 as it has built in WiFi)

Installing Raspbian OS on the Raspberry Pi

Raspbian is a free operating system based on Debian Linux, and it is optimized for Raspberry Pi.

I recommend the headless “LITE” version. It has no desktop environment or any graphical user interface, and it’s remotely accessible from a computer or device on the same network via SSH. We’re keeping things simple since this is the only way we are going to access the Raspberry Pi. The LITE version has all the functionality we’re looking for.

1. Download the latest Raspbian image from the official Raspberry Pi website.
2. Flash the Raspbian OS image to the SD card with Etcher or any other OS image burning software of your choice.

Setting up the Raspberry Pi

To get the Raspberry Pi ready to boot we need to:

1. Insert the MicroSD card into the Raspberry Pi
2. Connect the USB keyboard and the HDMI cable
3. Connect the Ethernet cable or if you have a Raspberry Pi 3 and want to use WiFi you should set up the network in the next section

When the Raspberry Pi has finished booting up, log in using username pi and password raspberry

Enabling WiFi and connecting to the network

Skip this step if you chose to connect with an Ethernet cable.

1. Open the “wpa-supplicant” configuration file

$ sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

2. add the following at the bottom of the file while adding your wifi name and password:

network={

   ssid="your_networks_name"   psk="your_networks_password"

}

3. Press Ctrl+X to save the code. Confirm with Y then Enter

4. Reboot the Raspberry Pi with the following command:

$ sudo reboot

Enabling SSH and changing the username and password

Now that the Raspberry Pi is connected to the internet, it’s recommended to change the default password.

1. Open the Raspberry Pi configuration tool and click the second option “Change User Password” and follow the instructions

$ sudo raspi-config

2. Select option 5 “Interfacing Options” then activate SSH

3. Reboot the Raspberry Pi. When it’s up, you have SSH enabled and it’s ready to be accessed remotely from your desktop computer

$ sudo reboot

Configuring remote access to the Raspberry Pi

Now, finally, the part when we install the required software on the Raspberry Pi. This part can be executed directly on the Pi through the terminal using a HDMI monitor and a USB keyboard. For convenience — and since we enable remote SSH connection — we’re going to connect from another desktop environment. This is the best and easiest way to remotely access and control the Pi whenever changes and configurations are needed.

So, basically, this is how you can access the command line interface of a Raspberry Pi remotely from another computer or any device on the same network using SSH. This can be done in two ways:

1. Using the Command Prompt or PowerShell (I’m using Windows on a Desktop computer), replace with your username and IP address

$ ssh username@ipaddress

If you do not know the IP address, type “hostname -I" in the Raspberry Pi command line.

2. The second method is using a client program like PuTTY or any other functioning client SSH software. Here’s an easy guide for using PuTTY.

Installing the required software on the Raspberry Pi

Before installing anything, it’s recommended to update the Raspberry Pi’s operating system and packages. Doing this regularly will keep it up-to-date.

1. Update the system packages list using the following command:

$ sudo apt-get update

2. Upgrade all your installed packages to their latest version:

$ sudo apt-get dist-upgrade

3. Download and install the latest version of Node.js:

// To download$ curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -

// To install$ sudo apt-get install -y nodejs

// Check if the installation was successful:$ node -v

4. Install the Angular CLI globally:

$ npm install -g @angular/cli

5. Install the Git version-control system:

$ sudo apt-get install git

Installing the database (MongoDB)

We need a database for storing the registered users and their credentials. Here are the required steps:

1. Install MongoDB

$ sudo apt-get install mongodb

2. Start the MongoDB process

$ sudo service mongodb start

3. Start the mongo Shell

$ mongo

3. Create a database called “smarthaus”

$ use smarthaus

In MongoDB, default database is test. If you didn’t create any database, then collections will be stored in test database.

Installing Smart Haus

1. Check the current work directory using this command:

$ pwd

/* It will probably print "/home/pi"   where "pi" is the current user directory */

It’s recommended to clone the project’s repository under the pi’s user directory but you can navigate somewhere else if you’re sure.

2. Clone the repository from:

$ git clone https://github.com/ameer157/smarthaus.git

Make sure to navigate inside the directory using:

$ cd smarthaus

Before installing any npm packages using “npm install” please refer to the npm guide to fix permissions to learn how to fix any “EACCESS ”errors you might face during installation. This is very important since it will prevent any npm permission errors, and allows you to install packages globally without using sudo. Using sudo with npm is not recommended and should be avoided.

3. Install all the required packages for the project:

$ npm install

Starting the Node.js server

Before starting the server we need to build the project using the Angular CLI tool. And lastly, we configure the Raspberry Pi so that it runs the server whenever it boots up.

1. Build the project using:

$ ng build --prod

2. Edit the rc.local file using nano:

$ sudo nano /etc/rc.local

3. Add the following on the line before exit 0 then exit and save the file:

su pi -c 'cd /home/pi/smarthaus/backend && sudo node server.js > log.txt &'

The Node.js server is now ready! It will run on every system boot up and save logs in under the same directory in a “log.txt” file.

Let’s run it now and see if it works using this command:

$ sudo node server

The system in now accessible from any device on your network via the Raspberry Pi’s IP address.

Please go ahead and fork this project and get involved in developing the missing parts ?

The end

We got ourselves a working home automation system running safely on a Raspberry Pi in our local network without the use of “the cloud” or somebody else’s server.

Real-time device status synchronization

Adding a new device synchronising data on-demand

My Raspberry Pi sitting next to my Fingbox and router in the living room ?

Rick and Morty providing tech support ??

I hope you enjoyed reading,
Please follow and share for more tech stuff ??

In simple terms, Raspberry Pi is a super cheap ($40) Linux based computer. That’s it. Seriously.

It can do whatever you can imagine a normal Linux computer can do, such as browse the web, write code, edit documents, and connect to I/O devices such a thumb drive, mouse, keyboard, etc. This tutorial will be focused on learning how to make your own Python dev-server with Raspberry Pi.

Step 0. Define the goal

Before we begin, it is important to understand what is it that we are trying to build. By the end of the tutorial, you will be able to run a basic website (using Flask) off of a Raspberry Pi on your local home network.

The goal of this tutorial is to demonstrate how a Pi can be used as a dev-server, more specifically, the example will be to host a simple website (using Flask).

Step 1. State the assumptions

Here are some assumptions that this tutorial will make:

1. You already have a Raspberry Pi set up with Raspbian OS. Here is a useful setup guide if you need one.
2. The Pi is connected to your home WiFi (and that you know the Pi’s IP address).
3. You will not require a screen going forward. assuming points 1 and 2 are complete.

We will use VS Code with the Remote VSCode extension to remotely create and edit files on the Pi. I definitely recommend that you use these two to follow along. Also, these will make working with remote files a lot easier, so that’s a plus.

Step 2. Find the Pi’s IP address

First, connect the Pi to a power supply, and ensure that it is correctly booted up and connected to the WiFi/Ethernet (basically, it needs to have an internet connection).

We will use ssh to connect to and communicate with the Pi. To do that remotely using a laptop, you need to know its IP address. This can be easily obtained using your ISP’s admin portal (usually available at http://192.168.0.1. Please note that this could be different for different ISPs.)

Usually, you should have your Pi connected to an address which may look similar to ‘192.168.0.12.’ Again, this will be different for different people. So please use the IP address that you found for your Pi in the admin portal. Going forward, this tutorial will use 192.168.0.12 as the Pi’s IP address.

Step 3. Connect to the Pi using ssh

Open VS Code and its built-in terminal window on your laptop. Connect to the Pi with an IP address of 192.188.0.12 using the following ssh command:

ssh -R 52698:localhost:52698 pi@192.168.0.12

The above command will set up a 2-way communication channel between your laptop and the Pi. If this is the first time you are connecting to the Pi, use raspberry as the password. You may be prompted to change your default password. It is highly recommended that you do so.

Terminal window after successfully connecting to the pi

Step 4. Create a project directory

You should now be in the home directory of the Pi. Let’s create a directory for the website we wish to build. Use the following command to create the directory:

mkdir MyFlaskWebsite

Use the ‘ls’ command to verify that you can indeed see a new folder named MyFlaskWebsite.

Create and verify the project directory

Step 5. Install Flask

We will use Flask to create a simple website. Flask is a Python based micro web framework. It uses Jinja (Python based template engine) as its template engine which makes it very usable and powerful. Use the following command to install flask on the Pi:

sudo apt-get install python3-flask

Install flask

Step 6. Write some basic code

Now that Flask is installed, we can start creating files and writing some code. First, navigate to your newly created project directory (from step 4) by using the following command:

cd MyFlaskWebsite

All project files and folders will reside inside this ‘MyFlaskWebsite’ directory. Now, create your first code file (app.py) using the following command:

touch app.py

On checking the directory using the ‘ls’ command, you should be able to see this newly created file.

Navigate to project directory and create a new file

Now, press F1 and choose ‘Remote Start Server.’ This should allow you to remotely edit files on the Pi using your laptop.

Start the remote server

Next, use the following command to start editing the newly created app.py file. It might take a few seconds but the empty file should then be visible in the window right above.

rmate app.py

Start editing the file remotely

Type out the code shown in the below picture. Here, we have defined a route to the home page of the website which should display ‘This is my flask website and it is so cool.’ Note that setting the host to 0.0.0.0 allows this website to be accessible by all devices connected to the same network.

Create a basic website

Save the file and use the following command to run the website on the Pi server:

python3 app.py

Run the website

On receiving the above success message, open a new browser window on any device within your network and type out the Pi’s IP address (in this case, it is 192.168.0.12) followed by the port the dev-server is running on (5000.) So the complete address will be http://192.168.0.12:5000/

You should see the text ‘This is my flask website and it is so cool.’ on the webpage.

Check the webpage in a browser

This confirms that your dev-server is active and is running the website you just created.

Step 7. Add more routes

Currently, the code consists of only 1 route which is the home page of the website. Add another route by typing out the following code. Note that you can dynamically make changes while the dev-server is running. It will automatically capture the delta (code change) and run a revised version once you refresh your browser window.

Add a meow route

To check whether or not the new route works as expected, go to http://192.168.0.12:5000/meow and the webpage should ‘MEOW’ at you.

Verify that the new route works as expected

Step 8. Add structure to your code

Now, adding more routes is cool but having the all the code in just 1 app.py file is not how a website should be structured. Usually, we would have a folder with HTML templates, a folder with static CSS files and another one for JS files. Let’s add these folders and move the code in appropriate folders to structure out code better. Use the following commands to create these directories:

mkdir templates

mkdir static

Use the ‘ls’ command to verify that these folders have been created.

Add structure to your code

Now, let’s create an HTML file for the homepage. Use the following commands to navigate to the templates directory. Then, create a new file named index.html and use rmate to edit the same:

cd templates

touch index.html

rmate index.html

Write some basic HTML code for the homepage inside index.html.

HTML code for the homepage

Make the following changes in app.py to use the index.html file. The below code will search for a file named index.html in the templates directory by default.

Use the new index.html file and render it using app.py

Navigate back to the project directory and run the website again.

Go back to the home page and you should see the content you put inside of index.html.

Now add some styling by creating ‘main.css’ inside the static directory. As always, use the ‘cd’ command to change the directory, ‘touch’ command to create a new file, and ‘rmate’ command to edit the same file.

Create the css file

Add some styling to the h4 tag. Note that we currently have 1 h4 tag in index.html which the css is supposed to modify.

Some css code

As always, test your changes by using the following command:

python3 app.py

Notice how the text within the h4 tag gets colored according to the CSS.

Step 9. Take advantage of Jinja

Jinja in a Python based template engine which adds a lot of powerful features to webpages. Although this tutorial is not focused on learning Jinja, let’s just look at a simple example of how Jinja can be useful.

Let’s just create a list of fruits in app.py and pass it as a parameter to index.html. We will then have index.html display that list on the webpage. Make the following changes in app.py and index.html.

_Pass the mylist as a parameter to index.html

_Display mylist on the webpage

Refresh your webpage and you should see the list of fruits on screen.

This speaks to how powerful and useful Jinja can be. For more info on Jinja, please refer to this.

Step 10. Next steps

Now that you have a fully functional Python dev-server, the possibilities going forward are practically infinite. Here are a few useful next steps that you may consider for your project:

1. Currently, the Pi is accessible only through the devices within your personal network. To expose the Pi to the outside world (access it through any device outside your personal network), you need something known as port forwarding. Basically, you need a domain name and a static IP address which is permanently assigned to the Pi. More info here and here.
2. Most applications will require a database for basic CRUD operations. Python supports SQlite right out of the box. Learn how to use SQlite with Flask here and here.
3. Here’s a cool Raspberry Pi starter kit on Amazon. The neat thing about this is that it has everything you need to get started and saves you the effort of searching individual items yourself.
4. Since you are not using a screen, it is important that you use the shutdown command for the Pi using the terminal. This ensures that the Pi and the SD card are not damaged:

sudo shutdown -h now

#UntilNextTime.