For example, say you need to install a Python library system wide. You could try to create a virtual environment on a shared well-known directory, or you could modify the environment variable PYTHONPATH to change where to look for packages.

But it may be simpler with an package manager like RedHat RPM or Debian DPKG, which can also help you keep track of dependencies and can even check if a package’s contents are tampered with after the installation with a checksum.

Also, system administration tools written in Python often require that you use an interpreter with all the required libraries ready to go. For example, imagine a system Python with the popular numpy module installed by default, and such package is used by the tool – just calling the import without initializing any virtual environments.

For the sake of argument, say you need to go the route of an RPM packaging. You’ll quickly realize that your RPM package has runtime dependencies (libraries than your Python library needs to run once installed) and build dependencies (libraries you need to build your library but that are not required to use the library).

In particular, build dependencies will force you to install those on the machines where you are packaging your application. For example, look at the “BuildRequires” tag from the poetry RPM spec from RedHat (showing a fragment here):

 This patch moves the vendored requires definition
# from vendors/pyproject.toml to pyproject.toml
# Intentionally contains the removed hunk to prevent patch aging
Patch1:         poetry-core-1.6.1-devendor.patch

BuildArch:      noarch
BuildRequires:  python3-devel
BuildRequires:  pyproject-rpm-macros

%if %{with tests}
# for tests (only specified via poetry poetry.dev-dependencies with pre-commit etc.)
BuildRequires:  python3-build
BuildRequires:  python3-pytest
BuildRequires:  python3-pytest-mock
BuildRequires:  python3-setuptools
BuildRequires:  python3-tomli-w
BuildRequires:  python3-virtualenv
BuildRequires:  gcc 
BuildRequires:  git-core
%endif

To complicate things further, you may:

• Need to build your library for a totally different OS that you have installed (say you have Fedora 42 but need and RPM for Alma Linux 9.5)

• Need to install an RPM that comes from a dubious source, and you want to make sure it doesn’t break your system while the packaging process is running (see the RPM scriptlets).

Prerequisites

In this tutorial, I’ll show you how you can handle those concerns using an Open Source tool called Mock. But first you will need the following to be able to follow this tutorial:

• A Linux distribution that uses RPM as packaging tool (RedHat Enterprise Edition, Fedora, Alma Linux, Rocky, and so on)

• Ability to install RPM packages on your build server (like mock, rpmdevtools) using tools like DNF or YUM.

• Understanding of how RPM packaging works (if you are unfamiliar, the Fedora RPM guide is a great starting point)

• You should understand what a container is and how PODMAN or Docker works.

• Understanding how a Python virtual environment works. We will not cover this here, but is useful to know that this alternative exists and how it works.

Here’s what we’ll cover:

• Why Mock?

• Packaging scenarios with Mock and Podman

• Conclusion

Why Mock?

As we discussed above, we already have Python virtual environments – so why bother to have an RPM of the same library?

Well, if you want to ensure consistent deployment across different systems, RPM packaging can be beneficial. It allows for easier management and distribution of software, especially in environments where system-wide installations are preferred over virtual environments.

Mock can help us with that. From the Mock Git README:

A 'simple' chroot build environment manager for building RPMs.

Mock is used by the Fedora Build system to populate a chroot environment, which is then used in building a source-RPM (SRPM). It can be used for long-term management of a chroot environment, but generally a chroot is populated (using DNF), an SRPM is built in the chroot to generate binary RPMs, and the chroot is then discarded.

This is very important: it means mock will install dependencies on a chroot environment, separated from the regular system, which will be discarded once the packaging is done.

Mock by itself doesn’t provide perfect isolation but when used with a container execution framework like PODMAN, it helps to protect the integrity of your system when packaging an unknown RPM:

Mock needs to execute some tasks under root privileges, therefore malicious RPMs can put your system at risk. Mock is not safe for unknown RPMs

By running mock inside Podman, you get the best of both worlds, as Podman will run with limited privileges by itself. Also Podman, being a container, can remove itself after execution, which helps out with the cleanup.

Let’s see a few scenarios that demonstrate where you can use mock.

Packaging Scenarios with Mock and Podman

Packaging a newer version of the module on an older Linux distribution

In this case, say we want to re-use the existing textual 0.6.2 package from Fedora 41 into Fedora 40. This is possible with mock, but to make it more secure we should run it inside a Podman container. This will give us more isolation from the real operating system.

During testing, I found than my home directory was tool small when running Podman. To fix this, I created a configuration override to point Podman root storage to a bigger partition on my machine (/mnt/data/podman/):

mkdir --parent ---verbose $HOME/.config/containers/
/bin/cat<<EOF>$HOME/.config/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/mnt/data/podman/"
graphroot = "/mnt/data/podman/"
EOF

Then I realized something else: I needed to preserve the results of our artifact generation. When you run a container with the —rm (remove) flag, all its contents are destroyed. In our case, we want to preserve the generated RPM package files. So what we do is to mount an external directory inside the Podman container using the —mount option: (--mount type=bind,src=$HOME/tmp,target=/mnt/result).

So far so good, right? Not quite. I found out that a Python dependency for Textual was missing too. It’s called Rich, and it needed an RPM as well. Luckily you can “chain” a list of dependencies as Source RPMS (SRPM) when building your main package, so Mock can make them available to you when preparing the main package (we must pass —localrepo instead of —resultdir and we use the --chain flag).

Now we are ready to build the package and its dependencies. This requires the following:

1. Create a local directory where the RPMS will be created

2. Run Podman on interactive mode so we can execute commands inside it

3. Install mock inside Podman using dnf.

4. Create a special user called mockbuilder to run mock and become that user

5. Execute mock passing the chain

mkdir --parent --verbose $HOME/tmp
podman run --mount type=bind,src=$HOME/tmp,target=/mnt/result --rm --privileged --interactive --tty fedora:40 bash
dnf install -y mock
useradd mockbuilder
usermod -a -G mock mockbuilder
chown mockbuilder /mnt/result/
su - mockbuilder
mock --localrepo /mnt/result/ --chain https://download.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/p/python-rich-13.7.1-5.fc41.src.rpm https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/source/tree/Packages/p/python-textual-0.62.0-2.fc41.src.rpm

For example, on my Raspberry PI 4 with Fedora 40, the final output looks like this:

...
INFO: Success building python-textual-0.62.0-2.fc41.src.rpm
INFO: Results out to: /mnt/result/results/default
INFO: Packages built: 2
INFO: Packages successfully built in this order:
INFO: /tmp/tmpc6651dxo/python-rich-13.7.1-5.fc41.src.rpm
INFO: /tmp/tmpc6651dxo/python-textual-0.62.0-2.fc41.src.rpm

Outside the container, we can test the installation by installing both Rich and Textual (you need root for this):

josevnz@raspberypi1:~$ sudo dnf install -y /home/josevnz/tmp/results/default/python-rich-13.7.1-5.fc41/python3-rich-13.7.1-5.fc40.noarch.rpm /home/josevnz/tmp/results/default/python-textual-0.62.0-2.fc41/python3-textual-doc-0.62.0-2.fc40.noarch.rpm /home/josevnz/tmp/results/default/python-textual-0.62.0-2.fc41/python3-textual-0.62.0-2.fc40.noarch.rpm
...
nstalled:
  python3-linkify-it-py-2.0.3-1.fc40.noarch            python3-markdown-it-py-3.0.0-4.fc40.noarch    python3-markdown-it-py+linkify-3.0.0-4.fc40.noarch  
  python3-markdown-it-py+plugins-3.0.0-4.fc40.noarch   python3-mdit-py-plugins-0.4.0-4.fc40.noarch   python3-mdurl-0.1.2-6.fc40.noarch                   
  python3-pygments-2.17.2-3.fc40.noarch                python3-rich-13.7.1-5.fc40.noarch             python3-textual-0.62.0-2.fc40.noarch                
  python3-textual-doc-0.62.0-2.fc40.noarch             python3-uc-micro-py-1.0.3-1.fc40.noarch      

Complete!

Note than the contents of the container were removed from the original window once you exit, except the mounted volume. This is great, as we don’t have to worry about uninstalling building packages ourselves.

But is it perfect?

Can you use Mock to package newer code on much older distributions?

Mock works really well as long your dependencies aren't too far away from the version you are running. For example, say you want to build the RPMS for Fedora 37 instead of Fedora 40:

sudo rm -rf $HOME/tmp/results/*
podman run --mount type=bind,src=$HOME/tmp,target=/mnt/result --rm --privileged --interactive --tty fedora:37 bash
dnf install -y mock
useradd mockbuilder && usermod -a -G mock mockbuilder && chown mockbuilder /mnt/result/ && su - mockbuilder
mock --nocheck --localrepo /mnt/result/ --chain https://download.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/p/python-rich-13.7.1-5.fc41.src.rpm https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/source/tree/Packages/p/python-textual-0.62.0-2.fc41.src.rpm
...
Package python3-poetry-core-1.0.8-3.fc37.noarch is already installed.
Package python3-pytest-7.1.3-2.fc37.noarch is already installed.
Package python3-setuptools-62.6.0-3.fc37.noarch is already installed.
Error: 
 Problem: nothing provides requested (python3dist(pygments) < 3~~ with python3dist(pygments) >= 2.13)

Uh oh, Fedora 37 doesn’t provide some of the dependencies. Can we build them in chain? I tried to add the SRPM for pygments (a generic syntax highlight library for Python), before building rich, as it is a dependency for it. So the dependency chain grew a little bit more:

mock --nocheck --localrepo /mnt/result/ --chain https://download.fedoraproject.org/pub/fedora/linux/releases/39/Everything/source/tree/Packages/p/python-pygments-2.15.1-4.fc39.src.rpm https://download.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/p/python-rich-13.7.1-5.fc41.src.rpm https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/source/tree/Packages/p/python-textual-0.62.0-2.fc41.src.rpm

And then I found that two more python dependencies were broken, this time for textual on Fedora 37:

...
no matching package to install: 'python3-syrupy'
No matching package to install: 'python3-time-machine'
Not all dependencies satisfied

Looks like a game of trial an error. How bad it can be?

Several tries later, I found that Syrupy (pytest plugin) added a dependency on Poetry (packaging tool), which complicated things a little bit, as Fedora 37 expects an older version of Poetry (poetry-1.1.14-1.fc37).

What could you do next? Well, you could try to get a version of Syrupy that works with this older version of Poetry. But that could potentially introduce vulnerabilities on your system or force you to use a version of Syrupy that doesn't work at all with Textual because of API changes.

It’s easier to work your dependencies upwards rather than downwards. In this case, I decided to stop my experiment as I don’t really need an RPM for Fedora 37 myself.

Building a newer non-packaged version of the software

Can mock help us with packaging an entirely new version of a package? Textual made huge improvements and added new features on the first official release 1.0.0. Let's see if we can take a few shortcuts to build an RPM that we can use with the system Python.

We will recycle the RPM Spec file from Textual we used before, but with a few modifications. First, let's prepare our sources again:

josevnz@raspberypi1:~$ podman run --mount type=bind,src=$HOME/tmp,target=/mnt/result --rm --privileged --interactive --tty fedora:40 bash
[root@ccae845daa84 /]# dnf install -y rpmdevtool
[root@ccae845daa84 /]# dnf install -y mock && useradd mockbuilder && usermod -a -G mock mockbuilder && chown mockbuilder /mnt/result/ && su - mockbuilder
[root@ccae845daa84 /]# for dep in https://download.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/p/python-rich-13.7.1-5.fc41.src.rpm https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/source/tree/Packages/p/python-textual-0.62.0-2.fc41.src.rpm; do rpm -ihv $dep; done

Then we update the RPM spec file for Textual, which describes how the RPM is created, bumping the version from 0.62.0 to 1.0.0.

What I like to do is to create a new SRPM for Textual. For that I do the following (I’m still inside the Podman container – yes you can reuse it as long it keeps running):

1. Install rpmdevtool, mock, as it contains a few tools I need to setup the environment to build the SRPM

2. Install the original SRPM for 0.6.2. Installing doesn’t need root and creates a new SRPM I can use to bootstrap my new installation. Steps 1 and 2 just below (this is optional if you are re-using the container from the previous example):

    [root@ccae845daa84 /]# dnf install -y rpmdevtool
    [root@ccae845daa84 /]# dnf install -y mock && useradd mockbuilder && usermod -a -G mock mockbuilder && chown mockbuilder /mnt/result/ && su - mockbuilder
    [root@ccae845daa84 /]# for dep in https://download.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/p/python-rich-13.7.1-5.fc41.src.rpm https://download.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/source/tree/Packages/p/python-textual-0.62.0-2.fc41.src.rpm; do rpm -ihv $dep; done
   

3. I bumped the version of the package from 0.6.2 on the SPEC file that gets extracted inside ~/rpmbuild/SPECS/python-textual.spec

4. Tell spectool to retrieve the proper compressed source tar file so we can used to prepare a new SRPM

5. Recreate the SRPM so it can be used by Mock.

   Steps 3, 4, and 5 below:

[root@ccae845daa84 /]# sed -i 's#0.62.0#1.0.0#' ~/rpmbuild/SPECS/python-textual.spec
[root@ccae845daa84 /]# sed -i 's#%{url}/archive/v%{version}/textual-%{version}.tar.gz#%{url}/archive/refs/tags/v%{version}.tar.gz#' ~/rpmbuild/SPECS/python-textual.spec
[root@ccae845daa84 /]# spectool --get-files ~/rpmbuild/SPECS/python-textual.spec --sourcedir
Downloading: https://github.com/Textualize/textual/archive/refs/tags/v1.0.0.tar.gz
|  28.3 MiB Elapsed Time: 0:00:02                                                                                                                       
Downloaded: v1.0.0.tar.gz
[root@ccae845daa84 /]# rpmbuild -bs ~/rpmbuild/SPECS/python-textual.spec
setting SOURCE_DATE_EPOCH=1717891200
Wrote: /root/rpmbuild/SRPMS/python-textual-1.0.0-2.fc40.src.rpm

Now we can rebuild the SRPM and make make sure mock can find it when running from the exposed volume:

[root@ccae845daa84 /]# cp -pv /root/rpmbuild/SRPMS/python-textual-1.0.0-2.fc40.src.rpm /tmp/
'/root/rpmbuild/SRPMS/python-textual-1.0.0-2.fc40.src.rpm' -> '/tmp/python-textual-1.0.0-2.fc40.src.rpm'
[root@ccae845daa84 /]# su - mockbuilder
[mockbuilder@ccae845daa84 ~]$ ls -l /tmp/python-textual-1.0.0-2.fc40.src.rpm
-rw-r--r--. 1 root root 29612335 Jan 11 00:12 /tmp/python-textual-1.0.0-2.fc40.src.rpm

Moment of truth, let’s build it:

[mockbuilder@ccae845daa84 ~]$ mock --nocheck --localrepo /mnt/result/ --chain https://download.fedoraproject.org/pub/fedora/linux/releases/41/Everything/source/tree/Packages/p/python-rich-13.7.1-5.fc41.src.rpm /tmp/python-textual-1.0.0-2.fc40.src.rpm
Wrote: /builddir/build/SRPMS/python-textual-1.0.0-2.fc40.src.rpm
Wrote: /builddir/build/RPMS/python3-textual-1.0.0-2.fc40.noarch.rpm
Wrote: /builddir/build/RPMS/python3-textual-doc-1.0.0-2.fc40.noarch.rpm
INFO: Done(/tmp/python-textual-1.0.0-2.fc40.src.rpm) Config(default) 2 minutes 38 seconds

Finally, test the installation by installing the RPMS outside the container:

josevnz@raspberypi1:~$ sudo dnf install /home/josevnz/tmp/results/default/python-rich-13.7.1-5.fc41/python3-rich-13.7.1-5.fc40.noarch.rpm /home/josevnz/tmp/results/default/python-textual-1.0.0-2.fc40/python3-textual-doc-1.0.0-2.fc40.noarch.rpm /home/josevnz/tmp/results/default/python-textual-1.0.0-2.fc40/python3-textual-1.0.0-2.fc40.noarch.rpm
Last metadata expiration check: 3:42:37 ago on Fri 10 Jan 2025 03:50:49 PM EST.
Package python3-rich-13.7.1-5.fc40.noarch is already installed.
Dependencies resolved.
=========================================================================================================================================================
 Package                                    Architecture                 Version                                Repository                          Size
=========================================================================================================================================================
Upgrading:
 python3-textual                            noarch                       1.0.0-2.fc40                           @commandline                       1.3 M
 python3-textual-doc                        noarch                       1.0.0-2.fc40                           @commandline                        24 M
Installing dependencies:
 python3-platformdirs                       noarch                       3.11.0-3.fc40                          fedora                              46 k

Transaction Summary
=========================================================================================================================================================
Install  1 Package
Upgrade  2 Packages

Total size: 25 M
Total download size: 46 k
Is this ok [y/N]: y
Downloading Packages:
python3-platformdirs-3.11.0-3.fc40.noarch.rpm                                                                             53 kB/s |  46 kB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                     41 kB/s |  46 kB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                 1/1 
  Installing       : python3-platformdirs-3.11.0-3.fc40.noarch                                                                                       1/5 
  Upgrading        : python3-textual-1.0.0-2.fc40.noarch                                                                                             2/5 
  Upgrading        : python3-textual-doc-1.0.0-2.fc40.noarch                                                                                         3/5 
  Cleanup          : python3-textual-0.62.0-2.fc40.noarch                                                                                            4/5 
  Cleanup          : python3-textual-doc-0.62.0-2.fc40.noarch                                                                                        5/5 
  Running scriptlet: python3-textual-doc-0.62.0-2.fc40.noarch                                                                                        5/5 

Upgraded:
  python3-textual-1.0.0-2.fc40.noarch                                       python3-textual-doc-1.0.0-2.fc40.noarch                                      
Installed:
  python3-platformdirs-3.11.0-3.fc40.noarch                                                                                                              

Complete!

Not bad, we can now build sophisticated TUIs using Textual and the system Python, without the need to create a virtual environment nor force the installation of unwanted packages in our build server.

Conclusion

As you can see, mock is a very valuable tool that can help you automate packaging Python libraries that are not yet available in your platform. It allows you to automate getting dependencies for the RPM and alerts you when some are missing in your platform.

As an added bonus, the fact than you can run it inside Podman gives you even more isolation from RPMs that could be dangerous when executed as root.

Extra documentation (RTFM, Read The Fine Manual)

• RPM-Macros

• Mock

• RPM dev tools

• RPM macro documentation

• Packaging Python3 RPMS

• PyPA specifications

• Fedora Textual RPM

The Empire State Building Run-Up (ESBRU)—the world’s first and most famous tower race—challenges runners from near and far to race up its famed 86 flights—1,576 stairs.

While visitors can reach the building’s Observatory via elevator in under one minute, the fastest runners have covered the 86 floors by foot in about 10 minutes.

Leaders in the sport of professional tower-running converge at the Empire State Building in what some consider the ultimate test of endurance.

I got lucky and managed to participate in this race. A few days after finishing the race, I realized that I wanted to know more about my performance, and what I could have done to better.

So naturally I went to the race organizer website and started looking at the numbers. And it was slow and tedious, plus it brought up more issues:

1. Getting the data for offline analysis is difficult. You can see your results and others for comparison, but I found that the tools didn't offer an option to download the raw data, and they were clumsy to use.

2. Most tools out there to analyze race results are paid or do not apply to this type of race. Knowing what to expect reduces your anxiety, allows you to train better, and keeps your expectations in check.

By now you've probably guessed that you can solve the data retrieval issues and post-race analysis using low-cost Open Source tools. This also allows you to apply different techniques to learn about the race and, depending on the quality of the data, even make performance predictions.

This is a very personal piece for me. I will share my race results and give you my biased opinion about the race. 😁

Table of Contents

1. How I Ended Up Running to the Top of the Empire State Building

2. What You Need to Follow this Tutorial

3. How to Get the Data using Web Scraping

4. How to Clean Up the Data

5. How to Analyze the Data

6. How to Visualize the results

7. How to Run the Applications

8. What Else Can We Learn?

How I Ended Up Running to the Top of the Empire State Building

Many of us have run a regular race at some point in our lives – there are many distances like 5K, 10K, Half Marathon, and Full Marathon. But there is no way to compare how you will perform while running the stairs all the way to the top of one of the most famous buildings in the world.

If you have ever been at the base of the skyscrapers in New York City and have looked up, you get the idea. Picture yourself running up the stairs, all the way to the top, without stopping.

Getting accepted is tough, because unlike a race like the New York Marathon, the Empire State Building can only accommodate around 500 runners (or should I say climbers?).

Add to that fact that the demand to participate is high, and then you can see that your chances to get in through the lottery are pretty slim (I read somewhere that there are only 50 lottery positions for more than 5,000 applicants).

You can imagine my surprise when I got an email saying that I was selected to participate after trying for 4 years in a row.

I panicked. Have you ever been at the base of the Empire State and looked up? Some days when it's cloudy you can't even see the top of the building.

I wasn't unprepared. But I had to adjust my training routine to be ready for this challenge with a small window of two months, and no experience doing a tower run.

The day of the race came and this is how it went for me:

• It was tough. I knew I had to pace myself, otherwise, the race would have ended for me on floor 20th as opposed to the 86th. You have to focus on a "keep going" mentality, regardless of how tired you feel. And then it is over, just like that.

• You don't sprint, you climb 2 steps at a time at a steady pace, and you use the handrails to take weight off your legs.

• No need to carb load or hydrate too much. If you do well, you will be done in around 30 minutes.

• Nobody is pushing anyone. At least for non-elite racers like me, I was alone for most of the race.

• I got passed and I passed a lot of people who forgot the 'pace yourself' rule. If you sprint, you will be toasted before floor 25, for sure.

I had a blast and got great satisfaction from having this race ticked off my bucket list, the same way I felt after running the NYC Marathon.

It was time now to do a post-race analysis using several of my favorite Open Source tools, which I'll explain in the next section.

What You Need to Follow this Tutorial

Like the race, most of the challenges to writing this application were mental. You only need to break the main problem down into smaller pieces and then tackle each piece at a time:

1. Get the data by scraping the website (very few sites allow you to export race results as a CSV).

2. Clean up the data, normalize it, and make it ready for automatic processing.

3. Ask questions. Then translate those questions into code and tests, ideally using statistics to get reliable answers.

4. Present the results. A UI (Text or Graphic) will do wonders due to its low consumption, but charts speak volumes too.

You should have some experience in a programming language to get the most out of this article. My code is written in Python (you will need version 3.8+) and runs on Linux (I used Fedora 37 distribution).

In a nutshell, I want to show that it is possible to do all the above with Open Source technologies. Then you can reuse this knowledge for other projects, not just for tower race analyses. 😅

I strongly recommend that you get the source code (It is Open Source!). Get your hands dirty, break the scripts, and have fun. You will need Git to clone the repository:

git clone https://github.com/josevnz/tutorials.git
cd tutorials/docs/EmpireStateRunUp/
python -m ~/virtualenv/EmpireStateRunUp
. ~/virtualenv/EmpireStateRunUp/bin/activate
pip install --upgrade pip
pip install --upgrade build
pip install --upgrade wheel
pip install --editable .

Or if you just want to run the code while reading this tutorial (using my latest version from Pypi):

python -m ~/virtualenv/EmpireStateRunUp
. ~/virtualenv/EmpireStateRunUp/bin/activate 
pip install --upgrade EmpireStateRunUp

We can now move to the next stage:a getting the data.

How to Get the Data using Web Scraping

The race results site doesn't have an export feature, and I never heard back from their support team to see if there was an alternate way to get the race data. So the only alternative left was to do some web scraping.

The website is pretty basic and only allows scrolling through each record, so I decided to do web scraping to get the results into a format I could use later for data analysis.

The rules of web scraping

There are very 3 simple rules:

1. Rule #1: Don't do it. Data flow changes, and your scraper will break the minute you are done getting the data. It will require time and effort. Lots of it.

2. Rule #2: Re-read rule number 1. If you can't get the data in any another format, then go to rule #3

3. Rule #3: Choose a good framework to automate what you can and prepare to do heavy data cleanup (also known as "give me patience for the stuff I can't control, like poorly done HTML and CSS").

I decided to use Selenium Web Driver as it calls a real browser, like Firefox, to navigate the website. Selenium allows you to automate browser actions while you get the same rendered HTML you see when you navigate the site.

Selenium is a complex tool and will require you to spend some time experimenting with what works and what does not. Below is a simple script I wrote to get all the runner's names and race detail links in one run:

import re
from time import sleep

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.webdriver import WebDriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions
# AthLinks is nice enough to post the race results and their interface is very human-friendly. Not so machine parsing friendly.
RESULTS = "https://www.athlinks.com/event/382111/results/Event/1062909/Course/2407855/Results"
LINKS = {}


def print_links(web_driver: WebDriver, page: int) -> None:
    for a in web_driver.find_elements(By.TAG_NAME, "a"):
        href = a.get_attribute('href')
        if re.search('Bib', href):
            name = a.text.strip().title()
            print(f"Page={page}, {name}={href.strip()}")
            LINKS[name] = href.strip()


def click(level: int) -> None:
    button = WebDriverWait(driver, 20).until(
        expected_conditions.element_to_be_clickable((By.CSS_SELECTOR, f"div:nth-child({level}) > button")))
    driver.execute_script("arguments[0].click();", button)
    sleep(2.5)


options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.get(RESULTS)
sleep(2.5)
print_links(driver, 1)
click(6)
print_links(driver, 2)
click(7)
print_links(driver, 3)
click(7)
print_links(driver, 4)
click(9)
print_links(driver, 5)
click(9)
print_links(driver, 6)
click(7)
print_links(driver, 7)
click(7)
print_links(driver, 8)
print(len(LINKS))

The code above is hardly reusable, but it gets the job done by doing the following:

1. Gets the main web-page with the driver.get(...) method

2. Then gets the <a href tags, and sleeps a little to get a chance to render the HTML

3. Then finds and clicks the > (next page) button

4. Does these steps a total of 8 times, as this is how many pages of results are available (each page has 50 runners)

To get the full race results I wrote scraper.py code. The code deals with navigating multiple pages and extracting the data. Demonstration below:

(EmpireStateRunUp) [josevnz@dmaf5 EmpireStateRunUp]$ esru_scraper /home/josevnz/temp/raw_data.csv
2023-12-30 14:05:00,987 Saving results to /home/josevnz/temp/raw_data.csv
2023-12-30 14:05:53,091 Got 377 racer results
2023-12-30 14:05:53,091 Processing BIB: 19, will fetch: https://www.athlinks.com/event/382111/results/Event/1062909/Course/2407855/Bib/19
2023-12-30 14:06:02,207 Wrote: name=Wai Ching Soh, position=1, {'name': 'Wai Ching Soh', 'url': 'https://www.athlinks.com/event/382111/results/Event/1062909/Course/2407855/Bib/19', 'overall position': '1', 'gender': 'M', 'age': 29, 'city': 'Kuala Lumpur', 'state': '-', 'country': 'MYS', 'bib': 19, '20th floor position': '1', '20th floor gender position': '1', '20th floor division position': '1', '20th floor pace': '42:30', '20th floor time': '1:42', '65th floor position': '1', '65th floor gender position': '1', '65th floor division position': '1', '65th floor pace': '54:03', '65th floor time': '7:34', 'gender position': '1', 'division position': '1', 'pace': '53:00', 'time': '10:36', 'level': 'Full Course'}
...

It does just minimal manipulation of the data from the web page. The purpose of this code is just to get the data as quickly as possible before the formatting changes.

Data cannot be used yet as-is – it needs cleaning up. And that's the next step in this article.

How to Clean Up the Data

Getting the data is just the first battle of many more to come. You will notice inconsistencies on the data and missing values. In order to make your numeric results good, you need to make assumptions.

Luckily for me, the dataset is very small (375+ records, one for each runner) so I was able to come up with a few rules to tidy up the data file I was going to use during my analysis.

I also supplemented my data with another data set that has the 3-digit country codes as well as other details, for a nicer presentation.

The data_normalizer.raw_read(raw_file: Path) -> Iterable[Dict[str, Any]] method does the heavy work of fixing the data for inconsistencies before saving into a CSV format.

There are no hard rules here, as cleanup has a high correlation with the data set. For example, to figure out to which wave each runner was assigned I had to make some assumptions based on what I saw the day of the race.

Let me show you what I mean with some code:

import datetime
from enum import Enum
from typing import Dict

"""
Runners started on waves, but for basic analysis, we will assume all runners were able to run
at the same time.
"""
BASE_RACE_DATETIME = datetime.datetime(
    year=2023,
    month=9,
    day=4,
    hour=20,
    minute=0,
    second=0,
    microsecond=0
)

class Waves(Enum):
    """
    22 Elite male
    17 Elite female
    There are some holes, so either some runners did not show up or there was spare capacity.
    https://runsignup.com/Race/EmpireStateBuildingRunUp/Page-4
    https://runsignup.com/Race/EmpireStateBuildingRunUp/Page-5
    I guessed who went into which category, based on the BIB numbers I saw that day
    """
    ELITE_MEN = ["Elite Men", [1, 25], BASE_RACE_DATETIME]
    ELITE_WOMEN = ["Elite Women", [26, 49], BASE_RACE_DATETIME + datetime.timedelta(minutes=2)]
    PURPLE = ["Specialty", [100, 199], BASE_RACE_DATETIME + datetime.timedelta(minutes=10)]
    GREEN = ["Sponsors", [200, 299], BASE_RACE_DATETIME + datetime.timedelta(minutes=20)]
    """
    The date people applied for the lottery determined the colors. Let's assume that
    General Lottery Open: 7/17 9AM- 7/28 11:59PM
    General Lottery Draw Date: 8/1
    """
    ORANGE = ["Tenants", [300, 399], BASE_RACE_DATETIME + datetime.timedelta(minutes=30)]
    GREY = ["General 1", [400, 499], BASE_RACE_DATETIME + datetime.timedelta(minutes=40)]
    GOLD = ["General 2", [500, 599], BASE_RACE_DATETIME + datetime.timedelta(minutes=50)]
    BLACK = ["General 3", [600, 699], BASE_RACE_DATETIME + datetime.timedelta(minutes=60)]

"""
Interested only in people who completed the 86 floors. So is it either a full course or dnf
"""
class Level(Enum):
    FULL = "Full Course"
    DNF = "DNF"

# Fields are sorted by interest
class RaceFields(Enum):
    BIB = "bib"
    NAME = "name"
    OVERALL_POSITION = "overall position"
    TIME = "time"
    GENDER = "gender"
    GENDER_POSITION = "gender position"
    AGE = "age"
    DIVISION_POSITION = "division position"
    COUNTRY = "country"
    STATE = "state"
    CITY = "city"
    PACE = "pace"
    TWENTY_FLOOR_POSITION = "20th floor position"
    TWENTY_FLOOR_GENDER_POSITION = "20th floor gender position"
    TWENTY_FLOOR_DIVISION_POSITION = "20th floor division position"
    TWENTY_FLOOR_PACE = '20th floor pace'
    TWENTY_FLOOR_TIME = '20th floor time'
    SIXTY_FLOOR_POSITION = "65th floor position"
    SIXTY_FIVE_FLOOR_GENDER_POSITION = "65th floor gender position"
    SIXTY_FIVE_FLOOR_DIVISION_POSITION = "65th floor division position"
    SIXTY_FIVE_FLOOR_PACE = '65th floor pace'
    SIXTY_FIVE_FLOOR_TIME = '65th floor time'
    WAVE = "wave"
    LEVEL = "level"
    URL = "url"

FIELD_NAMES = [x.value for x in RaceFields if x != RaceFields.URL]
FIELD_NAMES_FOR_SCRAPING = [x.value for x in RaceFields]
FIELD_NAMES_AND_POS: Dict[RaceFields, int] = {}
pos = 0
for field in RaceFields:
    FIELD_NAMES_AND_POS[field] = pos
    pos += 1

def get_wave_from_bib(bib: int) -> Waves:
    for wave in Waves:
        (lower, upper) = wave.value[1]
        if lower <= bib <= upper:
            return wave
    return Waves.BLACK

def get_description_for_wave(wave: Waves) -> str:
    return wave.value[0]

I used enums to make it clear what type of data I was working on, especially for the names of the fields. Consistency is key.

As for cleaning the data, well there were some obvious fixes I had to apply like:

1. Format of the times like pace, race time, and so on so it could be parsed later

2. Capitalize some values to make them easier to read

3. Early string to integer conversion for values like age, position, and so on. If that fails, assign 'not a number'.

By all means, we are not done massaging the data. A simple function takes care of this stage inside the data module:

# Omitted imports and Enum declarations as they were shown early on. 
# Check the source code for 'data.py' for more details
def raw_csv_read(raw_file: Path) -> Iterable[Dict[str, Any]]:
    record = {}
    with open(raw_file, 'r') as raw_csv_file:
        reader = csv.DictReader(raw_csv_file)
        row: Dict[str, Any]
        for row in reader:
            try:
                csv_field: str
                for csv_field in FIELD_NAMES_FOR_SCRAPING:
                    column_val = row[csv_field].strip()
                    if csv_field == RaceFields.BIB.value:
                        bib = int(column_val)
                        record[csv_field] = bib
                    elif csv_field in [ RaceFields.GENDER_POSITION.value, RaceFields.DIVISION_POSITION.value, RaceFields.OVERALL_POSITION.value,  RaceFields.TWENTY_FLOOR_POSITION.value,
                        RaceFields.TWENTY_FLOOR_DIVISION_POSITION.value, RaceFields.TWENTY_FLOOR_GENDER_POSITION.value, RaceFields.SIXTY_FLOOR_POSITION.value, RaceFields.SIXTY_FIVE_FLOOR_DIVISION_POSITION.value,
                        RaceFields.SIXTY_FIVE_FLOOR_GENDER_POSITION.value, RaceFields.AGE.value ]:
                        try:
                            record[csv_field] = int(column_val)
                        except ValueError:
                            record[csv_field] = math.nan
                    elif csv_field == RaceFields.WAVE.value:
                        record[csv_field] = get_description_for_wave(get_wave_from_bib(bib)).upper()
                    elif csv_field in [RaceFields.GENDER.value, RaceFields.COUNTRY.value]:
                        record[csv_field] = column_val.upper()
                    elif csv_field in [RaceFields.CITY.value, RaceFields.STATE.value,

                    ]:
                        record[csv_field] = column_val.capitalize()
                    elif csv_field in [RaceFields.SIXTY_FIVE_FLOOR_PACE.value, RaceFields.SIXTY_FIVE_FLOOR_TIME.value, RaceFields.TWENTY_FLOOR_PACE.value,
                        RaceFields.TWENTY_FLOOR_TIME.value, RaceFields.PACE.value, RaceFields.TIME.value ]:
                        parts = column_val.strip().split(':')
                        for idx in range(0, len(parts)):
                            if len(parts[idx]) == 1:
                                parts[idx] = f"0{parts[idx]}"
                        if len(parts) == 2:
                            parts.insert(0, "00")
                        record[csv_field] = ":".join(parts)
                    else:
                        record[csv_field] = column_val
                if record[csv_field] in ['-', '--']:
                    record[csv_field] = ""
                yield record
            except IndexError:
                raise

The esru_csv_cleaner script is the sum of the first stage cleanup effort, which takes the raw captured data and writes a CSV file with some important corrections:

esru_csv_cleaner --rawfile /home/josevnz/temp/raw_data.csv /home/josevnz/tutorials/docs/EmpireStateRunUp/empirestaterunup/results-full-level-2023.csv

Now with the data ready, we can proceed to load the data and ask some questions about the race.

How to Analyze the Data

Once the data is clean (or as clean as we can get it), it's time to move into running some numbers. Before writing more code, I took a piece of paper and asked myself a few questions about the race:

• There are any interesting buckets/ clusters for age, race time, wave, and country participation?

• A histogram for Age and Country would be nice to see

• Describe the data! (median, percentiles, and so on)

• Find outliers. There is a way to apply Z-scores here?

I decided to use Python Pandas for this task. This Open Source framework has an arsenal of tools to manipulate the data and to calculate statistics. It also has good tools to perform additional cleanup if needed.

So how does Pandas work?

Crash Course on Pandas

I strongly recommend that you check out 10 minutes to pandas if you are not familiar with the tool. For my DataFrame, I made the BIB an index as it is unique, and it has no special value for aggregation functions – but the 'id' attribute is unique.

It's important to note that also at this stage I needed to normalize the data, which I'll explain shortly:

# Omitted imports and Enum declarations as they were shown early on. 
# Check the source code for 'data.py' for more details
def load_data(data_file: Path = None, remove_dnf: bool = True) -> DataFrame:
    """
    * The code removes by default the DNF runners to avoid distortion on the results.
    * Replace unknown/ nan values with the median, to make analysis easier and avoid distortions
    """
    if data_file:
        def_file = data_file
    else:
        def_file = RACE_RESULTS_FULL_LEVEL
    df = pandas.read_csv(
        def_file
    )
    for time_field in [
        RaceFields.PACE.value,
        RaceFields.TIME.value,
        RaceFields.TWENTY_FLOOR_PACE.value,
        RaceFields.TWENTY_FLOOR_TIME.value,
        RaceFields.SIXTY_FIVE_FLOOR_PACE.value,
        RaceFields.SIXTY_FIVE_FLOOR_TIME.value
    ]:
        try:
            df[time_field] = pandas.to_timedelta(df[time_field])
        except ValueError as ve:
            raise ValueError(f'{time_field}={df[time_field]}', ve)
    df['finishtimestamp'] = BASE_RACE_DATETIME + df[RaceFields.TIME.value]
    if remove_dnf:
        df.drop(df[df.level == 'DNF'].index, inplace=True)

    # Normalize Age
    median_age = df[RaceFields.AGE.value].median()
    df[RaceFields.AGE.value].fillna(median_age, inplace=True)
    df[RaceFields.AGE.value] = df[RaceFields.AGE.value].astype(int)

    # Normalize state and city
    df.replace({RaceFields.STATE.value: {'-': ''}}, inplace=True)
    df[RaceFields.STATE.value].fillna('', inplace=True)
    df[RaceFields.CITY.value].fillna('', inplace=True)

    # Normalize overall position, 3 levels
    median_pos = df[RaceFields.OVERALL_POSITION.value].median()
    df[RaceFields.OVERALL_POSITION.value].fillna(median_pos, inplace=True)
    df[RaceFields.OVERALL_POSITION.value] = df[RaceFields.OVERALL_POSITION.value].astype(int)
    median_pos = df[RaceFields.TWENTY_FLOOR_POSITION.value].median()
    df[RaceFields.TWENTY_FLOOR_POSITION.value].fillna(median_pos, inplace=True)
    df[RaceFields.TWENTY_FLOOR_POSITION.value] = df[RaceFields.TWENTY_FLOOR_POSITION.value].astype(int)
    median_pos = df[RaceFields.SIXTY_FLOOR_POSITION.value].median()
    df[RaceFields.SIXTY_FLOOR_POSITION.value].fillna(median_pos, inplace=True)
    df[RaceFields.SIXTY_FLOOR_POSITION.value] = df[RaceFields.SIXTY_FLOOR_POSITION.value].astype(int)

    # Normalize gender position, 3 levels
    median_gender_pos = df[RaceFields.GENDER_POSITION.value].median()
    df[RaceFields.GENDER_POSITION.value].fillna(median_gender_pos, inplace=True)
    df[RaceFields.GENDER_POSITION.value] = df[RaceFields.GENDER_POSITION.value].astype(int)
    median_gender_pos = df[RaceFields.TWENTY_FLOOR_GENDER_POSITION.value].median()
    df[RaceFields.TWENTY_FLOOR_GENDER_POSITION.value].fillna(median_gender_pos, inplace=True)
    df[RaceFields.TWENTY_FLOOR_GENDER_POSITION.value] = df[RaceFields.TWENTY_FLOOR_GENDER_POSITION.value].astype(int)
    median_gender_pos = df[RaceFields.SIXTY_FIVE_FLOOR_GENDER_POSITION.value].median()
    df[RaceFields.SIXTY_FIVE_FLOOR_GENDER_POSITION.value].fillna(median_gender_pos, inplace=True)
    df[RaceFields.SIXTY_FIVE_FLOOR_GENDER_POSITION.value] = df[
        RaceFields.SIXTY_FIVE_FLOOR_GENDER_POSITION.value].astype(int)

    # Normalize age/ division position, 3 levels
    median_div_pos = df[RaceFields.DIVISION_POSITION.value].median()
    df[RaceFields.DIVISION_POSITION.value].fillna(median_div_pos, inplace=True)
    df[RaceFields.DIVISION_POSITION.value] = df[RaceFields.DIVISION_POSITION.value].astype(int)
    median_div_pos = df[RaceFields.TWENTY_FLOOR_DIVISION_POSITION.value].median()
    df[RaceFields.TWENTY_FLOOR_DIVISION_POSITION.value].fillna(median_div_pos, inplace=True)
    df[RaceFields.TWENTY_FLOOR_DIVISION_POSITION.value] = df[RaceFields.TWENTY_FLOOR_DIVISION_POSITION.value].astype(int)
    median_div_pos = df[RaceFields.SIXTY_FIVE_FLOOR_DIVISION_POSITION.value].median()
    df[RaceFields.SIXTY_FIVE_FLOOR_DIVISION_POSITION.value].fillna(median_div_pos, inplace=True)
    df[RaceFields.SIXTY_FIVE_FLOOR_DIVISION_POSITION.value] = df[
        RaceFields.SIXTY_FIVE_FLOOR_DIVISION_POSITION.value].astype(int)

    # Normalize 65th floor pace and time
    sixty_five_floor_pace_median = df[RaceFields.SIXTY_FIVE_FLOOR_PACE.value].median()
    sixty_five_floor_time_median = df[RaceFields.SIXTY_FIVE_FLOOR_TIME.value].median()
    df[RaceFields.SIXTY_FIVE_FLOOR_PACE.value].fillna(sixty_five_floor_pace_median, inplace=True)
    df[RaceFields.SIXTY_FIVE_FLOOR_TIME.value].fillna(sixty_five_floor_time_median, inplace=True)

    # Normalize BIB and make it the index
    df[RaceFields.BIB.value] = df[RaceFields.BIB.value].astype(int)
    df.set_index(RaceFields.BIB.value, inplace=True)

    # URL was useful during scraping, not needed for analysis
    df.drop([RaceFields.URL.value], axis=1, inplace=True)

    return df

I do a few things here after giving back the converted CSV back to the user, as a DataFrame:

• Replaced "Not a Number" (nan) values with the median to avoid affecting the aggregation results. This makes analysis easier.

• Dropped rows for runners that did not reach floor 86. Makes the analysis easier, and there are too few of them.

• Convert some string columns into native data types like integers, timestamps

• A few entries did not have the gender defined. That affected other fields like 'gender_position'. To avoid distortions, these were filled with the median.

In the end, this is how my DataFrame loading looked like:

(EmpireStateRunUp) [josevnz@dmaf5 EmpireStateRunUp]$ python3
Python 3.11.6 (main, Oct  3 2023, 00:00:00) [GCC 12.3.1 20230508 (Red Hat 12.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

And the resulting DataFrame instance:

>>> # Using custom load_data function that returns a Panda DataFrame
>>> from empirestaterunup.data import load_data
>>> load_data('empirestaterunup/results-full-level-2023.csv')
                    name  overall position            time gender  gender position  age  ...  65th floor division position 65th floor pace 65th floor time       wave        level     finishtimestamp
bib                                                                                      ...                                                                                                          
19         Wai Ching Soh                 1 0 days 00:10:36      M                1   29  ...                             1 0 days 00:54:03 0 days 00:07:34  ELITE MEN  Full Course 2023-09-04 20:10:36
22        Ryoji Watanabe                 2 0 days 00:10:52      M                2   40  ...                             1 0 days 00:54:31 0 days 00:07:38  ELITE MEN  Full Course 2023-09-04 20:10:52
16            Fabio Ruga                 3 0 days 00:11:14      M                3   42  ...                             2 0 days 00:57:09 0 days 00:08:00  ELITE MEN  Full Course 2023-09-04 20:11:14
11        Emanuele Manzi                 4 0 days 00:11:28      M                4   45  ...                             3 0 days 00:59:17 0 days 00:08:18  ELITE MEN  Full Course 2023-09-04 20:11:28
249             Alex Cyr                 5 0 days 00:11:52      M                5   28  ...                             2 0 days 01:01:19 0 days 00:08:35   SPONSORS  Full Course 2023-09-04 20:11:52
..                   ...               ...             ...    ...              ...  ...  ...                           ...             ...             ...        ...          ...                 ...
555     Caroline Edwards               372 0 days 00:55:17      F              143   47  ...                            39 0 days 04:57:23 0 days 00:41:38  GENERAL 2  Full Course 2023-09-04 20:55:17
557        Sarah Preston               373 0 days 00:55:22      F              144   34  ...                            41 0 days 04:58:20 0 days 00:41:46  GENERAL 2  Full Course 2023-09-04 20:55:22
544  Christopher Winkler               374 0 days 01:00:10      M              228   40  ...                            18 0 days 01:49:53 0 days 00:15:23  GENERAL 2  Full Course 2023-09-04 21:00:10
545          Jay Winkler               375 0 days 01:05:19      U               93   33  ...                            18 0 days 05:28:56 0 days 00:46:03  GENERAL 2  Full Course 2023-09-04 21:05:19
646           Dana Zajko               376 0 days 01:06:48      F              145   38  ...                            42 0 days 05:15:14 0 days 00:44:08  GENERAL 3  Full Course 2023-09-04 21:06:48

[375 rows x 24 columns]

Once the data was loaded, I was able to start asking questions. For example, to detect the outliers I used a Z-score.

All the analysis logic was kept together on a single module called 'analyze', separate from presentation, data loading, or reports, to promote reuse.

from pandas import DataFrame
import numpy as np
def get_zscore(df: DataFrame, column: str):
    filtered = df[column]
    return filtered.sub(filtered.mean()).div(filtered.std(ddof=0))

def get_outliers(df: DataFrame, column: str, std_threshold: int = 3) -> DataFrame:
    """
    Use the z-score, anything further away than 3 standard deviations is considered an outlier.
    """
    filtered_df = df[column]
    z_scores = get_zscore(df=df, column=column)
    is_over = np.abs(z_scores) > std_threshold
    return filtered_df[is_over]

Also, it is very simple to get common statistics just by calling describe on our data:

from pandas import DataFrame
def get_5_number(criteria: str, data: DataFrame) -> DataFrame:
    return data[criteria].describe()

For example, let me show you summary metrics for different aspects of the race:

>>> from empirestaterunup.data import load_data
>>> df = load_data('empirestaterunup/results-full-level-2023.csv')
>>> from empirestaterunup.analyze import get_5_number
>>> from empirestaterunup.analyze import SUMMARY_METRICS
>>> print(SUMMARY_METRICS)
('age', 'time', 'pace')
>>> for key in SUMMARY_METRICS:
...     ndf = get_5_number(criteria=key, data=df)
...     print(ndf)
... 
count    375.000000
mean      41.309333
std       11.735968
min       11.000000
25%       33.000000
50%       40.000000
75%       49.000000
max       78.000000
Name: age, dtype: float64
count                          375
mean     0 days 00:23:03.461333333
std      0 days 00:08:06.313479117
min                0 days 00:10:36
25%                0 days 00:18:09
50%                0 days 00:21:20
75%         0 days 00:25:13.500000
max                0 days 01:06:48
Name: time, dtype: object
count                          375
mean     0 days 01:55:17.306666666
std      0 days 00:40:31.567395588
min                0 days 00:53:00
25%                0 days 01:30:45
50%                0 days 01:46:40
75%         0 days 02:06:07.500000
max                0 days 05:34:00
Name: pace, dtype: object

Making sure data web scraping, data loading, and analytics work well is a must. Testing is an integral part of writing code, so I kept adding more of it and went back to writing unit tests.

Let's check how to test our code (feel free to skip the next section if you are familiar with unit testing)

Testing, testing, and after that...more testing

I assume you are familiar with writing small, self-contained pieces of code to test your code. These are called unit tests.

The unittest unit testing framework was originally inspired by JUnit and has a similar flavor as major unit testing frameworks in other languages. It supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework. (From the Python docs)

I tried to have a simple unit test for every method I wrote on the code. This saved me lots of headaches down the road. As I refactored the code, I found better ways to get the same results, producing correct numbers.

A Unit test in this context is a class that extends unittest.TestCase. Each method that starts with test_ is a test that must pass several assertions.

For example, to make sure the analytics worked as expected, I wrote a test module called test_analyze:

# Not all test cases are shown, please check the full code of 'test/test_analyze.py'
import unittest
from pandas import DataFrame
from empirestaterunup.analyze import get_country_counts
from empirestaterunup.data import load_data

class AnalyzeTestCase(unittest.TestCase):
    df: DataFrame

    @classmethod
    def setUpClass(cls) -> None:
        cls.df = load_data()

    def test_get_country_counts(self):
        country_counts, min_countries, max_countries = get_country_counts(df=AnalyzeTestCase.df)
        self.assertIsNotNone(country_counts)
        self.assertEqual(2, country_counts['JPN'])
        self.assertIsNotNone(min_countries)
        self.assertEqual(3, min_countries.shape[0])
        self.assertIsNotNone(max_countries)
        self.assertEqual(14, max_countries.shape[0])


if __name__ == '__main__':
    unittest.main()

So far we got the data, and made sure it meets the expectations. I wrote separate tests for the analytics code and also for the scraper.

Testing the user interface requires a different approach, as it needs to simulate clicks and wait for screen changes. Sometimes failures are easy to spot (like crashes), but sometimes issues are much more subtle (did we get the right data displayed?).

Will revisit this particular testing modality after we introduce first how to visualize the results.

How to Visualize the Results

I wanted to use the terminal as much as possible to visualize my findings, and to keep requirements to a minimum. I decided to use the Textual framework to accomplish that.

This framework is very complete and allows you to build text applications that are responsive and beautiful to look at.

They are also easy to write, so before we go deeper into the resulting applications, let's pause to learn about Textual.

Text User Interfaces (TUI) with Textual

The Textual project has a nice tutorial that you can read to get up to speed.

Let's see some code. One of the applications is called esru_outlier. TUI code lives on the apps module that shows several tables together with the outliers we found before, using the z-score.

OutlierApp (extends App) collects all the basic information on a table for each outlier group and then calls the RunnerDetailScreen to display details about a runner.

Outliers first screen (by Age, Running Time, and Pace)

Next is code with explanations that shows how to build this screen:

# Only the code of the application shown here
# This application shows 3 tables: SUMMARY_METRICS = (RaceFields.AGE.value, RaceFields.TIME.value, RaceFields.PACE.value)
# Every application in Textual extends the App class
class OutlierApp(App):
    DF: DataFrame = None
    BINDINGS = [ ("q", "quit_app", "Quit"), ]  # Bind 'q' to 'quit_app' method `action_quit_app`, which in turn exists the app
    CSS_PATH = "outliers.tcss"  # Styling can be done externally, similar to using CSS
    ENABLE_COMMAND_PALETTE = False

    def action_quit_app(self):
        self.exit(0)

    def compose(self) -> ComposeResult:
        """
        Here we 'Yield' Widgets/ components that will be rendered in order on the TUI
        How do the components get their layout on the screen? They use a cascading style sheet (CSS): outliers.tcss and
        some explicit layout containers like the class `Vertical` that can contain other Widgets
        Here we have a header, tables, and a footer 
        """
        yield Header(show_clock=True)
        for column_name in SUMMARY_METRICS:
            table = DataTable(id=f'{column_name}_outlier')
            table.cursor_type = 'row'
            table.zebra_stripes = True
            table.tooltip = "Get runner details"
            if column_name == RaceFields.AGE.value:
                label = Label(f"{column_name} (older) outliers:".title())
            else:
                label = Label(f"{column_name} (slower) outliers:".title())
            yield Vertical(
                label,
                table
            )
        yield Footer()

    def on_mount(self) -> None:
        """
        Here we populate each table with data from the DataFrame. Each table has outliers of different types.
        All can be obtained with the `get_outliers` method.
        """
        for column in SUMMARY_METRICS:
            table = self.get_widget_by_id(f'{column}_outlier', expect_type=DataTable)
            columns = [x.title() for x in ['bib', column]]
            table.add_columns(*columns)
            table.add_rows(*[get_outliers(df=OutlierApp.DF, column=column).to_dict().items()])

    @on(DataTable.HeaderSelected)
    def on_header_clicked(self, event: DataTable.HeaderSelected):
        """
        When the user selects a column header it generates a 'HeaderSelected' event.
        The annotation on this method tells Textual that we will handle this event here
        We can extract the table, the selected column, and then sort the table contents.
        """
        table = event.data_table
        table.sort(event.column_key)

    @on(DataTable.RowSelected)
    def on_row_clicked(self, event: DataTable.RowSelected) -> None:
        """
        Similarly, when the user selects a row it generates a RowSelected method
        What we do on the 'on_row_clicked' method is capture the event, get the row contents, and construct
        a new modal screen (RunnerDetailScreen) which we push on top of the regular screen.
        There we show the runner details differently. 
        """
        table = event.data_table
        row = table.get_row(event.row_key)
        runner_detail = RunnerDetailScreen(df=OutlierApp.DF, row=row)
        self.push_screen(runner_detail)

The class RunnerDetailScreen (extends ModalScreen) handles showing the racer details using formatted Markdown, which shows up when you click on the table that was rendered before:

Rendered Markdown with details about the selected runner

And here's the code that allows that with explanations:

# Omitted imports and helper methods, only showing TUI-related code. See the 'apps.py' file for full code
class RunnerDetailScreen(ModalScreen):
    ENABLE_COMMAND_PALETTE = False  # Disable the search bar, it is active by default and is not needed here
    CSS_PATH = "runner_details.tcss"  # Handle the styles using external CSS

    def __init__(
            self,
            name: str | None = None,
            ident: str | None = None,
            classes: str | None = None,
            row: List[Any] | None = None,
            df: DataFrame = None,
            country_df: DataFrame = None
    ):
        """
        Override the constructor and load useful data like country ISO codes
        We get the Pandas DataFrame with the details that will be shown to the user
        """
        super().__init__(name, ident, classes)
        self.row = row
        self.df = df
        if not country_df:
            self.country_df = load_country_details()
        else:
            self.country_df = country_df

    def compose(self) -> ComposeResult:
        """
        In compose we prepare the markdown, and we let the MarkdownViewer handle details like 
        a nice automatic table of contents.
        Notice that we call `self.log.info('xxx'). We use that for debugging when this application
        is called using 'textual'.
        """
        bib_idx = FIELD_NAMES_AND_POS[RaceFields.BIB]
        bibs = [self.row[bib_idx]]
        columns, details = df_to_list_of_tuples(self.df, bibs)
        self.log.info(f"Columns: {columns}")
        self.log.info(f"Details: {details}")
        row_markdown = ""
        position_markdown = {}
        split_markdown = {}
        for legend in ['full', '20th', '65th']:
            position_markdown[legend] = ''
            split_markdown[legend] = ''
        for i in range(0, len(columns)):
            column = columns[i]
            detail = details[0][i]
            if re.search('pace|time', column):
                if re.search('20th', column):
                    split_markdown['20th'] += f"\n* **{column.title()}:** {detail}"
                elif re.search('65th', column):
                    split_markdown['65th'] += f"\n* **{column.title()}:** {detail}"
                else:
                    split_markdown['full'] += f"\n* **{column.title()}:** {detail}"
            elif re.search('position', column):
                if re.search('20th', column):
                    position_markdown['20th'] += f"\n* **{column.title()}:** {detail}"
                elif re.search('65th', column):
                    position_markdown['65th'] += f"\n* **{column.title()}:** {detail}"
                else:
                    position_markdown['full'] += f"\n* **{column.title()}:** {detail}"
            elif re.search('url|bib', column):
                pass  # Skip uninteresting columns
            else:
                row_markdown += f"\n* **{column.title()}:** {detail}"
        yield MarkdownViewer(f"""# Full Course Race details     
## Runner BIO (BIB: {bibs[0]})
{row_markdown}
## Positions
### 20th floor        
{position_markdown['20th']}
### 65th floor        
{position_markdown['65th']}
### Full course        
{position_markdown['full']}                
## Race time split   
### 20th floor        
{split_markdown['20th']}
### 65th floor        
{split_markdown['65th']}
### Full course        
{split_markdown['full']}         
        """)
        # This button is used to close this screen and send the user to the previous screen
        btn = Button("Close", variant="primary", id="close")
        btn.tooltip = "Back to main screen"
        yield btn

    @on(Button.Pressed, "#close")
    def on_button_pressed(self, _) -> None:
        """
        Simple logic, pop the previous screen and make this one disappear
        """
        self.app.pop_screen()

This class is reusable. There are other classes (like BrowserApp in this tutorial) that also send data when a user clicks on a table row, and those details get displayed using this modal screen.

We can customize the appearance using CSS (yes, like a web application). It looks a lot like a web application's CSS (but it's not exactly the same). For example to add style to a button, here's the code:

Button {
    dock: bottom;
    width: 100%;
    height: auto;
}

As you can see, Textual is a pretty powerful framework. It reminds me a lot of Java Swing, but without the extra complexity.

But is it just information in tabular format? I also wanted to have different graph types that could explain behavior like age cluster and gender distribution. For that, I wrote a few classes on the 'apps' module with the help of Matplotlib.

Plots with Matplotlib

I wanted to use some charts to display the data, and I made them with matplotlib. The code to generate an age plot box, that shows how old the participating runners were, is very straightforward.

Age box plot in Matplotlib that shows than most of the runners were in the 40-50 year old range.

And here's the code that produced that plot:

# Not all code is shown here (helper methods, imports)
# Please check the apps.py module to see all missing code
class Plotter:
    def plot_gender(self):
        """
        In this method, we get our data frame filtering by gender and get counts
        Then we create a pie plot
        """
        series = self.df[RaceFields.GENDER.value].value_counts()
        fig, ax = plt.subplots(layout='constrained')
        wedges, texts, auto_texts = ax.pie(
            series.values,
            labels=series.keys(),
            autopct="%%%.2f",
            shadow=True,
            startangle=90,
            explode=(0.1, 0, 0)
        )
        ax.set_title = "Gender participation"
        ax.set_xlabel('Gender distribution')

        # Legend with the fastest runners by gender
        fastest = find_fastest(self.df, FastestFilters.Gender)
        fastest_legend = [f"{fastest[gender]['name']} - {beautify_race_times(fastest[gender]['time'])}" for gender in
                          series.keys()]
        ax.legend(wedges, fastest_legend,
                  title="Fastest by gender",
                  loc="center left",
                  bbox_to_anchor=(1, 0, 0.5, 1))

Interesting – most of the runners were between 40-50 years old.

Now let's go back to testing TUI.

Testing the User Interfaces

When I started working on this small project, I knew that there was going to be a lot of testing. What I wasn't sure about was how I would be able to test the TUI.

I figured at least two ways would be useful with Textual: one being able to see the message flow between components and the other using unit tests with a twist:

Following the message flow with Textual

Textual supports an interesting development mode that allows you to change CSS and see the changes on your application without a restart. Also, you can see how the TUI events propagate, which is invaluable for debugging.

In one terminal, start the console:

(EmpireStateRunUp) [josevnz@dmaf5 EmpireStateRunUp]$ . ~/virtualenv/EmpireStateRunUp/bin/activate
(EmpireStateRunUp) [josevnz@dmaf5 EmpireStateRunUp]$ textual console
▌Textual Development Console v0.46.0                                                                                                                                             
▌Run a Textual app with textual run --dev my_app.py to connect.                                                                                                                  
▌Press Ctrl+C to quit.

Then in another terminal, start your application but using development mode:

(EmpireStateRunUp) [josevnz@dmaf5 EmpireStateRunUp]$ textual run --dev --command esru_browser

If you check back on your console terminal, you will see any messages you sent with App.log along with the events:

─────────────────────────────────────────────────────────────────────────── Client '127.0.0.1' connected ───────────────────────────────────────────────────────────────────────────
[18:28:17] SYSTEM                                                                                                                                                        app.py:2188
Connected to devtools ( ws://127.0.0.1:8081 )
[18:28:17] SYSTEM                                                                                                                                                        app.py:2192
---
[18:28:17] SYSTEM                                                                                                                                                        app.py:2194
driver=<class 'textual.drivers.linux_driver.LinuxDriver'>
[18:28:17] SYSTEM                                                                                                                                                        app.py:2195
loop=<_UnixSelectorEventLoop running=True closed=False debug=False>
[18:28:17] SYSTEM                                                                                                                                                        app.py:2196
features=frozenset({'debug', 'devtools'})
[18:28:17] SYSTEM                                                                                                                                                        app.py:2228
STARTED FileMonitor({PosixPath('/home/josevnz/EmpireStateCleanup/docs/EmpireStateRunUp/empirestaterunup/browser.tcss')})
[18:28:17] EVENT                                                                                                                                                 message_pump.py:706
Load() >>> BrowserApp(title='Race Runners', classes={'-dark-mode'}) method=None
[18:28:17] EVENT                                                                                                                                                 message_pump.py:697
Mount() >>> DataTable(id='runners') method=<ScrollView.on_mount>
[18:28:17] EVENT                                                                                                                                                 message_pump.py:697
Mount() >>> DataTable(id='runners') method=<Widget.on_mount>
[18:28:17] EVENT                                                                                                                                                 message_pump.py:697
Mount() >>> Footer() method=<Footer.on_mount>
[18:28:17] EVENT                                                                                                                                                 message_pump.py:697
Mount() >>> Footer() method=<Widget.on_mount>
[18:28:17] EVENT                                                                                                                                                 message_pump.py:697
Mount() >>> ToastRack(id='textual-toastrack') method=<Widget.on_mount>
...
RowHighlighted(cursor_row=0, row_key=<textual.widgets._data_table.RowKey object at 0x7fc8d98800d0>) >>> BrowserApp(title='Race Runners', classes={'-dark-mode'}) method=None
[18:28:17] EVENT                                                                                                                                                 message_pump.py:697
Mount() >>> ScrollBarCorner() method=<Widget.on_mount>
[18:28:17] EVENT                                                                                                                                                 message_pump.py:706
Resize(size=Size(width=2, height=1), virtual_size=Size(width=178, height=47), container_size=Size(width=178, height=47)) >>> ScrollBarCorner() method=None
[18:28:17] EVENT                                                                                                                                                 message_pump.py:706
Show() >>> ScrollBarCorner() method=None

Using unittest and Pilot

The framework has the Pilot class that you can use to make automated calls to Textual Widgets and wait for events. This means you can simulate user interaction with the application to validate that it behaves as expected. This is more powerful than the regular unit tests as you can also cover UI interactions with expected results:

import unittest
from textual.widgets import DataTable, MarkdownViewer
from empirestaterunup.apps import BrowserApp


class AppTestCase(unittest.IsolatedAsyncioTestCase):
    async def test_browser_app(self):
        app = BrowserApp()
        self.assertIsNotNone(app)
        async with app.run_test() as pilot:

            """
            Test the command palette
            """
            await pilot.press("ctrl+\\")
            for char in "jose".split():
                await pilot.press(char)
            await pilot.press("enter")
            # This returns the runner screen. Check that it has some contents
            markdown_viewer = app.screen.query(MarkdownViewer).first()
            self.assertTrue(markdown_viewer.document)
            await pilot.click("#close")  # Close the new screen, pop the original one
            # Go back to the main screen, now select a runner but using the table
            table = app.screen.query(DataTable).first()
            coordinate = table.cursor_coordinate
            self.assertTrue(table.is_valid_coordinate(coordinate))
            await pilot.press("enter")
            await pilot.pause()
            markdown_viewer = app.screen.query(MarkdownViewer).first()
            self.assertTrue(markdown_viewer)
            # After validating the markdown one more time, close the app
            # Quit the app by pressing q
            await pilot.press("q")

if __name__ == '__main__':
    unittest.main()

This is invaluable, and something that many times requires an external toolset to validate (for example in Java you have the class Robot).

How to Run the Applications

Finally, it's time to get familiar with mini applications (you can see an animated demonstration of the TUI applications here).

Browsing Through the Data

The esru_browser is a simple browser that lets you navigate through the raw race data.

esru_browser

The application shows all the race details for every Runner in a table that allows sorting by column.

The esru_browser window shows all runners' results. Here you can sort, search for runners, and click to get more details

And the command palette allows searching for runners by name (it's basically a search bar with fuzzy logic):

Matches show up on the palette as you type

Summary Reports

To get insights about racer behavior, you need some summary reports (as opposed to drilling down into each racer's details).

This application provides details about the following:

• Count, standard deviation, mean, min, max 45%, 50%, and 75% for age, time, and pace

• Group and count distribution for Age, Wave, and Gender

esru_numbers

Some interesting facts about the race:

• The average age was 41 years old, and 40 years old was the largest age group.

• The majority number of people belonged to the 'BLACK WAVE'.

• The majority of the people finished the race in between 20 and 30 minutes.

• The youngest runner was 11 years old, and the oldest was 78.

esru_numbers gives a bird's eye view of all the racers, categorized by buckets

Finding Outliers

This application uses the Z-score to find the outliers for several metrics for this race:

esru_outlier

the esru_outlier main screen shows you racers that did not follow regular patterns

Because these results drill down to the BIB number, you can click on a row and get more details about a runner:

And you can get details for each outlier. Yes, code is reusable and is the same to show details for any runner

Textual has excellent support for rendering Markdown as well as programming languages. Take a look at the code to see for yourself.

A Few Plot Graphics For You

The esru_plot application offers a few plot graphics to help you visualize the data. Inside, the class Plotter does all the heavy lifting

Age plots

The program can generate two flavors for the same data, one is a Box diagram:

The age box diagram we saw before

The second is a regular histogram:

Age histogram shows the same as the box diagram but the buckets are more visible. Same data, many ways to explain the racer demographics.

You can see from both graphics that the group age with the most participants is the 40-45-year-old bracket and the outliers are in the 10-20 and 70-80 year old groups.

Participants per country plot

This plot shows all the countries with the number of participants, with the best runner from each.

No surprises here: the overwhelming majority of racers come from the United States, followed by Mexico. Interestingly, the winner of the 2023 race is from Malaysia, with only 2 runners participating.

Gender distribution

The gender distribution pie showing the best racer for each category

The majority of the runners identified themselves as Males, followed by Females.

What Else Can We Learn?

NYC was well represented on the event. Yeah, I'm talking about the NYC police department running in full gear, not me on the left ;-)

Participating in this race was a great experience. The best part was that it fueled my curiosity and led me to write this code to get more interesting facts about the race.

There is plenty more to learn about the tools you just saw in this tutorial:

• There are a lot of public race datasets, and you can use them to apply what you learned here. Just take a look at this dataset of the New York City Marathon, period 1970-2018. What other questions you can ask about the data?

• You saw just the tip of what you can do with Textual. I encourage you to explore the apps.py module. Take a look at the example applications as well.

• Selenium Web driver is not just a tool for web scraping but for automated testing of web applications. It doesn't get better than having your browser perform automated testing for you. It is a big framework, so be prepared to spend time reading and running your tests. I strongly suggest you look at the examples. Trial an error will give you better results.

• Apply for the Empire Estate Run Up lottery or run through a charity, if you like this kind of race. Who said King Kong is the only one who could make it to the top?

• Sadly, I'm not in a position to offer you any training advice. Every person is different. I do recommend that you check with your doctor before you participate in a race like this, and get some professional advice from a running coach.

• But most important of all, believe you can do this (the race and writing some tools to process the race data) and have fun while doing it. This is a pre-requisite for any project.

We will cover the following:

• find

• xargs and nproc

• taskset

• numactl

• watch

• inotify-tools

I will present you with a challenge and the tools demonstrating how to solve each problem.

What You'll Need:

• A Linux distribution

• Curiosity

How to Handle Directories with Many Files

You may have encountered this problem before: you tried to do a ls on a directory with a very large number of files, but the command threw an 'argument list too long' error:

josevnz@orangepi5:/data/test_xargs$ ls *
-bash: /usr/bin/ls: Argument list too long

This is because POSIX-compatible systems have a limit for the maximum number of bytes you can pass as an argument:

[josevnz@dmaf5 Documents]$ getconf ARG_MAX
2097152

2 Million bytes may seem like a lot – or not enough depending on whom you ask. But it's also a protection against attacks or innocent mistakes with bad consequences.

In any case, how can you bypass this limitation? Well, there are many ways to do so.

Using Shell built-in

Bash built-in doesn't have the ARG_MAX limitation:

josevnz@orangepi5:/data/test_xargs$ echo *|ls
...
test_file055554  test_file111110  test_file166666  test_file222222  test_file277778  test_file333334  test_file388890  test_file444446
test_file055555  test_file111111  test_file166667  test_file222223  test_file277779  test_file333335  test_file388891  test_file444447
test_file055556  test_file111112  test_file166668  test_file222224  test_file277780  test_file333336  test_file388892  test_file444448

This is probably the simplest solution, but let's see another way.

Using find when you want formatting options

Or you can use this well known find flag:

find /data/test_xargs -type f -ls -printf '%name'

Or with formatting, to mimic ls:

find /data/test_xargs -type f -printf '%f\n

This is fast and also the most complete solution. But before moving on I'll show you yet another way.

Using xargs

The following works:

find /data/test_xargs -type f -print0 | xargs -0 ls

But it's inefficient, as you are forking 3 processes to display the contents of the directory. And on top of that, xargs is throttling how many files will be passed to the ls command.

Let's move on and check out a different problem.

How to Run More Programs Without Crashing the Server

First you walk then you run: Do it serially

So say that you want to compress all the files on the given directory from our previous example. A first try would be like this:

gzip *

Which will take a long time as gzip will process one file at the time.

You might think to do something like this to compress files in parallel:

josevnz@orangepi5:/data/test_xargs$ for file in $(ls data/test_xargs/*); do gzip $file &; done
-bash: /usr/bin/ls: Argument list too long

Again, ARG_MAX strikes again.

We know xargs or find now, so what if we do this:

for file in $(find $PWD); do echo gzip $file &; done
wait
echo "All files compressed?"

That will either make your server run out of memory or crush it under very heavy CPU utilization because you are forking a gzip instance for every file found.

Our first attempt at parallelism and throttling (the art of self control)

What you need is a way to throttle your compression requests, so you don't launch more processes than the number of CPUS you have.

Let's try that again with find and xargs:

find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %

Oh. That looks like a fancy one-liner. Let me explain how it works:

1. Use find to get all files on the given directory, use the null character as a separator to be able to process weird named ones.

2. nproc will tell you how many CPUS you have, then subtract 1 using Bash arithmetic like this using sub-shells: $(($(nproc)-1))

3. Finally, xargs will run no more than -P processes (In my case 8 CPUS - 1 = 7 jobs), replacing the '%' with the name of the file to compress

Note: There are other ways to get the number of CPUS on the machine, like parsing /proc/cpuinfo. There are other more efficient compression out there but gzip is available on pretty much any Linux/ Unix out there.

OK, time to see our next problem.

CPU Affinity with taskset to Maximize Execution Time

Despite limiting the number of CPUs, some intensive jobs can slow down other processes on your machine when looking for resources. There are a few things you can do to keep the performance of the server under control, like using taskset:

The taskset command is used to set or retrieve the CPU affinity
of a running process given its pid, or to launch a new command
with a given CPU affinity. CPU affinity is a scheduler property
that "bonds" a process to a given set of CPUs on the system.

In general, we always want to leave one of the CPUS 'free' for operating system tasks. The Kernel is normally pretty good keeping running processes glued to a specific CPU to avoid context switching, but if you want to enforce on which CPUS your process will run you can use tasket

taskset -c 1,2,3,4,5,6,7 find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %

taskset the only game in town? not so numactl fast!

What is NUMA and why you should care?

There are physical limitations to hardware that are encountered when many CPUs and lots of memory are required. The important limitation is that there is limited communication bandwidth between the CPUs and the memory.

One architecture modification that was introduced to address this is Non-Uniform Memory Access (NUMA).

So most simple desktop machines only have a single NUMA node, like mine:

[josevnz@dmaf5 ~]$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 15679 MB
node 0 free: 5083 MB
node distances:
node   0 
  0:  10
# Or with lscpu
[josevnz@dmaf5 ~]$ lscpu |rg NUMA
NUMA node(s):                    1
NUMA node0 CPU(s):               0-7

If you have more than one NUMA node, you may want to 'pin' or set the affinity of your program to use the CPUS and memory of the same node.

For example, on a machine with 16 cores, 0-7 on node 0, 8-15 on node 1, we could force our compression program to run on all the CPUS on node 1, and use the memory of node 1 like this:

numactl --physcpubind 8-15 --membind=1 find /data/test_xargs -type f -print0| xargs -0 -P $(($(nproc)-1)) -I % gzip %

Keeping an Eye on Things

Just watch what I do

The watch command allows you to periodically run a command, and even show you the differences before calls:

Every 10.0s: ls                                                                                                         orangepi5: Wed May 24 22:46:33 2023

test_file000001.gz
test_file000002.gz
test_file000003.gz
test_file000004.gz
test_file000005.gz
test_file000006.gz
test_file000007.gz
test_file000008.gz
test_file000009.gz
test_file000010.gz
...

Shows me the output of the ls command every 10 seconds. To detect changes on a directory this is simple, but not easy to automate and definitely not efficient.

Wouldn't be nice if the kernel was able to tall me about changes on my directories?

A better way to know about changes on the filesystem, with inotify-tools

You may need to install this separately, but it should be easy to do. On Ubuntu:

sudo apt-get install inotify-tools

On Fedora:

sudo dnf install -y inotify-tools

So how we can monitor for events on a given directory?

On one terminal we can run inotifywait:

josevnz@orangepi5:/data/test_xargs$ inotifywait --recursive /data/test_xargs/
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.

And on another terminal we can touch some files to simulate an event:

josevnz@orangepi5:/data/test_xargs$ pwd
/data/test_xargs
josevnz@orangepi5:/data/test_xargs$ touch test_file285707.gz test_file357136.gz test_file428565.gz

The original terminal will get the first event and exit:

Watches established.
/data/test_xargs/ OPEN test_file285707.gz

To make it listen for even forever we do this:

josevnz@orangepi5:/data/test_xargs$ inotifywait --recursive --monitor /data/test_xargs/

If we touch the file again on a separate terminal then this time we will see all the events:

Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/data/test_xargs/ OPEN test_file285707.gz
/data/test_xargs/ ATTRIB test_file285707.gz
/data/test_xargs/ CLOSE_WRITE,CLOSE test_file285707.gz
/data/test_xargs/ OPEN test_file357136.gz
/data/test_xargs/ ATTRIB test_file357136.gz
/data/test_xargs/ CLOSE_WRITE,CLOSE test_file357136.gz
/data/test_xargs/ OPEN test_file428565.gz
/data/test_xargs/ ATTRIB test_file428565.gz
/data/test_xargs/ CLOSE_WRITE,CLOSE test_file428565.gz

This is less taxing to the operating system than asking for directory changes every time, and filtering just the differences ourselves.

What's Next

There is so much more to explore. The tips above introduced you to some important concepts, so why not to learn much more about them?

• The Ubuntu forum has a great conversation about xargs, find, ulimit and other things. Knowledge is power.

• RedHat as a nice page about NUMA, taskset, interrupt handling. If you are serious about fine-tuning the performance of your processes, please read it.

• You liked inotify and want to use it from your Python script. Then take a look at pynotify.

• Find may be intimidating, but this tutorial will make it easier to understand.

• Source code for this tutorial can be found here.

In this tutorial I will show you how you can replace some of the tedious packaging of third party applications.

What You Need to Complete this Tutorial

• A Linux distribution (I used Fedora but this works with anything)

• Elevated privileges (if you want to install your own packages)

When Your Package Manager Isn't Simple Enough

Often times, you'll want to have the ultimate control over how you package an application. But there are a few occasions when this may be overkill:

1. The third party application is simple or small enough than a tar would be good enough to install it. Yet you want to enjoy the benefits of upgrades and roll-back, like the ones offered by RPM.

2. You need or want to package an application from one format (say .tar.gz) to Debian '.deb' or RPM.

3. You have to package multiple applications that are only offered in Source format or pre-packaged binaries, like when upgrading the operating system. And you don't want to spend an eternity re-packaging the third party applications.

How to Package an Existing Application the Old Way

I wrote an small demo application that dumps system facts (like disk utilization) in JSON format, called [jdumpertools](https://github.com/josevnz/jdumpertools). The application is very simple, is written in C, and has an RPM spec file that you can use to package the software.

There are a few manual steps required to create the RPM:

1. Download the source distribution (or binary): git clone https://github.com/josevnz/jdumpertools.git

2. Prepare the RPM spec file, which should take care of compilation (or just packaging) of the software, as well the location for the installation

3. Lint the spec file, fix common errors

So let's see how jdumbertools's RPM spec file works.

First, take a look at the spec file:

Name:           jdumpertools
# TODO: Figure out a better way to update version here and on Makefile
%global major 0
Version:        v%{major}.2
Release:        1%{?dist}
Summary:        Programs that can be used to dump Linux usage data in JSON format

License:        ASL 2.0
URL:            https://github.com/josevnz/jdumpertools
Source0:        %{name}-%{version}.tar.gz

BuildRequires:  bash,tar,gzip,rpmdevtools,rpmlint,make,gcc >= 10.2.1
Requires:       bash

%global debug_package %{nil}

%description

Jdumpertools is a collection of programs that can be used to dump
linux usage data in JSON format, so it can be ingested by other tools.

* jdu: Similar to UNIX '/bin/du' command.
* jutmp: UTMP database dumper

%prep
%setup -q -n jdumpertools

%build
make all

%install

/usr/bin/mkdir -p %{buildroot}/%{_bindir}
/usr/bin/mkdir -p %{buildroot}/%{_mandir}/man8
/usr/bin/cp -v -p jdu jutmp %{buildroot}/%{_bindir}
/usr/bin/cp -v -p jdu.1 jutmp.1 %{buildroot}/%{_mandir}/man8/
/usr/bin/gzip %{buildroot}/%{_mandir}/man8/*
/usr/bin/mkdir -p %{buildroot}/%{_libdir}
/usr/bin/cp -v -p libjdumpertools.so.%{major} %{buildroot}/%{_libdir}
/usr/bin/strip %{buildroot}/%{_bindir}/{jdu,jutmp}
/usr/bin/strip %{buildroot}/%{_libdir}/*

%clean
rm -rf %{buildroot}

%files
%{_bindir}/jdu
%{_bindir}/jutmp
%{_libdir}/libjdumpertools.so.%{major}
%{_libdir}/libjdumpertools.so
%license LICENSE
%doc README.md
%doc %{_mandir}/man8/jdu.1.gz
%doc %{_mandir}/man8/jutmp.1.gz


%changelog
* Sun Oct  3 2021 Jose Vicente Nunez <kodegeek.com@protonmail.com> - v0.2-1
- Applied fixes from rpmlint: man page, typos on spec file, striped binaries, etc.
* Mon Jan  4 2021 Jose Vicente Nunez <kodegeek.com@protonmail.com> - v0.1-1
- First version being packaged

And now let's build it:

[josevnz@dmaf5 jdumpertools]$ sudo dnf install -y rpmdevtools rpmlint
...
[josevnz@dmaf5 test]$ git clone https://github.com/josevnz/jdumpertools.git
Cloning into 'jdumpertools'...
remote: Enumerating objects: 228, done.
remote: Counting objects: 100% (228/228), done.
remote: Compressing objects: 100% (137/137), done.
remote: Total 228 (delta 132), reused 157 (delta 79), pack-reused 0
Receiving objects: 100% (228/228), 3.15 MiB | 9.67 MiB/s, done.
Resolving deltas: 100% (132/132), done.

[josevnz@dmaf5 test]$ cd jdumpertools/
[josevnz@dmaf5 jdumpertools]$ rpmbuild -ba jdumpertools.spec
...
+ exit 0
Provides: jdumpertools = v0.2-1.fc37 jdumpertools(x86-64) = v0.2-1.fc37 libjdumpertools.so()(64bit)
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libjdumpertools.so()(64bit) rtld(GNU_HASH)
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/josevnz/rpmbuild/BUILDROOT/jdumpertools-v0.2-1.fc37.x86_64
Wrote: /home/josevnz/rpmbuild/SRPMS/jdumpertools-v0.2-1.fc37.src.rpm
Wrote: /home/josevnz/rpmbuild/RPMS/x86_64/jdumpertools-v0.2-1.fc37.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.42keBq
+ umask 022
+ cd /home/josevnz/rpmbuild/BUILD
+ cd jdumpertools
+ rm -rf /home/josevnz/rpmbuild/BUILDROOT/jdumpertools-v0.2-1.fc37.x86_64
+ RPM_EC=0
++ jobs -p
+ exit 0
Executing(rmbuild): /bin/sh -e /var/tmp/rpm-tmp.aZjb6s
+ umask 022
+ cd /home/josevnz/rpmbuild/BUILD
+ rm -rf jdumpertools jdumpertools.gemspec
+ RPM_EC=0
++ jobs -p
+ exit 0
...
[josevnz@dmaf5 jdumpertools]$ ls -l $HOME/rpmbuild/RPMS/x86_64/jdumpertools-v0.2-1.fc37.x86_64.rpm
-rw-r--r--. 1 josevnz josevnz 22118 Jun  2 14:03 /home/josevnz/rpmbuild/RPMS/x86_64/jdumpertools-v0.2-1.fc37.x86_64.rpm

Then you install the RPM like any other RPM:

[josevnz@dmaf5 jdumpertools]$ sudo dnf install -y $HOME/rpmbuild/RPMS/x86_64/jdumpertools-v0.2-1.fc37.x86_64.rpm
Last metadata expiration check: 1:36:46 ago on Fri 02 Jun 2023 12:30:31 PM EDT.
Dependencies resolved.
=================================================================================================================================
 Package                         Architecture              Version                         Repository                       Size
=================================================================================================================================
Installing:
 jdumpertools                    x86_64                    v0.2-1.fc37                     @commandline                     22 k

Transaction Summary
=================================================================================================================================
Install  1 Package

Total size: 22 k
Installed size: 57 k
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                         1/1 
  Installing       : jdumpertools-v0.2-1.fc37.x86_64                                                                         1/1 
  Running scriptlet: jdumpertools-v0.2-1.fc37.x86_64                                                                         1/1 
  Verifying        : jdumpertools-v0.2-1.fc37.x86_64                                                                         1/1 

Installed:
  jdumpertools-v0.2-1.fc37.x86_64                                                                                                

Complete!

It's not terrible, specially if you plan to make updates – but can we do this in an easier way?

How to Install FPM

The getting started document the simplest reference you can refer to in order to get FPM up and running.

First you'll install some dependencies, for example in Fedora:

[josevnz@dmaf5 jdumpertools]$ sudo dnf install -y gem
[josevnz@dmaf5 jdumpertools]$ sudo dnf install -y rpm-build squashfs-tools

And then you'll install FPM itself:

[josevnz@dmaf5 jdumpertools]$ gem install --user-install fpm
Fetching insist-1.0.0.gem
Fetching clamp-1.0.1.gem
Fetching stud-0.0.23.gem
Fetching rexml-3.2.5.gem
Fetching mustache-0.99.8.gem
Fetching dotenv-2.8.1.gem
Fetching cabin-0.9.0.gem
Fetching pleaserun-0.0.32.gem
Fetching fpm-1.15.1.gem
Fetching backports-3.24.1.gem
...
Done installing documentation for stud, rexml, mustache, insist, dotenv, clamp, cabin, pleaserun, backports, arr-pm, fpm after 5 seconds
11 gems installed

How to Package jdumpertools as an RPM, Without a Spec File

Well, we need some files to package. This distribution comes with a Makefile, so easy as pie we do:

[josevnz@dmaf5 jdumpertools]$ make
gcc -Wall -g -Og -Wextra -Werror -Werror=format-security -std=c11   -DJDUMPERTOOLS_VERSION=v0.2 -fPIC jdumpertools.h jdumpertools.c -I /home/josevnz/test/jdumpertools -shared -Wl,-soname,libjdumpertools.so -o libjdumpertools.so.0
gcc jdumpertools.h jdu.c libjdumpertools.so.0 -Wall -g -Og -Wextra -Werror -Werror=format-security -std=c11   -DJDUMPERTOOLS_VERSION=v0.2 -L /home/josevnz/test/jdumpertools -l jdumpertools -o jdu
gcc jdumpertools.h jutmp.c -Wall -g -Og -Wextra -Werror -Werror=format-security -std=c11   -DJDUMPERTOOLS_VERSION=v0.2 -L /home/josevnz/test/jdumpertools -l jdumpertools -o jutmp
...
[josevnz@dmaf5 jdumpertools]$ ls
CODE_OF_CONDUCT.md  INSTALL.md  jdu.c           jdumpertools.spec  jutmp.c               Makefile        SECURITY.md
CONTRIBUTING.md     jdu         jdumpertools.c  jutmp              libjdumpertools.so.0  mazinger-z.png
Dockerfile          jdu.1       jdumpertools.h  jutmp.1            LICENSE               README.md
[josevnz@dmaf5 jdumpertools]$ fpm -t rpm -s dir --name jdumpertools --rpm-autoreq --rpm-os linux --rpm-summary 'Programs that can be used to dump Linux usage data in JSON format' --license 'ASL 2.0' --version v0.21 --depends bash --maintainer 'Jose Vicente Nunez <kodegeek.com@protonmail.com>' --url https://github.com/josevnz/jdumpertools jdu=/usr/bin/jdu jutmp=/usr/bin/jutmp jdu.1=/usr/share/man/man1/jdu.1.gz jutmp.1=/usr/share/man/man8/jutmp.1.gz
Created package {:path=>"jdumpertools-v0.21-1.x86_64.rpm"}

So no spec file, and we've got ourselves an RPM.

What if I want to create packages for other distributions? I just need to make a few changes on the command line:

Debian package:

[josevnz@dmaf5 jdumpertools]$ fpm -t deb -s dir --name jdumpertools --rpm-autoreq --rpm-os linux --rpm-summary 'Programs that can be used to dump Linux usage data in JSON format' --license 'ASL 2.0' --version v0.21 --depends bash --maintainer 'Jose Vicente Nunez <kodegeek.com@protonmail.com>' --url https://github.com/josevnz/jdumpertools jdu=/usr/bin/jdu jutmp=/usr/bin/jutmp jdu.1=/usr/share/man/man1/jdu.1.gz jutmp.1=/usr/share/man/man8/jutmp.1.gz
Debian 'Version' field needs to start with a digit. I was provided 'v0.21' which seems like it just has a 'v' prefix to an otherwise-valid Debian version, I'll remove the 'v' for you. {:level=>:warn}
Created package {:path=>"jdumpertools_0.21_amd64.deb"}

Self extracting script:

[josevnz@dmaf5 jdumpertools]$ fpm -t sh -s dir --name jdumpertools --rpm-autoreq --rpm-os linux --rpm-summary 'Programs that can be used to dump Linux usage data in JSON format' --license 'ASL 2.0' --version v0.21 --depends bash --maintainer 'Jose Vicente Nunez <kodegeek.com@protonmail.com>' --url https://github.com/josevnz/jdumpertools jdu=/usr/bin/jdu jutmp=/usr/bin/jutmp jdu.1=/usr/share/man/man1/jdu.1.gz jutmp.1=/usr/share/man/man8/jutmp.1.gz
Created package {:path=>"jdumpertools.sh"}

tar file:

[josevnz@dmaf5 jdumpertools]$ fpm -t tar -s dir --name jdumpertools --rpm-autoreq --rpm-os linux --rpm-summary 'Programs that can be used to dump Linux usage data in JSON format' --license 'ASL 2.0' --version v0.21 --depends bash --maintainer 'Jose Vicente Nunez <kodegeek.com@protonmail.com>' --url https://github.com/josevnz/jdumpertools jdu=/usr/bin/jdu jutmp=/usr/bin/jutmp jdu.1=/usr/share/man/man1/jdu.1.gz jutmp.1=/usr/share/man/man8/jutmp.1.gz
Created package {:path=>"jdumpertools.tar"}

This is already very convenient. Now I want to show you another use case for FPM.

How to Repackage Existing Software

Say that you want to distribute a CPAN module which doesn't have an RPM. You could spend quality time, or you could use FPM to do the work for you.

First, let's install a new dependency for Fedora:

[josevnz@dmaf5 jdumpertools]$ sudo dnf install -y perl-App-cpanminus

And then let's build our RPM

[josevnz@dmaf5 jdumpertools]$ fpm -t rpm -s cpan Archive::Tar
Created package {:path=>"perl-Archive-Tar-3.02-1.noarch.rpm"}

Did it work?

[josevnz@dmaf5 jdumpertools]$ rpm -qil perl-Archive-Tar-3.02-1.noarch.rpm
Name        : perl-Archive-Tar
Version     : 3.02
Release     : 1
Architecture: noarch
Install Date: (not installed)
Group       : default
Size        : 177677
License     : perl_5
Signature   : (none)
Source RPM  : perl-Archive-Tar-3.02-1.src.rpm
Build Date  : Fri 02 Jun 2023 04:36:45 PM EDT
Build Host  : dmaf5
Relocations : / 
Packager    : <josevnz@dmaf5>
Vendor      : Jos Boumans <kane[at]cpan.org>
URL         : http://example.com/no-uri-given
Summary     : Manipulates TAR archives
Description :
Manipulates TAR archives
/usr/local/bin/ptar
/usr/local/bin/ptardiff
/usr/local/bin/ptargrep
/usr/local/share/man/man1/ptar.1
/usr/local/share/man/man1/ptardiff.1
/usr/local/share/man/man1/ptargrep.1
/usr/local/share/man/man3/Archive::Tar.3pm
/usr/local/share/man/man3/Archive::Tar::File.3pm
/usr/local/share/perl5/5.36/Archive/Tar.pm
/usr/local/share/perl5/5.36/Archive/Tar/Constant.pm
/usr/local/share/perl5/5.36/Archive/Tar/File.pm

Now I'm going to show you how to package the clickhouse-driver module from PyPi.

[josevnz@dmaf5 jdumpertools]$ fpm -t rpm -s python 'clickhouse-driver'
Created package {:path=>"python-clickhouse-driver-0.2.6-1.x86_64.rpm"}

Say that now you want to create an RPM for OpenJDK 17. No problem, get the tar file and package it with a little help:

[josevnz@dmaf5 jdumpertools]$ curl --fail --location --remote-name https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.7%2B7/OpenJDK17U-jdk_x64_linux_hotspot_17.0.7_7.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  182M  100  182M    0     0  10.9M      0  0:00:16  0:00:16 --:--:-- 11.1M
[josevnz@dmaf5 jdumpertools]$ fpm -t rpm -s tar --url 'https://adoptium.net/' --description 'Eclipse Temurin is the name of the OpenJDK distribution from Adoptium' --version '17.0.7+7' --prefix /usr/local/openjdk OpenJDK17U-jdk_x64_linux_hotspot_17.0.7_7.tar.gz
[josevnz@dmaf5 jdumpertools]$ rpm -qil OpenJDK17U-jdk_x64_linux_hotspot_17-17.0.7+7-1.x86_64.rpm
Name        : OpenJDK17U-jdk_x64_linux_hotspot_17
Version     : 17.0.7+7
Release     : 1
Architecture: x86_64
Install Date: (not installed)
Group       : default
Size        : 329508762
License     : unknown
Signature   : (none)
Source RPM  : OpenJDK17U-jdk_x64_linux_hotspot_17-17.0.7+7-1.src.rpm
Build Date  : Fri 02 Jun 2023 05:05:05 PM EDT
Build Host  : dmaf5
Relocations : /usr/local/openjdk 
Packager    : <josevnz@dmaf5>
Vendor      : none
URL         : https://adoptium.net/
Summary     : Eclipse Temurin is the name of the OpenJDK distribution from Adoptium
Description :
Eclipse Temurin is the name of the OpenJDK distribution from Adoptium
/usr/local/openjdk/jdk-17.0.7+7/NOTICE
/usr/local/openjdk/jdk-17.0.7+7/bin/jar
/usr/local/openjdk/jdk-17.0.7+7/bin/jarsigner
/usr/local/openjdk/jdk-17.0.7+7/bin/java
...

I could keep going but I think you get the idea how much you can to with FPM.

What's Next?

We covered some important use cases, but the tool has much more to offer:

• FPM has many other usages, including transforming existing packages from other formats to the one you want.

• FPM also supports configuration files. If you are using it often then you should read how to use a configuration file for FPM as opposed to use a lengthy command line.

• You may also consider running FPM from inside a container, to avoid installing dependencies.

• If you are curious about how to run the jumpertools binaries, you can take a look at the README.md from the repository.

Sometimes the CLI gets too complex – and that's when you can complement it with more exploratory versions of the programs called text user interfaces (TUI).

TUIs like HTOP, Glances, Midnight Commander, and others allow you to mix in the power of the CLI without sacrificing the ease of use.

So what can you do when your Python CLI has too many options and becomes intimidating? Wouldn't be nice if you could have a way to 'self' discover the app, and then once you're familiar with it, perform your tasks quickly using the options supported by the script?

Python has a very healthy ecosystem of GUI and TUI frameworks that you can use to write nice-looking and intuitive applications. In this tutorial we will talk about Trogon and what you can do to make your application more friendly yet powerful for new and seasoned users alike.

I'll show you two of them that can help you solve the following two problems:

1. Avoid becoming overwhelmed and having to use intimidating APIs when writing applications. Will use the Click Python package to solve that problem.

2. Allow discoverability. This is very important when you have an application that supports many options or that you haven't used in a while. That is where Trogon comes handy.

We will reuse the source code of one of my Open Source applications, rpm_query as a base. Rpm_query is a collection of simple applications that can query your system RPM database from the command line.

What You'll Need for This Tutorial

1. Linux's distribution, preferably one that uses RPM (Like Fedora or RedHat enterprise Linux)

2. Python 3.8+

3. Git

4. Familiarity with Python virtual environments

5. An Internet connection so you can download dependencies, using pip.

I strongly suggest that you clone the repository and create a virtual environment so you can follow the tutorial:

git clone https://github.com/josevnz/CLIWithClickAndTrogon.git
cd CLIWithClickAndTrogon
python3 -m venv ~/virtualenv/CLIWithCLickAndTrogon 
. ~/virtualenv/CLIWithCLickAndTrogon/bin/activate

If you're all set, let's dive in.

What a Typical CLI (Command Line Interface) Looks Like – Quick Refresher

This script uses a module inside the reporter Python package to query the RPM database.

#!/usr/bin/env python
"""
# rpmq_simple.py - A simple CLI to query the sizes of RPM on your system
Author: Jose Vicente Nunez
"""
import argparse
import textwrap

from reporter import __is_valid_limit__
from reporter.rpm_query import QueryHelper

if __name__ == "__main__":

    parser = argparse.ArgumentParser(description=textwrap.dedent(__doc__))
    parser.add_argument(
        "--limit",
        type=__is_valid_limit__,  # Custom limit validator
        action="store",
        default=QueryHelper.MAX_NUMBER_OF_RESULTS,
        help="By default results are unlimited but you can cap the results"
    )
    parser.add_argument(
        "--name",
        type=str,
        action="store",
        help="You can filter by a package name."
    )
    parser.add_argument(
        "--sort",
        action="store_false",
        help="Sorted results are enabled bu default, but you fan turn it off"
    )
    args = parser.parse_args()

    with QueryHelper(
        name=args.name,
        limit=args.limit,
        sorted_val=args.sort
    ) as rpm_query:
        for package in rpm_query:
            print(f"{package['name']}-{package['version']}: {package['size']:,.0f}")

Let's install it, in editable mode:

. ~/virtualenv/CLIWithCLickAndTrogon/bin/activate
pip install --editable .

And see it in action:

(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_simple.py --help
usage: rpmq_simple.py [-h] [--limit LIMIT] [--name NAME] [--sort]

# rpmq_simple.py - A simple CLI to query the sizes of RPM on your system Author: Jose Vicente Nunez

options:
  -h, --help     show this help message and exit
  --limit LIMIT  By default results are unlimited but you can cap the results
  --name NAME    You can filter by a package name.
  --sort         Sorted results are enabled bu default, but you fan turn it off
(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_simple.py --name kernel --limit 5
kernel-6.2.11: 0
kernel-6.2.14: 0
kernel-6.2.15: 0

So it seems than most of the code on the rpmq_simple.py script is boilerplate for the command line interface, using the standard 'ArgParse' library.

ArgParse is powerful, but it is also intimidating at first, specially when you have to support multiple use cases.

A New Way to Process the CLI with Click

The Click framework promises to make it easier to parse out command line arguments. To demonstrate that, let's convert our script from ArgParse to Click (they both provide support for options but Click has a few interesting options we will use):

#!/usr/bin/env python
"""
# rpmq_click.py - A simple CLI to query the sizes of RPM on your system
Author: Jose Vicente Nunez
"""
import click

from reporter.rpm_query import QueryHelper


@click.command()
@click.option('--limit', default=QueryHelper.MAX_NUMBER_OF_RESULTS,
              help="By default results are unlimited but you can cap the results")
@click.option('--name', help="You can filter by a package name.")
@click.option('--sort', default=True, help="Sorted results are enabled bu default, but you fan turn it off")
def command(
        name: str,
        limit: int,
        sort: bool
) -> None:
    with QueryHelper(
            name=name,
            limit=limit,
            sorted_val=sort
    ) as rpm_query:
        for package in rpm_query:
            click.echo(f"{package['name']}-{package['version']}: {package['size']:,.0f}")


if __name__ == "__main__":
    command()

So you will notice to big changes here:

1. Most of the boilerplate code from ArgParse is gone, replaced by annotations.

2. Click works by adding decorators to a new function called 'command', that takes arguments and executes the RPM query.

If you run the new script you will see that it works exactly as before:

(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_click.py --help
Usage: rpmq_click.py [OPTIONS]

Options:
  --limit INTEGER  By default results are unlimited but you can cap the
                   results
  --name TEXT      You can filter by a package name.
  --sort BOOLEAN   Sorted results are enabled bu default, but you fan turn it
                   off
  --help           Show this message and exit.
(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_click.py --name kernel --limit 5
kernel-6.2.11: 0
kernel-6.2.14: 0
kernel-6.2.15: 0

So what did we gain? Our code is slightly simpler but also is now supported by Trogon, a new framework we will discuss soon.

How to Use setuptools and Click

The Click documentation recommends that we should use setuptools to create a wrapper for our tool, automatically. So we need to define a function where we handle all the command line options and logic and the wrapper creates a regular script for us on the right place during the package installation. It also points to the right version of Python, among other nice things.

The documentation has the deprecated syntax for setup.py, so we will use the more recent setup.cfg format instead:

[metadata]
name = CLIWithClickAndTrogon
version = 0.0.1
author = Jose Vicente Nunez Zuleta
author-email = kodegeek.com@protonmail.com
license = Apache 2.0
summary = Simple TUI that queries the RPM database
home-page = https://github.com/josevnz/cliwithclickandtrogon
description = Simple TUI that queries the RPM database. A tutorial.
long_description = file: README.md
long_description_content_type = text/markdown

[options]
packages = reporter
setup_requires =
    setuptools
    wheel
    build
    pip
    twine
install_requires =
    importlib; python_version == "3.9"
    click
scripts =
    scripts/rpmq_simple.py
    scripts/rpmq_click.py
[options.entry_points]
console_scripts =
    rpmq = reporter.scripts:command

I created a package called 'scripts' inside the package called 'reporter' with the CLI logic using click.

setuptools will generate a script called 'rpmq' for us that behaves exactly as the previous script does – but again, we don't need any boilerplate code to pass arguments to Click:

CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ pip install --editable .
(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq --help
Usage: rpmq [OPTIONS]

Options:
  --limit INTEGER  By default results are unlimited but you can cap the
                   results
  --name TEXT      You can filter by a package name.
  --sort BOOLEAN   Sorted results are enabled bu default, but you fan turn it
                   off
  --help           Show this message and exit.
(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq --name kernel --limit 5
kernel-6.2.11: 0
kernel-6.2.14: 0
kernel-6.2.15: 0

How to Make Your CLI Discoverable with Trogon

Let's solve the problem of making your CLI discoverable with Trogon. Besides adding the new trogon library as part of the requirements (requirements.txt and setup.cfg), we need to add a new decorator to our CLI:

#!/usr/bin/env python
"""
A simple CLI to query the sizes of RPM on your system
Author: Jose Vicente Nunez
"""
import click
from trogon import tui

from reporter.rpm_query import QueryHelper

@tui()
@click.command()
@click.option('--limit', default=QueryHelper.MAX_NUMBER_OF_RESULTS,
              help="By default results are unlimited but you can cap the results")
@click.option('--name', help="You can filter by a package name.")
@click.option('--sort', default=True, help="Sorted results are enabled bu default, but you fan turn it off")
def command(
        name: str,
        limit: int,
        sort: bool
) -> None:
    with QueryHelper(
            name=name,
            limit=limit,
            sorted_val=sort
    ) as rpm_query:
        for package in rpm_query:
            click.echo(f"{package['name']}-{package['version']}: {package['size']:,.0f}")


if __name__ == "__main__":
    command()

Just one annotation, @tui, and a new import.

Time to see it in action:

(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_trogon.py --help
Usage: rpmq_trogon.py [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  command
  tui      Open Textual TUI.

Same results, but you'll notice two changes:

1. If you want to use the CLI options, you need to prepend 'command' before the switches.

2. There is a new tui command.

Wait a second...what happened with the other flags? No worries, if you ask for more help for 'command', you will see them there:

(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_trogon.py command --help
Usage: rpmq_trogon.py command [OPTIONS]

Options:
  --limit INTEGER  By default results are unlimited but you can cap the
                   results
  --name TEXT      You can filter by a package name.
  --sort BOOLEAN   Sorted results are enabled bu default, but you fan turn it
                   off
  --help           Show this message and exit.

Ah, much better. Let's run the CLI similar to the way we did before:

(CLIWithClickAndTrogon) [josevnz@dmaf5 CLIWithClickAndTrogon]$ rpmq_trogon.py command --limit 5 --name kernel
kernel-6.2.11: 0
kernel-6.2.14: 0
kernel-6.2.15: 0

And what about support for setuptools? Just add the import and the annotation to the 'command function':

import click
from trogon import tui

from reporter.rpm_query import QueryHelper
@tui()
@click.command()
@click.option('--limit', default=QueryHelper.MAX_NUMBER_OF_RESULTS,
              help="By default results are unlimited but you can cap the results")
@click.option('--name', help="You can filter by a package name.")
@click.option('--sort', default=True, help="Sorted results are enabled bu default, but you fan turn it off")
def command(
        name: str,
        limit: int,
        sort: bool
) -> None:
    # .... real code goes here
    pass

Allow me to demonstrate now with TUI mode how auto discoverable mode works:

Nice! We got a TUI where some options are automatically populated for us. This gives us a clear idea how to use the programs without knowing too much about them.

What's Next

1. Download the source code for this tutorial and start experimenting.

2. Both Click and Trogon have great documentation and online support. Take advantage of them.

3. Click has many more complex examples, feel free to check out their gallery.

In this article, I want to share with you my findings. The whole process was fun and gave me ideas on how to use these tools to tackle similar problems in the future.

What is NFS?

Network File System (NFS) is a distributed file system protocol that allows a user to access files over a computer network.

Please note that this is not a full tutorial on NFS. For that, please take a look at the following tutorial.

In this article, we will focus only on detecting access to a shared drive using several techiques as well setting up two servers and one client.

Also, I do use a different OS to set up both the server and the client, so instructions on how to do the task change a little bit.

How to Set Up a NFS Server and Client

My lab setup has one NFS server and two clients:

On my setup, I will have three computers talking to each other. One of them will be the NFS server and the other two will be a client.

How to Configure the Server

I will prepare my OrangePI machine to be the NFS server. Do so, I will enter the following commands:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install nfs-kernel-server -y
sudo systemctl enable nfs-kernel-server.service --now

Next step is to tell the server we want to share.

For that, we will edit the /etc/exports file (sudo vi /etc/exports history):

/data *(ro,all_squash,async,no_subtree_check)

Please check the man page to understand what these options mean.

In a nutshell, export /data:

• Is read-only

• Maps IDs to anonymous ID

• This option allows the NFS server to violate the NFS protocol and reply to requests before any changes made by that request have been committed to stable storage

• This option disables subtree checking. It's the default.

Now it is time to activate our shared directories:

root@orangepi5:~# sudo exportfs -a
root@orangepi5:~# sudo showmount -e
Export list for orangepi5:
/data (everyone)

I did something similar to the other host, raspberrypi:

root@raspberrypi:~# cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#        to NFS clients.  See exports(5).
#
/var/log/suricata *(ro,all_squash,async,no_subtree_check)
root@raspberrypi:~# showmount -e
Export list for raspberrypi:
/var/log/suricata *

How to Configure the Client

First thing is to confirm we can indeed see the shared mount points from our server:

(tutorials) [josevnz@dmaf5 SpyOnNfs]$ sudo showmount -e orangepi5
Export list for orangepi5:
/data raspberrypi,dmaf5

Data is shared with two machines – just what we expected.

Now, there are several ways to mount this drive. One of them is manually, another one is at startup, and the last one, my preferred one, is on demand.

How to Set Up the AutoMount Client on Fedora Linux

First we set the service:

sudo dnf install -y autofs
sudo systemct enable autofs.service --now

Then we set this up, so we end mounting remote /data into local /misc/data. For that, sdd the following line to your /etc/auto.master:

[root@dmaf5 ~]# vi /etc/auto.misc
# After editing the file, adding our entry to the last line of the file ...
[root@dmaf5 ~]# cat /etc/auto.misc
#
# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# Details may be found in the autofs(5) manpage

cd              -fstype=iso9660,ro,nosuid,nodev :/dev/cdrom

data            -ro,soft,rsize=16384,wsize=16384 orangepi5:/data
suricata        -ro,soft,rsize=16384,wsize=16384 raspberrypi:/var/log/suricata

Restart the service one more time:

[root@dmaf5 ~]# systemctl enable autofs.service --now

And the smoke test:

[root@dmaf5 ~]# ls -l /misc/data
total 0
drwxrwxr-x. 1 root 1001 48 Apr  7 17:57 nexus
[root@dmaf5 ~]# ls /misc/suricata
certs       eve.json.7  files            http.log    stats.log.1     suricata.log.2        suricata-start.log.3  tls.log.4
core        fast.log    http-data.log    http.log.1  stats.log.2     suricata.log.3        suricata-start.log.4  tls.log.5
eve.json    fast.log.1  http-data.log.1  http.log.2  stats.log.3     suricata.log.4        suricata-start.log.5  tls.log.6
eve.json.1  fast.log.2  http-data.log.2  http.log.3  stats.log.4     suricata.log.5        suricata-start.log.6  tls.log.7
eve.json.2  fast.log.3  http-data.log.3  http.log.4  stats.log.5     suricata.log.6        suricata-start.log.7
eve.json.3  fast.log.4  http-data.log.4  http.log.5  stats.log.6     suricata.log.7        tls.log
eve.json.4  fast.log.5  http-data.log.5  http.log.6  stats.log.7     suricata-start.log    tls.log.1
eve.json.5  fast.log.6  http-data.log.6  http.log.7  suricata.log    suricata-start.log.1  tls.log.2
eve.json.6  fast.log.7  http-data.log.7  stats.log   suricata.log.1  suricata-start.log.2  tls.log.3

Now we are ready to play with our service.

How to Create a Python Program that Reads Files into the NFS Server

For our example, we want to determine if a Python application is reading data from this directory. This script has two features:

• Performs a one time read view of a file. This will teach us how to capture this type of scenerarios, when a file is not opened all the time.

• And the script also follows updates on a file periodically.

Here is how our test script looks like in action:

./scripts/test_script.py \
--quick_read /misc/data/nexus/log/jvm.log \
--follow /misc/suricata/eve.json \
--verbose
...
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,889 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,890 <dependency_failed type='leaf_type' ctxk='java/io/FileOutputStream' witness='java/net/SocketOutputStream' stamp='66511.794'/>
2023-09-10 14:48:22,890 <dependency_failed type='unique_concrete_method' ctxk='java/io/ByteArrayOutputStream' x='java/io/ByteArrayOutputStream write ([BII)V' witness='sun/security/ssl/HandshakeOutStream' stamp='66511.855'/>
2023-09-10 14:48:22,890 <dependency_failed type='unique_concrete_method' ctxk='java/io/ByteArrayOutputStream' x='java/io/ByteArrayOutputStream write ([BII)V' witness='sun/security/ssl/HandshakeOutStream' stamp='66511.855'/>
...
# Ctrl-C to exit

The code, written in Python, is pretty simple:

#!/usr/bin/env python
"""
Simple script to simulate light activity on NFS drives
Author Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import concurrent
import os
import time
from concurrent.futures import ThreadPoolExecutor, ALL_COMPLETED
from pathlib import Path
from argparse import ArgumentParser
import logging

logging.basicConfig(format='%(asctime)s %(message)s', encoding='utf-8', level=logging.DEBUG)


def forever_read(the_file: Path, verbose: bool = False):
    for line in continuous_read(the_file=the_file):
        if verbose:
            logging.warning(line.strip())


def continuous_read(the_file: Path):
    """
    Continuously read the contents of file
    :param the_file:
    :return:
    """
    with open(the_file, 'r') as file_data:
        file_data.seek(0, os.SEEK_END)
        while True:
            line = file_data.readline()
            if not line:
                time.sleep(0.1)
                continue
            yield line


def quick_read(the_file: Path, verbose: bool = False):
    """
    Red the whole file and close it once done
    :param verbose:
    :param the_file:
    :return:
    """
    with open(the_file, 'r') as file_data:
        for line in file_data:
            if verbose:
                logging.warning(line.strip())


if __name__ == "__main__":
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        '--verbose',
        action='store_true',
        default=False,
        help='Enable verbose mode'
    )
    PARSER.add_argument(
        '--quick_read',
        type=Path,
        required=True,
        help='Read a file once'
    )
    PARSER.add_argument(
        '--follow',
        type=Path,
        required=True,
        help='Read a file continuously'
    )
    OPTIONS = PARSER.parse_args()
    try:
        with ThreadPoolExecutor(max_workers=3) as tpe:
            futures = [
                tpe.submit(forever_read, OPTIONS.follow, OPTIONS.verbose),
                tpe.submit(quick_read, OPTIONS.quick_read, OPTIONS.verbose)
            ]
            concurrent.futures.wait(futures, return_when=ALL_COMPLETED)
    except KeyboardInterrupt:
        pass

Now, let's go over how we can see if our script is indeed accessing an NFS partition.

Common steps

First we need to learn where to look for. So on the machine, check for NFS in /etc/fstab (for mount points that are available since the machine was rebooted):

[root@dmaf5 ~]# rg -e 'rsize=' /etc/fstab

Then on the AutoMount files:

[root@dmaf5 ~]# rg -e 'rsize=' /etc/auto*
/etc/auto.misc
17:data            -ro,soft,rsize=16384,wsize=16384 orangepi5:/data
18:suricata        -ro,soft,rsize=16384,wsize=16384 raspberrypi:/var/log/suricata

The regular expressions are not exact science, but you get the idea what to look for next.

How to Use the tools

We need to confirm if there was access to any of the following partitions mounted over NFS:

• /misc/data

• /misc/suricata

Next, I will show you a set of tools that will make the task easier, each one of them with their own strength and limitations.

Starting with lsof and ripgrep combined.

How to Use Lsof and rg for Capturing and Filtering

[josevnz@dmaf5 docs]$ lsof -w -b| rg -e '/misc/data|/misc/suricata'
python    36509                 josevnz    3   unknown                           /misc/suricata/eve.json
python    36509 36510 python    josevnz    3   unknown                           /misc/suricata/eve.json
python    36509 36511 python    josevnz    3   unknown                           /misc/suricata/eve.json

I passed the -b option to lsof to avoid it from getting stuck, in case the NFS handle is stale.

A few things about lsof:

• If you are using Autofs, you should know than mount points eventually get un-mounted to save bandwidth. This can be problematic when trying to catch the access of a file that is only opened once.

• The short-lived read didn't show up because the filehandle was closed after we inspected the process.

• If you want to monitor ALL the processes on this machine, you may need to run as root. You can only inspect your own processes without special privileges.

Still, lsof is a great tool to investigate.

Next strategy involves monitoring from the beginning, to catch the elusive short-read. We will use strace.

How to Use strace

sudo dnf install -y strace
(tutorials) [josevnz@dmaf5 SpyOnNfs]$ strace -f ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json 2>&1| rg -e '/misc/data|/misc/suricata'
execve("./scripts/test_script.py", ["./scripts/test_script.py", "--quick_read", "/misc/data/nexus/log/jvm.log", "--follow", "/misc/suricata/eve.json"], 0x7ffd9ae29738 /* 46 vars */) = 0
execve("/home/josevnz/virtualenv/tutorials/bin/python", ["python", "./scripts/test_script.py", "--quick_read", "/misc/data/nexus/log/jvm.log", "--follow", "/misc/suricata/eve.json"], 0x7ffe269dbf88 /* 46 vars */) = 0
[pid 38241] openat(AT_FDCWD, "/misc/suricata/eve.json", O_RDONLY|O_CLOEXEC <unfinished ...>
[pid 38242] openat(AT_FDCWD, "/misc/data/nexus/log/jvm.log", O_RDONLY|O_CLOEXEC <unfinished ...>

The openat(AT_FDCWD) entries give away the two files our script is reading from NFS. But as you can tell this approach has some caveats:

• We are filtering the output. It is best to save the output to a file with 'tee' and then search there

• It requires starting the process with strace from the beginning. Yes, you could do a 'strace -p $PID' to attach later to the process, but you risk missing short-lived reads

Is there a different way? Time to move on to the next tool, tshark and see how to use a network capture to confirm access to the share.

How to Use tshark

We can also capture the network traffic and filter out only NFS. It is not perfect, but it may be sufficient.

First, find out which network interface is used to communicate with the NFS server. In my case it is easy – they all connected using a wired private network:

[josevnz@dmaf5 docs]$ ip --oneline address|rg -e 'eno|wlp'
3: eno1    inet 192.168.68.70/22 brd 192.168.71.255 scope global dynamic noprefixroute eno1\       valid_lft 4568sec preferred_lft 4568sec
4: wlp4s0    inet 192.168.1.95/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp4s0\       valid_lft 3423sec preferred_lft 3423sec
4: wlp4s0    inet6 fe80::ac40:5365:7f09:a5d2/64 scope link noprefixroute \       valid_lft forever preferred_lft forever

For this example it is eno1 with IP address '192.168.68.70'. Then capture the traffic, and with some luck we will get the file path:

[root@dmaf5 ~]# tshark -i eno1 -Y "nfs"
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eno1'
 ** (tshark:42326) 16:02:47.417145 [Main MESSAGE] -- Capture started.
 ** (tshark:42326) 16:02:47.417286 [Main MESSAGE] -- File: "/var/tmp/wireshark_eno1rEGxiu.pcapng"
   13 1.601197994 192.168.68.70 → 192.168.68.60 NFS 450 V4 Call GETATTR FH: 0x90ba4ee1  ; V4 Call GETATTR FH: 0x90ba4ee1
   14 1.601374466 192.168.68.70 → 192.168.68.60 NFS 258 V4 Call GETATTR FH: 0x90ba4ee1
   15 1.601395155 192.168.68.70 → 192.168.68.60 NFS 258 V4 Call GETATTR FH: 0x90ba4ee1
   16 1.602155254 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 13) GETATTR
   17 1.602368826 192.168.68.60 → 192.168.68.70 NFS 554 V4 Reply (Call In 13) GETATTR  ; V4 Reply (Call In 14) GETATTR
   19 1.602515091 192.168.68.70 → 192.168.68.60 NFS 274 V4 Call READ StateID: 0xa902 Offset: 57552896 Len: 12288
   20 1.602557170 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 15) GETATTR
   22 1.603156327 192.168.68.60 → 192.168.68.70 NFS 1730 V4 Reply (Call In 19) READ
   66 4.611124808 192.168.68.70 → 192.168.68.60 NFS 642 V4 Call GETATTR FH: 0x90ba4ee1  ; V4 Call GETATTR FH: 0x90ba4ee1  ; V4 Call GETATTR FH: 0x90ba4ee1
   67 4.611301059 192.168.68.70 → 192.168.68.60 NFS 258 V4 Call GETATTR FH: 0x90ba4ee1
   68 4.611809385 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 66) GETATTR
   69 4.611887552 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 66) GETATTR
   71 4.611976479 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 66) GETATTR
   72 4.620685968 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 67) GETATTR
   74 5.017200005 192.168.68.70 → 192.168.68.60 NFS 250 V4 Call GETATTR FH: 0x9419c00c
   75 5.017804843 192.168.68.70 → 192.168.68.59 NFS 242 V4 Call GETATTR FH: 0x314e720f
   76 5.017838787 192.168.68.60 → 192.168.68.70 NFS 310 V4 Reply (Call In 74) GETATTR
   77 5.018131217 192.168.68.70 → 192.168.68.60 NFS 326 V4 Call OPEN DH: 0x90ba4ee1/
   78 5.018711408 192.168.68.60 → 192.168.68.70 NFS 386 V4 Reply (Call In 77) OPEN StateID: 0x9984
   79 5.018855699 192.168.68.59 → 192.168.68.70 NFS 310 V4 Reply (Call In 75) GETATTR
   81 5.018980434 192.168.68.70 → 192.168.68.59 NFS 262 V4 Call GETATTR FH: 0xecd332cc
   82 5.019934959 192.168.68.59 → 192.168.68.70 NFS 310 V4 Reply (Call In 81) GETATTR
   83 5.020032853 192.168.68.70 → 192.168.68.59 NFS 262 V4 Call GETATTR FH: 0x261d4440
   84 5.020734032 192.168.68.59 → 192.168.68.70 NFS 310 V4 Reply (Call In 83) GETATTR
   85 5.020874175 192.168.68.70 → 192.168.68.59 NFS 330 V4 Call OPEN DH: 0xc9b4831b/

This is great, there is activity against two NFS servers, 192.168.68.59 and 192.168.68.60. But, is there a way to see the name of files?

tshark has a way to spit information by field. The problem is that NFS has lots of them:

[root@dmaf5 ~]# for field in $(tshark -G fields| cut -d'        ' -f3|rg -e '^nfs\.'); do echo "-e $field"; done|head -n 10
Running as user "root" and group "root". This could be dangerous.
-e nfs.unknown
-e nfs.svr4
-e nfs.knfsd_le
-e nfs.nfsd_le
-e nfs.knfsd_new
-e nfs.ontap_v3
-e nfs.ontap_v4
-e nfs.ontap_gx_v3
-e nfs.celerra_vnx
-e nfs.gluster

So, let's capture them into a variable (also need to enable some options):

[root@dmaf5 ~]# fields=$(for field in $(tshark -G fields| cut -d'       ' -f3|rg -e '^nfs\.'); do echo "-e $field"; done)
[root@dmaf5 ~]# tshark -i eno1 --enable-protocol nfs -o nfs.file_name_snooping:true -o nfs.file_full_name_snooping:true -T fields -E header=y -E separator=, -E quote=d $fields
Running as user "root" and group "root". This could be dangerous.
nfs.unknown,nfs.svr4,nfs.knfsd_le,nfs.nfsd_le,nfs.knfsd_new,nfs.ontap_v3,nfs.ontap_v4,nfs.ontap_gx_v3,n...

I managed to get the filename only once, then after interrupting and restarting the program I got no luck.

And yet no sign of the file name. The file handle was in the contents but this is not very useful if you want a quick way to see what was accessed.

Is there an easier way to do this? Sysdig may offer some answers.

How to Use Sysdig

While trying to find the elusive mount points, I stumbled into Sysdig:

Sysdig instruments your physical and virtual machines at the OS level by installing into the Linux kernel and capturing system calls and other OS events. Sysdig uses DTrace to get access to the system kernel.

Sysdig also makes it possible to create trace files for system activity, similarly to what you can do for networks with tools like tcpdump and Wireshark.

I decided to use the latest version (0.33.1) for Fedora 37 where my script is running):

sudo dnf install -y https://github.com/draios/sysdig/releases/download/0.33.1/sysdig-0.33.1-x86_64.rpm
# Wait a little bit, as a kernel module needs to be compiled and prepared...
Installed:
  bison-3.8.2-3.fc37.x86_64                    dkms-3.0.11-1.fc37.noarch          elfutils-libelf-devel-0.189-3.fc37.x86_64  flex-2.6.4-11.fc37.x86_64            kernel-devel-6.4.13-100.fc37.x86_64 
  kernel-devel-matched-6.4.13-100.fc37.x86_64  libzstd-devel-1.5.5-1.fc37.x86_64  m4-1.4.19-4.fc37.x86_64                    openssl-devel-1:3.0.9-1.fc37.x86_64  sysdig-0.33.1-1.x86_64              
  zlib-devel-1.2.12-5.fc37.x86_64

How easy is to probe out the script so it is indeed accessing the NFS mounted directories? Let's print three fields of interest and the name of the accesed file:

# `sysdig -l` will output every single field you can capture
[root@dmaf5 ~]# sysdig -p"%proc.cmdline,%fd.name" proc.name contains python and fd.name contains /misc
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
...

What if you want to capture all the data, and filter later? One way to do it is capturing to a file:

# Capture for one minute...
[root@dmaf5 ~]# timeout --preserve-status 1m sysdig -w /tmp/sysdig.dump
[root@dmaf5 ~]# ls -lh /tmp/sysdig.dump
-rw-r--r--. 1 root root 32M Sep 10 19:03 /tmp/sysdig.dump

And then replay the contents, with filtering (replay doesn't need elevated privileges):

[root@dmaf5 ~]# sysdig -r /tmp/sysdig.dump -p"%proc.cmdline,%fd.name" proc.name contains python and fd.name contains /misc|sort -u
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/data/nexus/log/jvm.log
python ./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose,/misc/suricata/eve.json

Sysdig supports scripting, using the LUA language. For example, it has a very convenient version of lsof:

[root@dmaf5 ~]# sysdig -cl|rg lsof
lsof            List (and optionally filter) the open file descriptors.

So let's use it:

[root@dmaf5 ~]# sysdig -c lsof|rg misc
automount           52410   52410   root    8       directory   /misc
automount           52410   52413   root    8       directory   /misc
automount           52410   52414   root    8       directory   /misc
automount           52410   52415   root    8       directory   /misc
automount           52410   52418   root    8       directory   /misc
automount           52410   52421   root    8       directory   /misc
python              75840   75840   josevnz 3       file        /misc/suricata/eve.json
python              75840   75841   josevnz 3       file        /misc/suricata/eve.json
python              75840   75842   josevnz 3       file        /misc/suricata/eve.json

What I liked about this tool:

• Can work with older kernels (like 4.xx)

• Has a powerful expression language for filtering

• Easy to learn and well documented

• You can write your own scripts if you know LUA

Before finishing up let's look at one more tool, BPF.

How to Use BPF probe

Originally Berkeley Packet Filter, is a kernel and user-space observability scheme for Linux.

The BPF is a very powerful tool, and this short article won't even scratch the surface.

Yes, this is huge. I'm learning this myself.

I found that the bcc repository has lots of ready to use scripts that we could use to track our NFS access, and even check for performance (you can find more examples here, and on the BPF Performance Book repository).

But it is more interesting to write tools yourself that monitor pretty much anything you want. For this tutorial, I will use some ready to use programs that use the traces to capture useful information.

As a first step, we will need to install a high level interpreter for our scripts. Again, on my Fedora Linux machine:

[josevnz@dmaf5 ~]$ sudo dnf install -y bpftrace.x86_64 bcc-tools.x86_64
# And check if the kernel has btf enabled
[josevnz@dmaf5 ~]$ ls -la /sys/kernel/btf/vmlinux
-r--r--r--. 1 root root 5635179 Sep 12 04:21 /sys/kernel/btf/vmlinux

On a separate terminal run again the NFS test script:

. ~/virtualenv/tutorials/bin/activate
cd SpyOnNfs/
./scripts/test_script.py --quick_read /misc/data/nexus/log/jvm.log --follow /misc/suricata/eve.json --verbose

You can trace all the files opened by a program, like top:

18:59:20 loadavg: 1.20 1.00 0.74 1/1175 28520

TID     COMM             READS  WRITES R_Kb    W_Kb    T FILE
28520   clear            2      0      60      0       R xterm-256color
28203   python           7      0      56      0       R eve.json
28347   filetop          2      0      15      0       R loadavg
824     systemd-oomd     2      0      8       0       R memory.swap.current
824     systemd-oomd     2      0      8       0       R memory.low
...

But it doesn't print the full path. It's more useful to ask a NFS snoop and see if one of our files shows up:

[josevnz@dmaf5 SuricataLog]$ sudo /usr/share/bcc/tools/nfsslower 1
# Commented out some warnings ...
Tracing NFS operations that are slower than 1 ms... Ctrl-C to quit
TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
19:02:25 python         28202  R 1460    62150       1.96 eve.json
19:02:28 python         28202  R 2446    62151       2.09 eve.json
19:02:31 python         28202  R 970     62154       1.99 eve.json
19:02:34 python         28202  R 3335    62155       2.43 eve.json
19:02:37 python         28202  R 4564    62158       1.84 eve.json
19:02:40 python         28202  R 5876    62162       1.89 eve.json
19:02:43 python         28202  R 4504    62168       1.61 eve.json
19:02:46 python         28202  R 3131    62173       1.92 eve.json

This is much better. Also, we can see than the latency is almost two milliseconds.

We can also monitor mount/ umount operations:

[josevnz@dmaf5 SuricataLog]$ sudo /usr/share/bcc/tools/mountsnoop 
# Commented out some warnings ...
2 warnings generated.
COMM             PID     TID     MNT_NS      CALL
mount.nfs        29012   29012   4026531841  mount("orangepi5:/data", "/misc/data", "nfs", MS_RDONLY, "sloppy,soft,rsize=16384,wsize=16384,vers=4.2,addr=192.168.68.59,clientaddr=192.168.68.68") = 0

This is good as well, we can see the activity over NFS we wanted to confirm.

Next Steps

You learned several tools and as you may have guessed, you can use them to snoop on more than just opened files on NFS.

It is always useful to know more than one tool. Sysdig has a special mention for being very versatile, powerful and yet easy to use. Also, it can be extended with scripts written in the LUA language.

BPF is another alternative and will give you incredible access to the kernel calls. Be prepared to spend time reading and learning how to use the tools.

The code for the scripts used on this tutorial can be obtained from my GitHub repository: SpyOnNfs.

Why have an on-premise artifact manager? There are many reasons for it:

• Use your private infrastructure: You may have proprietary code that needs to be safeguarded.

• Faster artifact download speeds: If you constantly download the same artifacts over the Internet, you can cache them on a central location, for the benefit of your multiple users across multiple servers by caching them.

• Control what artifacts make it to your build chain: Centralize the location of the artifacts, ensure they are approved for usage, and also confirm than they do not contain malicious code.

• Segregate who can have access to your artifacts: You may have more strict requirements on who can access some artifacts within your own organization.

In this article I will show you how you can download, install, and configure the OSS version of Nexus 3 using an Ansible playbook.

Nexus 3 will run on an Orange PI 5 computer with 8 GB or RAM, but this provisioning can be done on any machine with the minimum requirements. Part of the setup will consist of setting a proxy for PyPI.org, for the machines listed on my inventory file.

What you need to run the code from this tutorial

1. An Internet connection to download the source code for the Ansible playbook, Nexus, and PIP modules

2. Two or more Linux machines (I used Debian, Armbian and Fedora IOT), with at least 8 GB of RAM. My cluster has a mix of Raspberry PI 4 and an OrangePI 5.

3. Ansible controller will run on the Fedora machine, but any server can be the controller. Installation instructions for Ansible are easy to follow.

Playbook Organization

I divided the tasks in groups and the resulting playbook looks like this:

[josevnz@dmaf5 Nexus3OnOrangePI]$ tree -N ansible/
ansible/
├── inventories
│   └── home
│       └── hosts.yaml
├── roles
│   ├── clients
│   │   ├── tasks
│   │   │   └── main.yaml
│   │   └── templates
│   │       └── pip.conf.j2
│   └── nexus
│       ├── files
│       │   └── swagger.json
│       ├── tasks
│       │   ├── download.yaml
│       │   ├── install.yaml
│       │   ├── main.yaml
│       │   ├── post_install.yaml
│       │   ├── pre_install.yaml
│       │   ├── repositories.yaml
│       │   ├── third_party.yaml
│       │   └── user.yaml
│       └── templates
│           ├── logrotate.nexus3.j2
│           ├── nexus3.service.j2
│           ├── nexus.rc.j2
│           └── nexus.vmoptions.j2
├── site.yaml
├── vars
│   ├── clients.yaml
│   └── nexus.yaml
└── vault
    ├── nexus_password.enc
    └── README.md

13 directories, 21 files

Now a little bit of explaining:

• There are two roles: ‘nexus’ and ‘clients’. The nexus role is used to setup the artifact management software, while the client role sets up the pip settings on every machine.

• Vars contains variables used on each role, separated by files to make their usage more clear

• We have passwords, and we managed them using Ansible vault feature.

• The file ‘site.yaml’ Orchestrates the role execution:

- hosts: all
  tags: clients
  vars_files:
    - vars/clients.yaml
  roles:
    - clients
- hosts: nexus_server
  tags: nexus
  become_user: root
  become: true
  vars_files:
    - vars/nexus.yaml
  roles:
    - nexus

Now let’s move on to see the universe where the playbook will be executed.

The Host Inventory

In my case it is quite simple – I have two main groups: ‘clients’ and the machine where the Nexus 3 server itself will run:

all:
  children:
    nexus_server:
      hosts:
        orangepi5.home:
    home_lab:
      hosts:
        dmaf5.home:
        raspberrypi.home:
        orangepi5.home:

The next important task is to download and configure Nexus 3.

How to Install Nexus 3

The file main.yaml describes the order and purpose of each installation task for the Nexus role:

# Tasks listed here are related to the remote Nexus 3 server
# Included tasks are called in order
---
  - include_tasks: third_party.yaml
  - include_tasks: pre_install.yaml
  - include_tasks: download.yaml
  - include_tasks: install.yaml
  - include_tasks: post_install.yaml
  - include_tasks: user.yaml
  - include_tasks: repositories.yaml

Let’s see first what I like to call the “core tasks”:

1. third_party.yaml: In here we install the OpenJDK8 (Nexus 3 is written in Java) and logrotate to take care of the stale logs.

2. pre_install.yaml: A lot happens here, like creating required directories for nexus, dedicated non-privileged user that will run the process.

3. download.yaml: As the name says, we get a fresh version of the Nexus 3 OSS software and make sure it has the right checksum. We don’t want to install malware from the Internet.

Then come the tasks that fall into the “customized installation group”:

1. install.yaml: Unpack the software, prepare the systemd unit to start it automatically, setup JVM settings for Nexus, and deploy the logrotate configuration.

2. post_install.yaml: Exciting stuff happens here – the software is installed, and we run it for the first time. We also change the default password using the REST API, so we can move to the customization stage.

3. user.yaml: Here we prepare to provide our end users with proper access to the services offered by Nexus. We do this using a combination of the REST-API and Ansible client code:

# https://help.sonatype.com/repomanager3/installation-and-upgrades/post-install-checklist
# https://help.sonatype.com/repomanager3/integrations/rest-and-integration-api
---
- name: Enable anonymous user
  tags: anonymous
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/security/anonymous"
    method: PUT
    body_format: raw
    status_code: [ 200, 202, 204 ]
    headers:
      Content-Type: application/json
    body: |-
      { "enabled" : true, "userId" : "anonymous", "realmName" : "NexusAuthorizingRealm" }
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true
- name: Enable Docker security realm
  tags: docker_realm
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/security/realms/active"
    method: PUT
    body_format: raw
    status_code: [ 200, 202, 204 ]
    headers:
      Content-Type: application/json
    body: |-
      [ "NexusAuthenticatingRealm", "NexusAuthorizingRealm", "DockerToken" ]
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true

The logic is easy to follow, by using the ‘PUT’ http method you can tell is a modification operation (meaning existing roles and users already exist). Error detection is done by getting the HTTP codes returned by Nexus.

Next step is to prepare our local PyPi proxy. This is a multistep task and will be described in detail next.

How to Set Up PyPI Proxy on Nexus 3

The last file on the Nexus 3 role is ‘repositories.yaml’. In here we go through the following steps:

1. Check if the proxy was already setup (GET or read only operation)

2. If it doesn’t exist, create a new one (POST method with JSON payload with details to create whole new repository)

Notice than this playbook doesn’t offer the option to update repository settings. It is possible to do with the REST API, but I will leave that as an exercise to the reader.

The tasks to prepare the PyPi proxy are shown below:

# Create proxy for repositories
# https://help.sonatype.com/repomanager3/integrations/rest-and-integration-api
# PyPi: https://pip.pypa.io/en/stable/user_guide/
---
- name: Check if the PyPi proxy exists
  tags: pypi_proxy_exists
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/repositories/pypi/proxy/python_proxy"
    method: GET
    body_format: raw
    status_code: [ 200, 202, 204, 404 ]
    headers:
      Content-Type: application/json
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true
  register: python_local
- name: Create PyPI proxy
  tags: pypi_proxy_create
  ansible.builtin.uri:
    user: ""
    password: ""
    url: "/v1/repositories/pypi/proxy"
    method: POST
    body_format: raw
    status_code: [ 201 ]
    headers:
      Content-Type: application/json
    body: |-
      {
        "name": "python_proxy",
        "online": true,
        "storage": {
          "blobStoreName": "default",
          "strictContentTypeValidation": true
        },
        "proxy": {
          "remoteUrl": "https://pypi.org/",
          "contentMaxAge": -1,
          "metadataMaxAge": 1440
        },
        "negativeCache": {
          "enabled": true,
          "timeToLive": 1440
        },
        "httpClient": {
          "blocked": false,
          "autoBlock": true,
          "connection": {
            "retries": 0,
            "timeout": 60,
            "enableCircularRedirects": false,
            "enableCookies": true,
            "useTrustStore": false
          }
        }
      }
    force_basic_auth: true
    return_content: true
  any_errors_fatal: true
  when: python_local.status == 404

We are almost there. Now we need to tell our PyPi clients than we should use our local Nexus and not the direct PyPi site to get our Python libraries.

How to Set the Clients

The clients role is much simpler and only requires deploying a template for pip.conf with enough information to force the search on our new repository:

# Tasks here are meant to be used on our clients user
---
- name: Create installation directory for pip.conf
  tags: pip_basedir
  ansible.builtin.file:
    state: directory
    path: ""
    owner: ""
    group: ""
    mode: "u+rwx,go-rwx"
- name: Copy pip.conf file
  tags: pip_copy
  ansible.builtin.template:
    src: pip.conf.j2
    dest: "/pip.conf"
    owner: ""
    group: ""
    mode: u=rxw,g=r,o=r

The resulting file gets deployed on ‘~/.config/pip/pip.conf’ of every machine:

# https://pip.pypa.io/en/stable/topics/configuration/
[global]
timeout = 60
[install]
index = http://orangepi5.home:8081/repository/python_proxy/pypi
index-url = http://orangepi5.home:8081/repository/python_proxy/simple/
trusted-host = orangepi5.home

The file above shows an example of how the final version of the file will look once deployed on my cluster (yours will be different with the resolved URL).

It is time now to run the whole playbook and see what it looks like.

How to Run the Playbook

To run the playbook, we pass a few arguments:

1. The location of our host inventory

2. The location of the encrypted password file and a master file containing the master password to unlock the contents of the protected file

3. And finally the location of our main playbook file

cd ansible
ansible-playbook --inventory  inventories --extra-vars @vault/nexus_password.enc --vault-password-file $HOME/vault/ansible_vault_pass site.yaml

How to test the new PyPI proxy

To test our new proxy, we will install Python Rich using pip and a virtual environment.

josevnz@orangepi5:~$ python3 -m venv ~/virtualenv/rich
(rich) josevnz@orangepi5:~$ . ~/virtualenv/rich/bin/activate
(rich) josevnz@orangepi5:~$ pip install rich
Looking in indexes: http://orangepi5.home:8081/repository/python_proxy/simple/
Collecting rich
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/rich/13.3.4/rich-13.3.4-py3-none-any.whl (238 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 238.7/238.7 KB 14.8 MB/s eta 0:00:00
Collecting pygments<3.0.0,>=2.13.0
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/pygments/2.15.0/Pygments-2.15.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 23.8 MB/s eta 0:00:00
Collecting markdown-it-py<3.0.0,>=2.2.0
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/markdown-it-py/2.2.0/markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.5/84.5 KB 6.9 MB/s eta 0:00:00
Collecting mdurl~=0.1
  Downloading http://orangepi5.home:8081/repository/python_proxy/packages/mdurl/0.1.2/mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: pygments, mdurl, markdown-it-py, rich
Successfully installed markdown-it-py-2.2.0 mdurl-0.1.2 pygments-2.15.0 rich-13.3.4

And then we can confirm than the cache was indeed used by seeing the new artifacts on the new repository:

See the PyPi artifacts

Let’s see a demo of the client in action, installing something else:

Further Customization Using the REST-API

Every Nexus installation allows you to download a JSON file that describes the API supported by the server. For example, in my server you can get a copy like this from my orangepi5.home server:

curl --fail --remote-name http://orangepi5.home:8081/service/rest/swagger.json

Also, the UI allows you to try the other REST API endpoints to customize your installation.

REST API testing

Conclusion

I recommend spending some time and reading the Nexus 3 book to get yourself familiar with the features this tool can offer.

The community prepared Debian and RPM installers, if you need this kind of setup as opposed to using Ansible.

Nexus 3 has lots of configurable settings. We covered only the surface here. While preparing this article I found 'ThoTeam Nexus3-oss repository' with a very complete and up-to-date playbook, but it was way more complex than anything I required for my home lab.

Archiva is another Open Source artifact manager, it is more limited in functionality but also simpler to setup.

There is a post-installation checklist with some tasks I did not need to complete for my home lab. Please check it out to make sure your setup is complete.

What if it's not just one server but many? The amount of time you will need to fix all of them will grow exponentially – and because is a manual process it will be more error-prone.

And then the nightmare scenario: You don't have an up-to-date backup, or you have incomplete backups. Or the worst – there are no backups at all. This last case is more common than you think, especially in home labs where you are tinkering and playing around with stuff by yourself.

In this tutorial, I'll show you how you can do a full infrastructure provisioning of a pair of web servers on a Cloud provider, with SSL certificates and monitoring metrics with Prometheus.

What You Need for This Setup

The first thing you need is a cloud provider. Oracle Cloud offers a Free Tier version of their cloud services, which allows you to setup virtual machines for free. This is great for a home lab with lots of rich features that you can use to try new tools and techniques.

You'll also need an automation tool. I used Ansible because its doesn't have many requirements (you only need an SSH daemon and public key authentication to get things going). I also like it because it works equally well regardless of the cloud environment you are trying to provision.

In this tutorial we will use the Open Source version of this tool, as it is more than sufficient for our purposes.

What's included in the Ansible playbook

An Ansible playbook is nothing more than a set of instructions you define to execute tasks that will change the status of a host. These actions are carried out on an inventory of hosts you define.

Here, you are going to learn about the following:

• How to clean inventory sources by using the proper layout in your playbooks.

• How to provision two NGINX instances, with the request of their proper free SSL certificates using Certbot.

• How to set up the local Linux firewalls and add a Prometheus node_exporter agent and one scraper to collect that data.

• Concepts like variables, roles (with task inclusion), and conditional execution.

• Important techniques like task tagging, debug messages, and static validation with ansible-lint.

All the code can be found in this GitHub repository.

What You Should Know Before Trying This

Because we will cover several tasks here, you will probably need to be familiar with several things (I'll provide links as we go along):

• This is not an introductory course on Ansible but more of a "how all things fit together" with a more detailed, but not too complex, playbook.

• An OCI Cloud Free Tier account

• Privileged account, most likely SUDO

• Basic knowledge of TCP/IP and firewalls with firewalld

• How to use RPM and how to package applications (we will not do that here, but it helps to understand when an RPM is better than a complex task in Ansible)

What is not included here

OCI Cloud has a complete REST API to manage a lot of aspects of their cloud environment. Their setup page (specifically the SDK) is also very detailed.

You'll Probably Do Things Differently in Production.

Installing the OCI-Metrics-datasource instead of Prometheus agents on a virtual machine

You can go to this page to install it on your Grafana instance (Bare metal or Cloud). Also you need to setup your credentials and permissions as explained here.

This is probably the most efficient way to monitor your resources as you do not need to run agents on your virtual machines. But I will install instead a Prometheus node_exporter agent and scraper that will be visible from a Grafana Cloud instance.

An exposed Prometheus on the Internet endpoint is not a good idea

It is very clear, I'm exposing my Prometheus scraper to the Internet so Grafana cloud can reach it. On an Intranet with a private cloud and your local Grafana, this is not an issue – but here, a Prometheus agent pushing data to Grafana would be a better option.

Still, Grafana provides a list of public IP addresses that you can use to setup your allow list.

So the following will work:

Oracle Cloud Ingress Rules

But it is not the best. Instead, you want to restrict the specific IP addresses that can pull data from your exposed services. The prometheus exporter can be completely hidden from Grafana on port 9100. Instead we only need to expose the Prometheus scraper that listens on port 9000.

For this home lab, it is not a big deal having such services fully exposed. But if you have a server with sensitive data, you must restrict who can reach the service!

An alternative to the Prometheus endpoint is to push the data to Grafana by using a Grafana agent but I will not cover that option here.

Playbook Analysis

Ansible lets you have a single file with the playbook instructions, but eventually you will find that such a structure is difficult to maintain.

For my playbook I decided to keep the suggested structure:

tree -A 
.
├── inventory
│   └── cloud.yaml
├── oracle.yaml
├── roles
│   └── oracle
│       ├── files
│       │   ├── logrotate_prometheus-node-exporter
│       │   ├── prometheus-node-exporter
│       │   └── requirements_certboot.txt
│       ├── handlers
│       │   └── main.yaml
│       ├── meta
│       ├── tasks
│       │   ├── controller.yaml
│       │   ├── main.yaml
│       │   ├── metrics.yaml
│       │   └── nginx.yaml
│       ├── templates
│       │   ├── prometheus-node-exporter.service
│       │   ├── prometheus.service
│       │   └── prometheus.yaml
│       └── vars
│           └── main.yaml
└── site.yaml

Below is a brief description of how the content is organized:

1. You can have more than one site. You control that inside the [site.yaml](file:///home/josevnz/OracleCloudHomeLab/site.yaml) file.

2. The host list is inside the inventory directory. You can have more than one inventory file or scripts to generate the hostlist, or a combination of both.

3. The roles/oracle group the tasks. We only have one role called 'oracle' because that's the cloud provider I'm focusing on here.

4. Our playbook uses metadata in the form of variables, with each one defined on the 'vars' directory. That way we can customize the behaviour of the playbook in multiple places:

---
# Common variables for my Oracle Cloud environments
controller_host: XXXX.com
ssl_maintainer_email: YYYYYY@ZZZZ.com
architecture: arm64
prometheus_version: 2.38.0
prometheus_port: 9090
prometheus_node_exporter_nodes: "['X-server1:{{ node_exporter_port }}', 'Y-server2:{{ node_exporter_port }}' ]"
node_exporter_version: 1.4.0
node_exporter_port: 9100
internal_network: QQ.0.0.0/24

The roles/oracle files directory contains files that can be copied as is to the remote directory. The templates' directory is similar, but the files in there can be customized for each host by using the Jinja templating language.

# A template for the prometheus scraper configuration file
---
global:
    scrape_interval: 30s
    evaluation_interval: 30s
    scrape_timeout: 10s
    external_labels:
        monitor: 'oracle-cloud-metrics'

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: {{ prometheus_node_exporter_nodes }}
    tls_config:
      insecure_skip_verify: true

The 'tasks' directory is where we store our tasks, that is the actions that will modify the server state. Note that Ansible will not execute tasks if it's not necessary. The idea is that you can re-run a playbook as many times as needed and the final state will be the same.

# Fragment of the nginx tasks file. See how we notify a handler to restart nginx after the SSL certificate is renewed.
---
- name: Copy requirements file
  ansible.builtin.copy:
    src: requirements_certboot.txt
    dest: /opt/requirements_certboot.txt
  tags: certbot_requirements

- name: Setup Certbot
  pip:
    requirements: /opt/requirements_certboot.txt
    virtualenv: /opt/certbot/
    virtualenv_site_packages: true
    virtualenv_command: /usr/bin/python3 -m venv
  tags: certbot_env

- name: Get SSL certificate
  command:
    argv:
      - /opt/certbot/bin/certbot
      - --nginx
      - --agree-tos
      - -m {{ ssl_maintainer_email }}
      - -d {{ inventory_hostname }}
      - --non-interactive
  notify:
    - Restart Nginx
  tags: certbot_install

There is one special directory called 'handlers'. There we define actions that must happen if a task changes the state of our host.

We now have a picture of how all the pieces work together, so let's talk about some specific details.

Firewall provisioning

With Ansible, you can replace a sequence of commands like this:

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

With a firewalld module:

---
- name: Enable HTTP at the Linux firewall
  firewalld:
    zone: public
    service: http
    permanent: true
    state: enabled
    immediate: yes
  notify:
    - Reload firewall
  tags: firewalld_https

- name: Enable HTTPS at the Linux firewall
  firewalld:
    zone: public
    service: https
    permanent: true
    state: enabled
    immediate: yes
  notify:
    - Reload firewall
  tags: firewalld_https

Common tasks have nice replacements

So instead of running SUDO with a privileged command:

sudo dnf install -y nginx
sudo systemctl enable nginx.service --now

You can have something like this:

# oracle.yaml file, which tells which roles to call, included from site.yaml
---
- hosts: oracle
  serial: 2
  remote_user: opc
  become: true
  become_user: root
  roles:
  - oracle
# NGINX task (roles/oracle/tasks/nginx.yaml)
- name: Ensure nginx is at the latest version
  dnf:
    name: nginx >= 1.14.1
    state: present
    update_cache: true
  tags: install_nginx
# And a handler that will restart NGINX after it gets modified (handlers/main.yaml)
---
- name: Restart Nginx
  ansible.builtin.service:
    name: nginx
    state: restarted
- name: Reload firewall
  ansible.builtin.systemd:
    name: firewalld.service
    state: reloaded

How to Run the Playbooks

Normally you don't wait to have the whole playbook written, but you run the pieces you need in the proper order. At some point you will have your whole playbook finished and ready to go.

Make sure the playbook behaves properly with --check before making any changes

The very first step is to check your playbook file for errors. For that you can use yamllint:

yamllint roles/oracle/tasks/main.yaml

But doing this for every yaml file in your playbook can be tedious an error-prone. As an alternative, you can run the playbook in a 'dry-run' mode, to see what will happen without actually making any changes:

Another way to gradually test a complex playbook is by executing a specific task by using a tag or group of tags. That way you can do controlled execution of your playbook:

Keep in mind that this will not execute any dependencies that you may have defined on you playbook, tough:

Use Ansible-lint when ansible-playbook --check is not good enough

Some errors are more subtle and will not get caught with ansible-playbook --check. To get a more complete check on your playbooks before minor issues become a headache you can use ansible-lint. So let's get it installed:

python3 -m venv ~/virtualenv/ansiblelint && . ~/virtualenv/ansiblelint/bin/activate
pip install --upgrade pip
pip install --upgrade wheel
pip install ansible-lint

Now we can check the playbook:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml 
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 1 violation(s) that are fatal
syntax-check[specific]: couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path.
roles/oracle/tasks/nginx.yaml:2:3

Strange, firewalld is available on our Ansible installation. What else was installed by ansible-lint?

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible --version
ansible [core 2.14.0]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/josevnz/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/josevnz/virtualenv/ansiblelint/lib64/python3.9/site-packages/ansible
  ansible collection location = /home/josevnz/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/josevnz/virtualenv/ansiblelint/bin/ansible
  python version = 3.9.9 (main, Nov 19 2021, 00:00:00) [GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] (/home/josevnz/virtualenv/ansiblelint/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Ansible-lint installed its own ansible [core], and firewalld is part of ansible.posix collection. We will use Ansible Galaxy to install it:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ which ansible-galaxy
~/virtualenv/ansiblelint/bin/ansible-galaxy
(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-galaxy collection install ansible.posix
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/download/ansible-posix-1.4.0.tar.gz to /home/josevnz/.ansible/tmp/ansible-local-18099xpw_8usc/tmp8msc9uf5/ansible-posix-1.4.0-_f17f525
Installing 'ansible.posix:1.4.0' to '/home/josevnz/.ansible/collections/ansible_collections/ansible/posix'
ansible.posix:1.4.0 was installed successfully

Running it again:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml 
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 50 violation(s) that are fatal
name[play]: All plays should be named. (warning)
oracle.yaml:2

fqcn[action-core]: Use FQCN for builtin module actions (service).
roles/oracle/handlers/main.yaml:2 Use `ansible.builtin.service` or `ansible.legacy.service` instead.

fqcn[action-core]: Use FQCN for builtin module actions (command).
roles/oracle/handlers/main.yaml:6 Use `ansible.builtin.command` or `ansible.legacy.command` instead.

Some warnings are pedantic ('Use FQCN for builtin module actions (command)') and others require attention (Commands should not change things if nothing needs doing.).

Ansible-lint found many smells on the playbook, there is one option to re-write the files and correct some of these errors automatically:

There are some guidelines you can follow to correct these issues. Below are some that can be directly applied to the warnings we got earlier:

Note that all the errors are easy to solve. Some commands decide on their own if they should make changes or not but have a hard time communicating back to Ansible:

- name: Get SSL certificate
  ansible.builtin.shell:
    argv:
      - /opt/certbot/bin/certbot
      - --nginx
      - --agree-tos
      - -m "{{ ssl_maintainer_email }}"
      - -d "{{ inventory_hostname }}"
      - --non-interactive
  notify:
    - Restart Nginx
  tags: certbot_install

In our case, certboot prints a message if the certificate is not yet due for renewal. If that output is missing then we trigger the Nginx restart (see defining changed):

- name: Get SSL certificate
  ansible.builtin.shell:
    argv:
      - /opt/certbot/bin/certbot
      - --nginx
      - --agree-tos
      - -m {{ ssl_maintainer_email }}
      - -d {{ inventory_hostname }}
      - --non-interactive
  register: certbot_output # Registers the certbot output.
  changed_when: 
    - '"Certificate not yet due for renewal" not in certbot_output.stdout'
  notify:
    - Restart Nginx
  tags: certbot_install

I do want to use shell, as I need to expand the variable for certbot, but ansible-lint is still not happy:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 1 violation(s) that are fatal
command-instead-of-shell: Use shell only when shell functionality is required.
roles/oracle/tasks/nginx.yaml:47 Task/Handler: Get SSL certificate

You can skip specific rules or tags by adding them to your configuration file:
# .config/ansible-lint.yml
warn_list:  # or 'skip_list' to silence them completely
  - command-instead-of-shell  # Use shell only when shell functionality is required.

                   Rule Violation Summary                    
 count tag                      profile rule associated tags 
     1 command-instead-of-shell basic   command-shell, idiom 

Failed after min profile: 1 failure(s), 0 warning(s) on 8 files.

Time to treat this error as a warning, as I know they are not issues, by creating a .config/ansible-lint.yml:

(ansiblelint) [josevnz@dmaf5 OracleCloudHomeLab]$ ansible-lint site.yaml
WARNING  Overriding detected file kind 'yaml' with 'playbook' for given positional argument: site.yaml
WARNING  Listing 1 violation(s) that are fatal
command-instead-of-shell: Use shell only when shell functionality is required. (warning)
roles/oracle/tasks/nginx.yaml:47 Task/Handler: Get SSL certificate


                        Rule Violation Summary                         
 count tag                      profile rule associated tags           
     1 command-instead-of-shell basic   command-shell, idiom (warning) 

Passed with min profile: 0 failure(s), 1 warning(s) on 8 files.

Much better now, the warning is not treated as an error.

Jinja best practices

If you plan to use variables and Jinja templates, make sure you quote them (example: "dest: /opt/prometheus-{{ prometheus_version }}.linux-{{ architecture }}.tar.gz")

Constrain where the playbook runs with --limit and --tags

Say that you are only interested in running your playbook on a certain host. In that case, you can also do that by using the --limit flag:

ansible-playbook --inventory inventory --limit fido.yourcompany.com --tags certbot_renew site.yaml

Here we did run only a task tagged certbot_renew on the host fido.yourcompany.com.

How to deal with a real issue

Let's make this interesting: say that I was eager to update one of my requirements for certboot, and I changed versions if pip to '22.3.1':

pip==22.3.1
wheel==0.38.4
certbot==1.32.0
certbot-nginx==1.32.0

When I run the playbook we have a failure:

This is an issue with the versions if specified on the requirements_certboot.txt file. When you install a Python library using a virtual environment you can specify versions like this:

pip==22.3.1 wheel==0.38.1 certbot==1.23.0 certbot-nginx==1.23.0

To fix the issue, we will revert the versions used on the file and then re-run the requirements file and Certbot installation task:

- name: Setup Certbot
  pip:
    requirements: /opt/requirements_certboot.txt
    virtualenv: /opt/certbot/
    virtualenv_site_packages: true
    virtualenv_command: /usr/bin/python3 -m venv
    state: forcereinstall
  tags: certbot_env

ansible-playbook --inventory inventory --tags certbot_env site.yaml

See it in action:

How to run the whole playbook

ansible-playbook --inventory inventory site.yaml

It is time to run the whole playbook:

Wrapping up

This tutorial only touches the surface of what you can do with Ansible. So below are a few more resources you should explore to learn more:

• Improving inventories: How to create dynamic inventory files in Ansible, How to write a Python script to create dynamic Ansible inventories, How to write an Ansible plugin to create inventory files

• Sometimes your playbooks will run slow, and you may need to Assess resource consumption with Ansible callback plugins.

• And there will be a time when deeper debugging is needed.

All it takes is a single click, and you're in trouble.

This kind of email has a very clear definition:

Phishing is a cybercrime in which a target or targets are contacted by email, telephone or text message by someone posing as a legitimate institution to lure individuals into providing sensitive data such as personally identifiable information, banking and credit card details, and passwords.

In this article, I'll explain what phishing is and how to recognize the signs that an email may not be legit. For that, we will learn to do the following:

• Recognize some obvious flags of a phishing email

• Use some command tools on Linux to carefully inspect suspicious links

• Analyze the suspicious emails with several free online tools

All this while having some fun.

Example of a Phishing Email

Let me share a quite clever example email (some details have been changed to protect the innocent):

Phishing email pretending to be GoDaddy

Let me show you how you can quickly spot scammers, without using a single line of code

You will need the following to go through some of the steps of this tutorial:

• A Linux installation, with curl installed.

• A Web browser (Brave or Firefox are good choices)

• Curiosity

Now let's move on and see what we've got in our mailbox...

Common Sense Phishing Red Flags

Right out of the box, this email violates two simple rules, despite having proper grammar and nice presentation:

First, of all, it forces you to act immediately to fix an issue (Urgent action required), no questions asked (Click the nice button).

To make it worse, there's no way to verify that the person contacting you really works for the company. Reputable companies ask you to log into their website and offer a case # so you can track the issue. Neither of those are here.

Second, despite their best efforts, scammers make qualitative mistakes. Do you see that customer # on the upper right part of the screenshot? I compared it to mine on the real website and guess what? It's a different number.

But where is the fun of analyzing this if we cannot do even a little bit of poking? Well, when I moved my mouse over the button image I could see the link and it was pointing to tiny URL (an URL shortening service):

https://tinyurl.com/xszszasxdxdxdxdxdxdxdzs?a=xxx@xxxx.com

So whoever is doing this is trying to conceal the real URL. No problem, copy the URL address (never click it), change the email part of the GET request to some garbage (?a=xxx@xxx.com)) and then run it through curl. I got this:

<table width="75%" bgcolor="#FFFFFF" align="center" cellpadding="10">
        <tr>
            <td>
                <h2>URL Terminated</h2>
                <p>
                    The TinyURL (xszszasxdxdxdxdxdxdxdzs) you visited was used by its creator in violation of our terms of use.
                    TinyURL has a strict no abuse policy and we apologize for the intrusion this user has caused you.
                    Such violations of our terms of use include:
                </p>
                <ul>
                    <li>Spam - Unsolicited Bulk E-mail</li>
                    <li>Fraud or Money Making scams</li>
                    <li>Malware</li>
                    <li>or any other use that is illegal.</li>
                </ul>
                <p>

So the good people from Tiny URL noticed this too and terminated the URL. Nice work!

Let's use other tools to confirm what we know already.

Online Tools You Can Use to Analyze Suspicious URLs

Tiny URL was nice enough to tell us about the original URL:

https://parasolhealth.org/resources/sass/hgjhgbgb/%20hxghxhgcgzvzvhgxvgzhxgvvgvcgvhgvjhvxhgvzhgvshgvhgvhgvhgwvhgwvhgwvhgwvhgvhgvdshvshgvhgvhgdvhgdsvhgdvhjgdvjhdgdvhgfvhgvf/vhgvjhgvghgvghvhgvghvhgvjlnkjndkjdkjdhbgytdvghdvhvshgvshgvjsvhvahgvhvwgvhwvhvajgvsgshgvhsgvjhsvgavjgvsgvahgvahgvhgsvjgavhgsvhgsvhjvshgvahgvsjvshgvajvshvhgwvhgvehgvehgvehjvegvejhgvhgavhavhs/dhbjhjfhjfkbkjfhbjkbfjbjdbkjbsjhbdjbjkdbhbdjkbjdbjdbjhbdkjbsjbjkdbjkdhbjdbjbsjhbsjbjdkbjhdbkjhbdkjbsbdjbjdbkjhbjhbsjkhbdjbjdbjdbjhsbjhbejhbejhbjwhbjhwbjkwhbjbhbs/jdbhdhdbkjbsjbsjbwjbjwbjkbwhbehbjhbejbebebjebjbejbjhbsjhbshbahbjhsbshbjkhdbjhbjhbdbdjkbdhbjhsbjhbajhbsjbkjshbhbdjhbjdhbjkbshbsjhbsjbdbdhbdhbjehbjhebjhbrrhbjbjekhbjhbjsbjhsbjhbdjhd/jbdjhbdkjbdjhbkjabjhbsjbdjbksjbhsbjhdbjhbjkbdjhbjhbkjbejhbwkhbjkwhbjhwbjkwhbjhwbjhbwhbwkjhbwjhbjhbajhbajhbsjhbsjhbdjkhbdjhbdjhbjdhbjshbjhsbjhbjhsbkjhbdjhbsjbjabjhabjkbs/redirect.php

If you go to the Virus Total website and search for the URL you will see that this was also reported here:

Interestingly enough, only a single vendor reported the URL as malicious. That will do it for me :-)

Also Abuse IP DB doesn't know anything about the offending website. However keep this tool around as it is known to reports multiple other actors.

There is anything else we can learn from the original message? Most email readers allow you to copy and paste the email headers. I'm sharing mine here (with a few changes):

Received: from MN2PR19MB4030.namprd19.prod.outlook.com (2603:10b6:208:1e8::11)
 by MW3PR19MB4204.namprd19.prod.outlook.com with HTTPS; Tue, 4 Oct 2022
 16:35:05 +0000
Received: from BN9PR03CA0959.namprd03.prod.outlook.com (2603:10b6:408:108::34)
 by MN2PR19MB4030.namprd19.prod.outlook.com (2603:10b6:208:1e8::11) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.31; Tue, 4 Oct
 2022 16:35:01 +0000
Received: from BN7NAM10FT104.eop-nam10.prod.protection.outlook.com
 (2603:10b6:408:108:cafe::cc) by BN9PR03CA0959.outlook.office365.com
 (2603:10b6:408:108::34) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.24 via Frontend
 Transport; Tue, 4 Oct 2022 16:34:59 +0000
Authentication-Results: spf=softfail (sender IP is 170.10.162.128)
 smtp.mailfrom=bounce.com; dkim=none (message not signed)
 header.d=none;dmarc=fail action=oreject header.from=godaddy.com;compauth=fail
 reason=000
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 bounce.com discourages use of 170.10.162.128 as permitted sender)
Received: from host.solutiononellc.com (170.10.162.128) by
 BN7NAM10FT104.mail.protection.outlook.com (10.13.157.118) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.5676.17 via Frontend Transport; Tue, 4 Oct 2022 16:34:59 +0000
Received: from ip250.ip-37-187-205.eu ([37.187.205.250]:38823)
    by altar47.supremepanel47.com with esmtpsa  (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    (Exim 4.95)
    (envelope-from <postmaster@bounce.com>)
    id 1ofksk-0005Zd-LV
    for xxx@xxxx.com;
    Tue, 04 Oct 2022 16:34:58 +0000

Using [MXToolbox](https://mxtoolbox.com/Public/Tools/EmailHeaders.aspx?huid=4205dc8f-5147-4da5-a448-d633f2bbca61) shows that 2 of the email addresses used in the chain are **blacklisted**, another red flag.

![Image](https://www.freecodecamp.org/news/content/images/2022/10/godaddy_scammer_mxtoolbox.png)
_2 blocked emails from this list. Another read flag_

I think that's good enough. Delete the email and move on with your life, and be sure a new email is coming your way (hopefully landing in the SPAM folder automatically).

## What's Next?

There are many tools on the Internet you can use to identify phishing emails, but there is no substitute for common sense. It if looks too good to be true then it probably is.

As usual, do not click the link right away! Do a little investigating first, just to be safe.

If you have an Android phone, one of your guilty pleasures might be playing some very entertaining games. Or it could be that there is an application that only runs on your phone.

And then you think – what if you could run the same games on your desktop PC?

To simplify the scenario, let's assume the applications run on Android.

One approach to solve your problem is to run an Android emulator on your PC. But some of them, like Android-x86, require rebooting your machine so they can take control of the hardware.

If you don't mind a small performance hit you can run a virtual machine at the same time as your native operating system. Specifically on Linux, there are several choices, like QEMU and VirtualBox, to name a few.

By the end of this article you will be able to do the following:

• Install VirtualBox on Fedora Linux

• Run android-x86 and finish the basic setup

• Install an application from the Google Play Store, just like on your phone.

Basic Requirements

Before you start, I assume that you have the following:

• Ability to run commands as the superuser (like SUDO)

• An account on Google.com, so you can use the Play store from within the virtual machine.

How to Install VirtualBox

The first step is to install VirtualBox. For practical purposes, our installation will be basic, just enough to run our games:

sudo dnf install -y kernel-devel kernel-devel-5.14.18-100.fc33.x86_64
curl --remote-name --location https://www.virtualbox.org/download/oracle_vbox.asc
sudo rpm --import ./oracle_vbox.asc
sudo dnf install -y https://download.virtualbox.org/virtualbox/6.1.36/VirtualBox-6.1-6.1.36_152435_fedora33-1.x86_64.rpm
sudo dnf install -y virtualbox-guest-additions.x86_64
sudo /sbin/vboxconfig

How to Install the Android-x86 ISO

The first step is to download the ISO image from Android-x86. This ISO contains the Android operating system that will be installed on our virtual hard drive.

After that we can set up our virtual machine like this:

How a finished virtual machine looks like on VirtualBox

A few things to note:

• After booting the first time, I found that 1GB for the Android image was not enough. Performance improved a lot after I bumped the ram to 3GB.

• Another change was the 'Graphics Controller'. Originally it was VMSVGA but then Android refused to start in graphic mode, so I switched to VboxVGA and it worked.

• 2 CPUS and 8GB of disk space were enough for my game.

• Finally, I specified that the IDE controller was the android-x86 ISO.

To start the virtual machine, you click the 'Start' button on the GUI, and then you will have to make a few decisions like bootable partition:

Partitioning your virtual disk. We assign 8 GB and make sure the partition can boot

Once this is done you can choose your new partition to perform the installation:

After the new partition is created, you can choose it and you can install the Android OS there

Then the installation will proceed:

The installer copies the files from the Android ISO image into the virtual hard drive

After the installation is complete, you can shut down the virtual machine.

First Boot

Now you'll need to go to the advanced options and select the virtual disk (instead of the ISO image) to boot:

You can either boot from disk on this menu or change the boot order on the virtual machine

After that, Android will ask you some basic setup information, just like it does on your phone. The final result may look like this:

The virtual machine looks exactly like your Android phone.

How to Install Games from the Google Play Store

In my case I decided to install a game where I can fight forces of evil as 1970 Mazinger Z/ Tranzor Z (Yes, I love Go Nagai Mazinger Z). To do that, search on the play store and install the game:

After Android is running and your credentials are set you can download and install any Android program you want.

And now, success! We got the game up and running.

Sorry, but now it is time to play as Mazinger Z!

What Did We Learn Here?

• We managed to install a virtual machine engine and successfully run the Android operating system along with our regular Fedora OS

• You saw how you can try and discard whole operating systems' setup, without going through the hassle of setting up a dual boot system with Grub on Linux

Another nice feature of running the game inside a virtual machine is that you can fully freeze the game, then come back and restore it at exactly the same point where you left it.

Finally, you can do many more things with a virtual machine than just running games, for example:

• You can analyze malware safely, run un-trusted applications, and contain any damage they can cause.

• Try a new operating system version before deciding to commit a proper installation (not a big issue these days as most of them provide a lice CD you can boot to try), but this is still very convenient.

• Be able to run multiple operating systems simultaneously, without rebooting your machine. You most likely will start trying more advanced options of your virtual machine of choice, like VirtualBox.

Playing games on your PC is a gateway for learning more complex stuff later. Also the fun factor is undeniable. Enjoy!

What I found was a very complete Open Source platform with lots of features and excellent documentation. So I wanted to share my experience with you.

What will we cover in this article?

• How to install the Falco agent on the host that you want to monitor for events (anomalies/violations)

• How to tune Falco to reduce false positives and get the information you really need

• How to use Prometheus to collect Falco events into a central location, with the help of the exporters and a scraper.

• Finally, how to connect the scraper with Grafana for visualization and alerting

What do you need for this tutorial?

• A machine or machines with Linux installed. A virtual machine should work.

• You will need superuser permissions to be able to install/setup Docker, RPM, and systemd processes

• We will use Docker containers, so basic knowledge of Docker is required

• Working knowledge of Python/Bash, as we will write a few scripts to test and improve our configuration.

At the end you will be able to setup each one of the following components:

Don't be intimidated – I'll provide links to the documentation and a thorough explanation of each one of these tasks as we move along.

Table of contents

1. What is Falco?

2. How to Install Falco

3. Basic Configuration

4. How to Test the Default Configuration

5. Defaults Are Not Always Good

6. Falco Integrations

7. Learning More

What is Falco?

The best way to describe this tool is to learn what it can do:

Falco can detect and alert on any behavior that involves making Linux system calls.

Falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, Falco can easily detect incidents including but not limited to:

• A shell is running inside a container or pod in Kubernetes.

• A container is running in privileged mode, or is mounting a sensitive path, such as /proc, from the host.

• A server process is spawning a child process of an unexpected type.

• Unexpected read of a sensitive file, such as /etc/shadow.

• A non-device file is written to /dev.

• A standard system binary, such as ls, is making an outbound network connection.

• A privileged pod is started in a Kubernetes cluster.

How to Install Falco

I will install Falco using an RPM (similar instructions exist for apt-get, and even Docker containers). In my case I felt the native installation was the best, and the RPM made it very easy to do:

[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm                                                                                                                                                                                                       1.7 MB/s |  12 MB     00:07    
Dependencies resolved.
==============================================================================================================================================================================================================================================================
 Package                                                          Architecture                                      Version                                                                     Repository                                               Size
==============================================================================================================================================================================================================================================================
Installing:
 falco                                                            x86_64                                            0.31.1-1                                                                    @commandline                                             12 M
Installing dependencies:
 dkms                                                             noarch                                            2.8.1-4.20200214git5ca628c.fc30                                             updates                                                  78 k
 elfutils-libelf-devel                                            x86_64                                            0.179-2.fc30                                                                updates                                                  27 k
 kernel-devel                                                     x86_64                                            5.6.13-100.fc30                                                             updates                                                  14 M

Transaction Summary
==============================================================================================================================================================================================================================================================
Install  4 Packages

Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm                                                                                                                                                                          253 kB/s |  27 kB     00:00    
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm                                                                                                                                                                        342 kB/s |  78 kB     00:00    
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm                                                                                                                                                                                1.9 MB/s |  14 MB     00:07    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         1.8 MB/s |  14 MB     00:07     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                      1/1 
  Installing       : kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Installing       : elfutils-libelf-devel-0.179-2.fc30.x86_64                                                                                                                                                                                            2/4 
  Installing       : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Installing       : falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Running scriptlet: falco-0.31.1-1.x86_64

Basic Configuration

Unless we want to do very basic output processing, we want to enable the JSON output:

# Whether to output events in json or text
json_output: true

It will become evident why pretty soon.

Next start the Falco agent:

[josevnz@macmini2 falco]$ sudo systemctl start falco.service 
[josevnz@macmini2 falco]$ sudo systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
     Docs: https://falco.org/docs/
  Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 26888 (falco)
    Tasks: 1 (limit: 2310)
   Memory: 65.8M
   CGroup: /system.slice/falco.service
           └─26888 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:

How to Test the Default Configuration

Depending on your configuration, you may or may not get any events right after starting Falco:

[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765

Worry not. We will run a few commands that will cause Falco to record some warning and alerts. Time to see how this works!

How to Run a Privileged Container

Using privileged containers is considered a bad practice, so let's see if this event is detected by Falco:

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow

And our Falco log?

May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}

It shows up as an informational event. Definitely one of those things to keep an eye on. Ask yourself if the application on the container needs elevated privileges.

You also probably noticed that each message has tags. Pay attention to the "mitre_*" ones, they do relate to the Mitre Attack knowledge base of attacks and mitigations. Yep, you will spend some time reading those.

How to Create a File on the /root Directory

This example shows how to abuse the root user combined with volumes in a container...

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$

Falco reaction:

May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

Sensitive mount detected!

Let's Raise the Stakes by Creating a File on /bin

Alright let's say we do this:

[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here

What does Falco think about it?

May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}

An error, binary directory opened for writing. Good catch.

Defaults Are Not Always Good

After Falco is running for a while it is a good idea to get a sense of what kind of events we want to ignore and which ones we want to investigate.

The first step is to get a list of all the events, using our JSON format on the payload:

sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt

The 'output=cat' tells journalctl to give us the message payload without timestamps (don't worry, the JSON message itself has timestamps).

Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

So far it looks interesting, but what about this?

{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}

This is a normal/ legitimate operation. Let's find a way to harden this rule or remove it completely.

First, open the /etc/falco/falco_rules.yaml file and look for the rule 'Delete or rename shell history' (JSON output we saw earlier):

- list: docker_binaries
  items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]

 macro: var_lib_docker_filepath
  condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)

- rule: Delete or rename shell history
  desc: Detect shell history deletion
  condition: >
    (modify_shell_history or truncate_shell_history) and
       not var_lib_docker_filepath and
       not proc.name in (docker_binaries)
  output: >
    Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
  priority:
    WARNING
  tags: [process, mitre_defense_evasion]

Falco rules are explained in detail on the official documentation. Just by looking at this piece you will notice a few things.

About the conditions:

1. Support complex logic,

2. macros like var_lib_docker_filepath

3. lists like (docker_binaries)

4. and special variables with fields like proc.name.

It is recommended that do you not change this file. Instead you should override what you need on the /etc/falco/falco_rules.local.yaml:

# Add new rules, like this one
# - rule: The program "sudo" is run in a container
#   desc: An event will trigger every time you run sudo in a container
#   condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
#   output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
#   priority: ERROR
#   tags: [users, container]

# Or override/append to any rule, macro, or list from the Default Rules

For the sake of example, say that we do care when the history of the super-user (root) is overridden, but everybody else if fine. The best part is that you don't have to override the whole rule.

So the original rule will get a condition appended:

- rule: Delete or rename shell history
  append: true
  condition: and user.name=root

It is always a good idea to validate that your rules are properly written. For that you need can tell Falco to check the original rules and your overrides together:

[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml 
Fri May  6 20:48:00 2022: Validating rules file(s):
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.yaml
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May  6 20:48:01 2022: Ok

# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service

How to Make a Simple Event Explorer in Python

You'll probably agree that getting a sense of what rules are noise and which ones are useful is tedious.

We need to normalize this data, and we will use a Python script that will:

• Remove non-JSON data

• Aggregate event types without the timestamps

• Generate a few aggregation statistics, so we can focus on the most frequent events in our system

A small Python script can do the trick. I'm leaving out the UI rendering part (please check the code to see the full picture), and instead will show you the file parsing bits:

#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table


def filter_events(journalctl_out: Path) -> dict[any, any]:
    """
    :param journalctl_out:
    :return:
    """
    with open(journalctl_out, 'r') as journalctl_file:
        for row in journalctl_file:
            if re.search("^{", row):
                data = json.loads(row)
                if 'rule' in data and 'output_fields' in data:
                    yield data


def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
    rule = local_event['rule']
    if rule not in aggregated_events:
        aggregated_events[rule] = {
            'count': 0,
            'priority': local_event['priority'],
            'last_timestamp': "",
            'last_fields': ""
        }
    aggregated_events[rule]['count'] += 1
    aggregated_events[rule]['last_timestamp'] = local_event['time']
    del local_event['output_fields']['evt.time']
    aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_event",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        event_table = create_event_table()
        for event in filter_events(ARGS.falco_event):
            aggregate_events(local_event=event, aggregated_events=AGGREGATED)
        add_rows_to_create_event_table(AGGREGATED, event_table)
        EventDisplayApp.run(
            event_file=ARGS.falco_event,
            title="Falco aggregated events report",
            event_table=event_table
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

Once the file is loaded as a dictionary, we only need to iterate to it to aggregate the events, then show the results as a neat table sorted by count:

How to Show the Falco Rules

If you are like me, you are always looking at the /etc/falco/falco_rules.yaml file to understand what is being monitored. A brief view of those rules (without looking at the verbose YAML file with comments) is a nice addition:

#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp


def load_rulez(falco_rulez: Path) -> dict[any, any]:
    rulez = {}
    with open(falco_rulez, 'rt') as falco_file:
        for rule_data in yaml.full_load(falco_file):
            if 'rule' in rule_data:
                rule_name = rule_data['rule']
                del rule_data['rule']
                rulez[rule_name] = rule_data
    return rulez


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_rules",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        RULES = load_rulez(ARGS.falco_rules)
        RULE_TBL = create_rules_table()
        add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
        RulesDisplayApp.run(
            rules_file=ARGS.falco_rules,
            title="Falco brief rule display",
            rules_table=RULE_TBL
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

You could improve this script by adding rule filtering by certain criteria, for example (rule name, priority, enabled/ disabled). This version doesn't do any filtering:

Falco Integrations

You probably noticed two things from our earlier experimentation:

1. The payload of the events do not have the host. If you want to locate an offending server, you need to improve how a multi-host event is reported (parsing a journalctl file from many hosts is not practical).

2. We want to get alerts in a centralized location. It would be nice to have a way to "push" those events instead of us going to fish.

It is time to consolidate those alerts in a single place.

How to Use Falco Exporter

The Falco exporter will allow us to share the Falco alerts with Prometheus scraper. We need first to enable gRPC in the /etc/falco/falco.yaml

# gRPC server using an unix socket
grpc:
  enabled: true
  bind_address: "unix:///var/run/falco.sock"
  # when threadiness is 0, Falco automatically guesses it depending on the number of online cores
  threadiness: 0

# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
  enabled: true

Restart Falco:

[root@macmini2 ~]# systemctl restart falco.service 
[root@macmini2 ~]# systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
     Docs: https://falco.org/docs/
  Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 28288 (falco)
    Tasks: 11 (limit: 2310)
   Memory: 80.9M
   CGroup: /system.slice/falco.service
           └─28288 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock

Quickly make sure everything is OK (reminder, the Falco agent is running on macmini2):

josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$

Then we run the falco-exporter. To make it easier, we will use a Docker container with a few overrides in the command line.

[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready

# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06

For completeness, let me show you also how to capture the host performance metrics using the node exporter (we will use it later to keep an eye on how many resources are used by Falco and to make sure our installation is not hurting the server):

docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host

The node-exporter and the falco-exporter will run on every host that needs their data scraped. Now you need to wait to collect all these metrics into a single location. For that will use the Prometheus agent:

---
# /etc/prometheus.yaml on raspberrypi
global:
    scrape_interval: 30s
    evaluation_interval: 30s
    scrape_timeout: 10s
    external_labels:
        monitor: 'nunez-family-monitor'

scrape_configs:
  - job_name: 'falco-exporter'
    static_configs:
      - targets: ['macmini2.home:9376']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
  - job_name: 'docker-exporter'
    static_configs:
      - targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']

    tls_config:
      insecure_skip_verify: true

Then make sure the Prometheus scraper can talk with each one of the nodes. We visit the web UI:

Good, Prometheus is able to scrape Falco. We can even run a simple query to see a few events:

Next we need to setup the UI view for the events, and for that we will use Grafana.

There are many ways to install Grafana. In my case I will use a Grafana Docker container (I will run Grafana on the same host where Prometheus is running: raspberripi.home):

docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu

After Grafana comes up, you will need to change your password and will also need to connect with Prometheus:

Once Grafana is up, we can import the Falco dashboard as explained here.

Once the dashboard is imported we can generate a few events to trigger Falco on the host where is installed:

[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done

After a little you should see something like this on your Grafana Dashboard:

The events are flowing, and you can see from which host they came.

How to Create Alerts for Your Falco Events

Ideally if you have the Falco events in Grafana, you can make these actionable items and generate alerts from then.

I don't want to get bombarded by non-critical alerts, so the first thing to know is to what level of events to filter:

Anything with priority below 3 will be treated as an alert.

Grafana has good documentation on how to setup an alert, so I will show here the end result only:

The next step is to send the alerts somewhere.

Alerts Need to Go Somewhere – How to Define a Contact Point using Discord

For this example we will use Discord as the end for the alerts. Discord has a very detailed guide on how to setup a WebHook, so I will only show you here the end results of my discord Webhook:

We copy that URL and then will configure a new Grafana contact point that uses our Discord webhook (we are setting this a default contact point for all the alerts):

From there we can send a test message to Discord, just to confirm that this pipeline works:

We're getting closer. By now if we go back to our alert definition we should see it is on the 'firing' state:

And if everything goes well we see also our first Falco alert in Discord:

We can see here all the fields we get on the journalctl output. The difference is that all these messages will come from all the servers where you define the Falco-Prometheus-Grafana bridge.

Honorable Mention: How to Aggregate Alerts using Falcon Sidekick/ Falcon Sidekick-UI

Falco Sidekick is another way to gather and send events to other destinations, like the Falco Sidekick-UI. But it won't tell you the originating host (at least until Falco 0.31.1).

This is most likely not an issue for an alert coming from a K8s cluster or a containerized application where the image name will give you plenty of information. But if your event happens on a bare-metal environment, and you have more than 2 machines, it will become a headache.

For that reason I won't cover Sidekick here – you may want to stick with the Grafana integration for the time being.

Learning More

Falco has a great interactive learning environment. You should try it to see what else is possible. There is a lot of things I did not cover here, like rule exceptions for example.

Also, did you know that Falco can be extended using plugins? You can have fun and learn using C++ or Go as the language of choice

The Falco blog has lots of interesting articles, including posts for the latest threats.

Finally, the project has a very active community on many channels. Pick yours and explore.

Feel free to fork my code and report any issues if you find any. But more important, explore and learn by doing.

This list is not universal and, depending on what you are looking for, it may not be enough to solve your problem. But it should be a good start.

Before we start you should be familiar with a few things:

• How to run commands on Linux

• Protocols like DNS, HTTP, and TLS

• A scripting language like Python

Don't worry too much. I will give you enough information so you can follow along with the tutorial.

And what will you learn?

• Basic usage of strace, nslookup, and RPM

• How to use some interesting features of the Python debugger

• How to analyze traffic with Wireshark

The Problem: Failing to upload a file to asciinema

So I recorded an asciicast, using the cool Open Source project asciinema, for my small Open Source project SuricataLog. Then I decided to share it with the world.

But unlike the other recordings, this one refused to be uploaded:

[josevnz@dmaf5 SuricataLog]$ asciinema upload demo-ascii.cast 
asciinema: upload failed: <urlopen error [Errno 32] Broken pipe>
asciinema: retry later by running: asciinema upload demo-ascii.cast

Asciinema doesn't tell us much about the error. For example:

• What server and port does the tool try to use to upload the file?

• Which part of the protocol handshake is failing?

• Is the destination a problem or is it an issue on my side?

We will use a few tools to see what is going on here.

How to run the program with strace

What is strace?

strace is a diagnostic, debugging, and instructional userspace utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.

System administrators, diagnosticians, and troubleshooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them.

strace is super useful when you don't have the source code of an application and yet you need to understand what is wrong when you call a program. Time to see it in action:

josevnz@dmaf5 SuricataLog]$ strace asciinema upload demo-ascii.cast
xecve("/usr/bin/asciinema", ["asciinema", "upload", "demo-ascii.cast"], 0x7ffdcddb1160 /* 55 vars */) = 0
brk(NULL)                               = 0x55e912d58000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fff2f136480) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=92299, ...}) = 0
mmap(NULL, 92299, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f69dd26a000
close(3)                                = 0
# 
# Commented out LOTS output
# ...
close(4)                                = 0
socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("109.107.38.233")}, 16) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(33771), sin_addr=inet_addr("192.168.1.22")}, [28 => 16]) = 0
connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("109.107.37.0")}, 16) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(35023), sin_addr=inet_addr("192.168.1.22")}, [28 => 16]) = 0
close(4)                                = 0
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("109.107.38.233")}, 16) = 0
setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
getsockopt(4, SOL_SOCKET, SO_TYPE, [1], [4]) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(55682), sin_addr=inet_addr("192.168.1.22")}, [128 => 16]) = 0
ioctl(4, FIONBIO, [0])                  = 0
getpeername(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("109.107.38.233")}, [16]) = 0
getpid()                                = 45070
getpid()                                = 45070
getpid()                                = 45070
getpid()                                = 45070
getpid()                                = 45070
getpid()                                = 45070
write(4, "\26\3\1\2\0\1\0\1\374\3\3\327\2*\v\316GT*\262\207\235\264\317\254\37$|,V\205\362"..., 517) = 517
read(4, "\26\3\3\0z", 5)                = 5
read(4, "\2\0\0v\3\3E\217G?\335.;\212\237pn\16\257$\2\324J\324y\17\306\263\325i\264p"..., 122) = 122
read(4, "\24\3\3\0\1", 5)               = 5
read(4, "\1", 1)                        = 1
read(4, "\27\3\3\0\27", 5)              = 5
read(4, "0{}\22t9\264\265\340j\362\30\342\360\234\205\1\370\33\246\1z'", 23) = 23
read(4, "\27\3\3\17\335", 5)            = 5
read(4, "\17\5\310\261\355\271\227oUaI\366\361]\3\275q)\5{\367z\20\233\345\352k?\371\272\23\237"..., 4061) = 4061
stat("/etc/pki/tls/certs/8d33f237.0", 0x7ffd20be3620) = -1 ENOENT (No such file or directory)
read(4, "\27\3\3\1\31", 5)              = 5
read(4, "t\27\337\366G6\226Qs\273\327\314,\205\221\222Xu\233\21%\0s\340\270\224\330\t\2774\222h"..., 281) = 281
read(4, "\27\3\3\0005", 5)              = 5
read(4, "\204{\314\232\311\0P-*$\245\315\271\236c\210N\315\5\371\364\23\235\16\0350N0K\246\336\374"..., 53) = 53
write(4, "\24\3\3\0\1\1\27\3\3\0005\361\311\347\t\254m#\273\204\350\16\343\34P\320sS\211\30\232<"..., 64) = 64
ioctl(4, FIONBIO, [0])                  = 0
write(4, "\27\3\3\1\251\271\2673-\30\313\253\363\320H0\224\370Q\353(#?,\216\3\341\315|J\353\303"..., 430) = 430
write(4, "\27\3\3@\21\20\221\240\331\2737\10\244pv\312B\n\rn\272\33\336T\216\f\303\374k\177c\25"..., 16406) = 16406
write(4, "\27\3\3@\21\214\30\262\240s\216\240\354e\31\304Q\337Oy\21y\373\241g\311\224)\26\320\10{"..., 16406) = 16406
write(4, "\27\3\3@\21\36\323\240\376\276\224\35\f\10!@\36D\347\33ay\2617Hpv\4d\267y7"..., 16406) = 16406
write(4, "\27\3\3@\21\366x\264\242O2\7?\7\334\221W\24\2\f)\"@\20\375~\354\243W\32\0c"..., 16406) = 16406
write(4, "\27\3\3@\21\354\32W\36\265g\304\314\376\205\315\20\22\10c\333\342\264\330\366SS\4\217\356:V"..., 16406) = 16406
write(4, "\27\3\3@\21\1\274\35\335\271n\235e\202\202\207\221~\313\0y\210\344\312\32r\347\306x]\241C"..., 16406) = 16406
write(4, "\27\3\3@\21I\315\202\274\342\274\26\335qx\22-\226\322\320\203\231\274wLB\250\252\2\352\367\""..., 16406) = 8716
write(4, "\377\4m\341\317\376SUr\rQ\221\207\22#\262\314B7\33_v\310\271\fl\v\242\fK\v?"..., 7690) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=45070, si_uid=1000} ---
close(4)                                = 0
close(3)                                = 0
write(2, "\33[0;31masciinema: upload failed:"..., 76asciinema: upload failed: <urlopen error [Errno 32] Broken pipe>
) = 76
write(2, "\33[0;31masciinema: retry later by"..., 79asciinema: retry later by running: asciinema upload demo-ascii.cast
) = 79
munmap(0x7fa1aa089000, 12447744)        = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fa1bad0fa70}, {sa_handler=0x7fa1baf551d0, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fa1bad0fa70}, 8) = 0
munmap(0x7fa1ac649000, 593920)          = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Look at this socket call (man 2 getpeername):

getpeername(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("109.107.38.233")}, [16]) = 0

And below that, as you can see, we are actually writing data to the website and the connection breaks:

write(4, "\27\3\3@\21\366x\264\242O2\7?\7\334\221W\24\2\f)\"@\20\375~\354\243W\32\0c"..., 16406) = 16406
write(4, "\27\3\3@\21\354\32W\36\265g\304\314\376\205\315\20\22\10c\333\342\264\330\366SS\4\217\356:V"..., 16406) = 16406
write(4, "\27\3\3@\21\1\274\35\335\271n\235e\202\202\207\221~\313\0y\210\344\312\32r\347\306x]\241C"..., 16406) = 16406
write(4, "\27\3\3@\21I\315\202\274\342\274\26\335qx\22-\226\322\320\203\231\274wLB\250\252\2\352\367\""..., 16406) = 8716
write(4, "\377\4m\341\317\376SUr\rQ\221\207\22#\262\314B7\33_v\310\271\fl\v\242\fK\v?"..., 7690) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=45070, si_uid=1000} ---

So who is '109.107.38.233'?:

[josevnz@dmaf5 SuricataLog]$ nslookup 109.107.38.233
233.38.107.109.in-addr.arpa    name = cip-109-107-38-233.gb1.brightbox.com.

You can see on the about webpage that brightbox.com provides the hosting for asciinema.

So what is wrong? It is not that the site is down or unreachable. Can we dig further?

If I only had the source code – deep diving with the Python debugger

[josevnz@dmaf5 SuricataLog]$ file /usr/bin/asciinema
/usr/bin/asciinema: Python script, ASCII text executable

Oh, yes we do! Ever curious, you open the asciinema script:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'asciinema==2.0.2','console_scripts','asciinema'
import re
import sys 

# for compatibility with easy_install; see #2198
__requires__ = 'asciinema==2.0.2'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point


def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = ( 
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )   
    return next(matches).load()


globals().setdefault('load_entry_point', importlib_load_entry_point)


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('asciinema==2.0.2', 'console_scripts', 'asciinema')())

The main script was generated with easy install, which means asciinema.py is just a wrapper around the interesting code. To find out where the interesting stuff is, let's run the script through the Python pdb debugger:

[josevnz@dmaf5 SuricataLog]$ python3 -m pdb /usr/bin/asciinema upload demo-ascii.cast 
> /usr/bin/asciinema(3)<module>()
-> import re
(Pdb) n
> /usr/bin/asciinema(4)<module>()
-> import sys
(Pdb) c
asciinema: upload failed: <urlopen error [Errno 32] Broken pipe>
asciinema: retry later by running: asciinema upload demo-ascii.cast
The program exited via sys.exit(). Exit status: 1

Not quite what we need. The program runs, hits the exception, and then it restarts at the beginning.

Let's cheat a little. Was asciinema installed with an RPM (I use Fedora Linux)?

[josevnz@dmaf5 SuricataLog]$ rpm -qif /usr/bin/asciinema
Name        : asciinema
Version     : 2.0.2
Release     : 6.fc33

And we are trying to upload a file, anything that looks like an uploader?

josevnz@dmaf5 SuricataLog]$ rpm -qil asciinema|grep -i uploa
/usr/lib/python3.9/site-packages/asciinema/commands/__pycache__/upload.cpython-39.opt-1.pyc
/usr/lib/python3.9/site-packages/asciinema/commands/__pycache__/upload.cpython-39.pyc
/usr/lib/python3.9/site-packages/asciinema/commands/upload.py

Ah, getting interesting! Let's open 'upload.py':

from asciinema.commands.command import Command
from asciinema.api import APIError


class UploadCommand(Command):

    def __init__(self, api, filename):
        Command.__init__(self)
        self.api = api 
        self.filename = filename

    def execute(self):
        try:
            result, warn = self.api.upload_asciicast(self.filename)

            if warn:
                self.print_warning(warn)

            self.print(result.get('message') or result['url'])

        except OSError as e:
            self.print_error("upload failed: %s" % str(e))
            return 1

        except APIError as e:
            self.print_error("upload failed: %s" % str(e))
            self.print_error("retry later by running: asciinema upload %s" % self.filename)
            return 1

        return 0

Let's put a few breakpoints inside the UploadCommand (Lines 14, 26 on my copy of the code):

[josevnz@dmaf5 SuricataLog]$ python3 -m pdb /usr/bin/asciinema upload demo-ascii.cast 
> /usr/bin/asciinema(3)<module>()
-> import re
(Pdb) b /usr/lib/python3.9/site-packages/asciinema/commands/upload.py:14
Breakpoint 1 at /usr/lib/python3.9/site-packages/asciinema/commands/upload.py:14
(Pdb) c
> /usr/lib/python3.9/site-packages/asciinema/commands/upload.py(14)execute()
-> result, warn = self.api.upload_asciicast(self.filename)
(Pdb) c
> /usr/lib/python3.9/site-packages/asciinema/commands/upload.py(26)execute()
-> self.print_error("upload failed: %s" % str(e))
(Pdb) n
asciinema: upload failed: <urlopen error [Errno 32] Broken pipe>
> /usr/lib/python3.9/site-packages/asciinema/commands/upload.py(27)execute()
-> self.print_error("retry later by running: asciinema upload %s" % self.filename)
(Pdb) ll
 12          def execute(self):
 13              try:
 14 B                result, warn = self.api.upload_asciicast(self.filename)
 15      
 16                  if warn:
 17                      self.print_warning(warn)
 18      
 19                  self.print(result.get('message') or result['url'])
 20      
 21              except OSError as e:
 22                  self.print_error("upload failed: %s" % str(e))
 23                  return 1
 24      
 25              except APIError as e:
 26 B                self.print_error("upload failed: %s" % str(e))
 27  ->                self.print_error("retry later by running: asciinema upload %s" % self.filename)
 28                  return 1
 29      
 30              return 0

We got an APIError. Anything interesting with that type of exception?

(Pdb) source APIError
11      class APIError(Exception):
12          pass
(Pdb) e.args
('<urlopen error [Errno 32] Broken pipe>',)

So nothing special, derived directly from Exception. Also, the arguments to the exception are just the error message.

Of course the next step is to see if this error comes from a known library (Search on the Internet for the error).

I found this issue on GitHub. Reading further you can see than the generous author of Asciicast is paying the storage from his own pocket so all of us can enjoy the online storage for free:

The max size was set 2MB which appears to be too low. I have upped it to 5MB. This isn't much, but I'm paying for the storage (S3) from my own pocket, so I can't offer GBs of storage for every user. Let me know if that works for you. I'm fine with increasing it even more, but now I want to figure out the good middle ground between user needs and hosting costs.

So let's confirm that this is indeed the cause:

[josevnz@dmaf5 SuricataLog]$ ls -lh demo-ascii.cast 
-rw-rw-r-- 1 josevnz josevnz 12M Apr 21 15:44 demo-ascii.cast

So far the big size of the file seems to be the culprit.

I'm still running the debugger, and I would love to see what asciinema modules were loaded. For that switch to the 'interact' mode and get that listing with a list comprehension and a regular expression:

(Pdb) interact
*interactive*
>>> import re
>>> import sys
>>> import pprint
>>> pprint.pprint([name for name in sys.modules.keys() if re.search('asciinema', name)], indent=True)
['asciinema.asciicast.events',
 'asciinema.asciicast.v1',
 'asciinema.asciicast.v2',
 'asciinema.asciicast',
 'asciinema.term',
 'asciinema.pty',
 'asciinema',
 'asciinema.config',
 'asciinema.commands',
 'asciinema.commands.command',
 'asciinema.commands.auth',
 'asciinema.asciicast.raw',
 'asciinema.http_adapter',
 'asciinema.urllib_http_adapter',
 'asciinema.api',
 'asciinema.commands.record',
 'asciinema.player',
 'asciinema.commands.play',
 'asciinema.commands.cat',
 'asciinema.commands.upload',
 'asciinema.__main__']
>>>

The following look like they could hold some clues:

• asciinema.urllib_http_adapter

• asciinema.commands.upload

• asciinema.http_adapter

Exit the debugger (or go to another terminal) and search for the urllib_http_adapter:

[josevnz@dmaf5 SuricataLog]$ find /usr/lib/python3.9/site-packages/asciinema/ -name 'urllib_http_adapter*'
/usr/lib/python3.9/site-packages/asciinema/__pycache__/urllib_http_adapter.cpython-39.opt-1.pyc
/usr/lib/python3.9/site-packages/asciinema/__pycache__/urllib_http_adapter.cpython-39.pyc
/usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py

If you open the file you will see that 'post' method is the one we want to troubleshoot:

class URLLibHttpAdapter:

    def post(self, url, fields={}, files={}, headers={}, username=None, password=None):
        content_type, body = MultipartFormdataEncoder().encode(fields, files)

        headers = headers.copy()
        headers["Content-Type"] = content_type

        if password:
            auth = "%s:%s" % (username, password)
            encoded_auth = base64.encodebytes(auth.encode('utf-8'))[:-1]
            headers["Authorization"] = b"Basic " + encoded_auth

        request = Request(url, data=body, headers=headers, method="POST")

        try:
            response = urlopen(request)
            status = response.status
            headers = self._parse_headers(response)
            body = response.read().decode('utf-8')
        except HTTPError as e:
            status = e.code
            headers = {}
            body = e.read().decode('utf-8')
        except (http.client.RemoteDisconnected, URLError) as e:
            raise HTTPConnectionError(str(e))

        return (status, headers, body)

A breakpoint in line 65 will get us where we need to be:

[josevnz@dmaf5 SuricataLog]$ python3 -m pdb /usr/bin/asciinema upload demo-ascii.cast 
> /usr/bin/asciinema(3)<module>()
-> import re
(Pdb) b /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py:65
Breakpoint 1 at /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py:65
(Pdb) c
> /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py(65)post()
-> headers = headers.copy()
(Pdb) ll
 62          def post(self, url, fields={}, files={}, headers={}, username=None, password=None):
 63              content_type, body = MultipartFormdataEncoder().encode(fields, files)
 64      
 65 B->            headers = headers.copy()
 66              headers["Content-Type"] = content_type
 67      
 68              if password:
 69                  auth = "%s:%s" % (username, password)
 70                  encoded_auth = base64.encodebytes(auth.encode('utf-8'))[:-1]
 71                  headers["Authorization"] = b"Basic " + encoded_auth
 72      
 73              request = Request(url, data=body, headers=headers, method="POST")
 74      
 75              try:
 76                  response = urlopen(request)
 77                  status = response.status
 78                  headers = self._parse_headers(response)
 79                  body = response.read().decode('utf-8')
 80              except HTTPError as e:
 81                  status = e.code
 82                  headers = {}
 83                  body = e.read().decode('utf-8')
 84              except (http.client.RemoteDisconnected, URLError) as e:
 85                  raise HTTPConnectionError(str(e))
 86      
 87              return (status, headers, body)
(Pdb) args
self = <asciinema.urllib_http_adapter.URLLibHttpAdapter object at 0x7f59ed3e4640>
url = 'https://asciinema.org/api/asciicasts'
fields = {}
files = {'asciicast': ('ascii.cast', <_io.BufferedReader name='demo-ascii.cast'>)}
headers = {'User-Agent': 'asciinema/2.0.2 CPython/3.9.9 Linux/5.14.18-100.fc33.x86_64-x86_64-with-glibc2.32', 'Accept': 'application/json'}
username = 'XXXX'
password = 'XXXX0f1-1d73-43fc-XX36-c9d7ZZZAAAA'

Very interesting – we could use definitely use the following fields to exercise the upload functionality without Python (obtained using args from the debugger):

• url = 'https://asciinema.org/api/asciicasts'

• headers = {'User-Agent': 'asciinema/2.0.2 CPython/3.9.9 Linux/5.14.18-100.fc33.x86_64-x86_64-with-glibc2.32', 'Accept': 'application/json'}

• username = 'XXXX'

• password = 'XXXX0f1-1d73-43fc-XX36-c9d7ZZZAAAA'

What exception we will get? We set 2 more breakpoints and let the debugger run until it reaches them:

(Pdb) b 81
Breakpoint 2 at /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py:81
(Pdb) b 85
Breakpoint 3 at /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py:85
(Pdb) c
> /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py(85)post()
-> raise HTTPConnectionError(str(e))
(Pdb) e
URLError(BrokenPipeError(32, 'Broken pipe'))

The type of the error is BrokenPipeError:

A subclass of ConnectionError, raised when trying to write on a pipe while the other end has been closed, or trying to write on a socket which has been shutdown for writing. Corresponds to errno EPIPE and ESHUTDOWN.

One last thing – do we read the whole file in memory before sending it to the website?

[josevnz@dmaf5 SuricataLog]$ python3 -m pdb /usr/bin/asciinema upload demo-ascii.cast 
> /usr/bin/asciinema(3)<module>()
-> import re
(Pdb) b /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py:49
Breakpoint 1 at /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py:49
(Pdb) c
> /usr/lib/python3.9/site-packages/asciinema/urllib_http_adapter.py(49)iter()
-> yield (data, len(data))
(Pdb) len(data)
12444283

12MB, not huge for today's computer memory but also not small.

Do you remember the parameters we managed to capture before with the help of the debugger (URL, user, and so on)? We now know enough to use a different tool (curl) to try to upload the file:

[josevnz@dmaf5 SuricataLog]$ curl --fail --http1.1 --verbose --user $USER:$(cat ~/.config/asciinema/install-id) https://asciinema.org/api/asciicasts --form asciicast=@demo-ascii.cast
*   Trying 109.107.37.0:443...
* Connected to asciinema.org (109.107.37.0) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.asciinema.org
*  start date: Mar  9 06:02:26 2022 GMT
*  expire date: Jun  7 06:02:25 2022 GMT
*  subjectAltName: host "asciinema.org" matched cert's "asciinema.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Server auth using Basic with user 'XXXX'
> POST /api/asciicasts HTTP/1.1
> Host: asciinema.org
> Authorization: Basic XXXXX=
> User-Agent: curl/7.71.1
> Accept: */*
> Content-Length: 12444495
> Content-Type: multipart/form-data; boundary=------------------------0d76dac3e1f8aed4
> Expect: 100-continue
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* Mark bundle as not supporting multiuse
* The requested URL returned error: 413 Request Entity Too Large
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (22) The requested URL returned error: 413 Request Entity Too Large

The error 413 Request Entity Too Large means:

The HTTP 413 Payload Too Large response status code indicates that the request entity is larger than limits defined by server; the server might close the connection or return a Retry-After header field.

So curl is much better than Python on telling us the truth about why our file was rejected.

How much data did we manage to transmit before our connection was cut off? Let's see if we can find that out using a packet sniffer.

How to use Wireshark and the SSLKEYLOGFILE to inspect the HTTP traffic

You can capture the traffic between your machine and the asciinema website using a network sniffer like Wireshark or the well known tcpdump.

The traffic will be encrypted as we use HTTPS, but using a feature supported by many programs known as a 'TLS master encryption secrets' you can decrypt the session. For that let's enable the feature on the client:

export SSLKEYLOGFILE=$HOME/keylogfile.txt

If is supported, the $SSLKEYLOGFILE file will be populated with the keys:

[josevnz@dmaf5 SuricataLog]$ export SSLKEYLOGFILE=$HOME/keylogfile.txt
[josevnz@dmaf5 SuricataLog]$ /usr/bin/asciinema upload demo-ascii.cast 
asciinema: upload failed: <urlopen error [Errno 32] Broken pipe>
asciinema: retry later by running: asciinema upload demo-ascii.cast
[josevnz@dmaf5 SuricataLog]$ ls -l $SSLKEYLOGFILE
-rw-rw-r-- 1 josevnz josevnz 832 Apr 21 21:02 /home/josevnz/keylogfile.txt

[josevnz@dmaf5 SuricataLog]$ cat /home/josevnz/keylogfile.txt

# TLS secrets log file, generated by OpenSSL / Python
SERVER_HANDSHAKE_TRAFFIC_SECRET 2987e32066d608a3de0cdd896f62801290045c2616abfaef5fac1c6986131847 4dd1a1bc1261a84886b28ee72798d89ba77d7de7051b3dcdafd548a621ed1124
EXPORTER_SECRET 2987e32066d608a3de0cdd896f62801290045c2616abfaef5fac1c6986131847 1ec8d94b7ec373a984abed25fa0dfaa6346fe67feea0516d7e2e46a666a12614
SERVER_TRAFFIC_SECRET_0 2987e32066d608a3de0cdd896f62801290045c2616abfaef5fac1c6986131847 e1d8fa6dba5eea00d4e52af0ce7e7007da0ade4c9dd9da3d9a060b55880531f1
CLIENT_HANDSHAKE_TRAFFIC_SECRET 2987e32066d608a3de0cdd896f62801290045c2616abfaef5fac1c6986131847 903bf381f927d783e72846201e87203ff130d9cf21f84cf0b923834d69c3fe76
CLIENT_TRAFFIC_SECRET_0 2987e32066d608a3de0cdd896f62801290045c2616abfaef5fac1c6986131847 495b5acf783869d74a7521e3b9c3f7bfc6dbc25e24ba95f684e96f6b2a435206
SERVER_HANDSHAKE_TRAFFIC_SECRET 82cab66e906c3cd3c58b3aeeecd66b2a12e521704d3e19e2f008550705e78e00 5a0d699640bd460530bd38148cf979e585b9a43c1bd545974561df18841fa5f4
EXPORTER_SECRET 82cab66e906c3cd3c58b3aeeecd66b2a12e521704d3e19e2f008550705e78e00 32b69cb41b8db36371e7d207a45e20d401bb05e0cd8bf492e3ace009e2845d12
SERVER_TRAFFIC_SECRET_0 82cab66e906c3cd3c58b3aeeecd66b2a12e521704d3e19e2f008550705e78e00 1f42b4392b2cc14789c4eaec4dae275c6a040ae3b11fc6bba58c90c7b80caa96
CLIENT_HANDSHAKE_TRAFFIC_SECRET 82cab66e906c3cd3c58b3aeeecd66b2a12e521704d3e19e2f008550705e78e00 bd93073bda56e559743a1f1ffc48c062089addcfc007c7defe08c28ac0ee6287
CLIENT_TRAFFIC_SECRET_0 82cab66e906c3cd3c58b3aeeecd66b2a12e521704d3e19e2f008550705e78e00 32b615c0dd25cb7b430a0cff44871e3263bd67af973e4b2f7fb19aab4df468d8
SERVER_HANDSHAKE_TRAFFIC_SECRET 68dcc859bc4edb51354a9f583e036d0b2787a337ee894e253925e273a5cd3889 a52a20827ce04dfc4ee557608ed5a0bfb6794ace0c4a1b69a1d56e5f16d8570b
EXPORTER_SECRET 68dcc859bc4edb51354a9f583e036d0b2787a337ee894e253925e273a5cd3889 8179afb8e7c7a77e35143c40a6bb62ccea2e644e48cc95b91b05f525bc59ada7
SERVER_TRAFFIC_SECRET_0 68dcc859bc4edb51354a9f583e036d0b2787a337ee894e253925e273a5cd3889 3d4abf6a9ea06395648a45428ca78c24962d8cc11440fe1d72f035ae35e61010
CLIENT_HANDSHAKE_TRAFFIC_SECRET 68dcc859bc4edb51354a9f583e036d0b2787a337ee894e253925e273a5cd3889 1d812a6c3c012a8fa4a6017ee573b47a5b361d15b861938ebca9194ecbc2a250
CLIENT_TRAFFIC_SECRET_0 68dcc859bc4edb51354a9f583e036d0b2787a337ee894e253925e273a5cd3889 6348a88dc9b6a350d72a7154140b824db80ba4f48c9e1fabcee76da8d248b041

Good. Next step is to capture the traffic. We'll use tcpdump with a simple expression to filter out the captured traffic:

[josevnz@dmaf5 temp]$ sudo tcpdump -i eno1 -v -v -v 'host asciinema.org' -w ~/temp/asciinema.org.pcap
dropped privs to tcpdump
tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes

And in another window run the asciinema client (we'll do it twice to have more data):

[josevnz@dmaf5 SuricataLog]$ /usr/bin/asciinema upload demo-ascii.cast 
asciinema: upload failed: <urlopen error [Errno 32] Broken pipe>
asciinema: retry later by running: asciinema upload demo-ascii.cast
[josevnz@dmaf5 SuricataLog]$ 
[josevnz@dmaf5 SuricataLog]$ /usr/bin/asciinema upload demo-ascii.cast 
asciinema: upload failed: <urlopen error [Errno 104] Connection reset by peer>
asciinema: retry later by running: asciinema upload demo-ascii.cast

Now kill the tcpdump capture on the other window:

tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C113 packets captured
118 packets received by filter
0 packets dropped by kernel

Let's replay the pcap file to see what got recorded:

[josevnz@dmaf5 temp]$ tcpdump -r ~/temp/asciinema.org.pcap
reading from file /home/josevnz/temp/asciinema.org.pcap, link-type EN10MB (Ethernet), snapshot length 262144
07:17:18.244941 IP dmaf5.home.59896 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [S], seq 1651239781, win 64240, options [mss 1460,sackOK,TS val 3293505858 ecr 0,nop,wscale 7], length 0
07:17:18.337023 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59896: Flags [S.], seq 2395275599, ack 1651239782, win 65160, options [mss 1460,sackOK,TS val 3934370169 ecr 3293505858,nop,wscale 7], length 0
07:17:18.337070 IP dmaf5.home.59896 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [.], ack 1, win 502, options [nop,nop,TS val 3293505950 ecr 3934370169], length 0
07:17:18.337643 IP dmaf5.home.59896 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3293505951 ecr 3934370169], length 517
07:17:18.429273 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59896: Flags [.], ack 518, win 506, options [nop,nop,TS val 3934370263 ecr 3293505951], length 0
07:17:18.433850 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59896: Flags [.], seq 1:1449, ack 518, win 506, options [nop,nop,TS val 3934370267 ecr 3293505951], length 1448
07:17:18.433863 IP dmaf5.home.59896 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [.], ack 1449, win 501, options [nop,nop,TS val 3293506047 ecr 3934370267], length 0
07:17:18.433966 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59896: Flags [P.], seq 1449:2897, ack 518, win 506, options [nop,nop,TS val 3934370267 ecr 3293505951], length 1448
07:17:18.433981 IP dmaf5.home.59896 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [.], ack 2897, win 496, options [nop,nop,TS val 3293506047 ecr 3934370267], length 0
07:17:18.434089 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59896: Flags [.], seq 2897:4345, ack 518, win 506, options [nop,nop,TS val 3934370267 ecr 3293505951], length 1448
...
07:17:30.612523 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59898: Flags [.], ack 11148, win 501, options [nop,nop,TS val 3934382447 ecr 3293518134], length 0
07:17:30.612524 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59898: Flags [.], ack 12596, win 501, options [nop,nop,TS val 3934382447 ecr 3293518134], length 0
07:17:30.612558 IP dmaf5.home.59898 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [.], seq 35764:37212, ack 4724, win 499, options [nop,nop,TS val 3293518226 ecr 3934382447], length 1448
07:17:30.612563 IP dmaf5.home.59898 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [P.], seq 37212:38660, ack 4724, win 499, options [nop,nop,TS val 3293518226 ecr 3934382447], length 1448
07:17:30.612637 IP dmaf5.home.59898 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [.], seq 38660:40108, ack 4724, win 499, options [nop,nop,TS val 3293518226 ecr 3934382447], length 1448
07:17:30.612643 IP dmaf5.home.59898 > cip-109-107-37-0.gb1.brightbox.com.https: Flags [P.], seq 40108:41556, ack 4724, win 499, options [nop,nop,TS val 3293518226 ecr 3934382447], length 1448
07:17:30.613064 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59898: Flags [P.], seq 4724:5080, ack 12596, win 501, options [nop,nop,TS val 3934382448 ecr 3293518134], length 356
07:17:30.613106 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59898: Flags [P.], seq 5080:5104, ack 12596, win 501, options [nop,nop,TS val 3934382448 ecr 3293518134], length 24
07:17:30.614231 IP cip-109-107-37-0.gb1.brightbox.com.https > dmaf5.home.59898: Flags [R.], seq 5104, ack 12596, win 501, options [nop,nop,TS val 3934382448 ecr 3293518134], length 0

Time to fire up wireshark. I like to use a GUI for this as the filtering capabilities are nice, and you can explore the contents of the PCAP file much easier.

The contents of the traffic capture:

So we follow the first time we got a TLS hello message, right-click on the protocol preferences -> Transport Layer Security and then "pre-Master-Secret log filename":

Now time for the fun part. If you right-click the first hello message and say "follow TLS stream" a new window will open the whole conversation to the moment we got our connection reset, no encryption!

So we only managed to sent 33 KB before being cut-off by the asciinema server. How rude! :satisfied:

Because the data payload is not so big I will show it to you next, make sure you pay attention to the following:

1. I changed the Authorization: Basic contents as I don't want to leak my user/password encoded in base64.

2. Content-Length: 12444474. That's how asciinema knows how big is the file we want to upload, so the server rejects it.

3. Asciinema uses Nginx.

4. You can see the close message at the end (entity too large).

POST /api/asciicasts HTTP/1.1
Accept-Encoding: identity
Content-Length: 12444474
Host: asciinema.org
User-Agent: asciinema/2.0.2 CPython/3.9.9 Linux/5.14.18-100.fc33.x86_64-x86_64-with-glibc2.32
Accept: application/json
Content-Type: multipart/form-data; boundary=d5c6b2543ee94511943126c6a3c5d33a
Authorization: Basic XXXXX=
Connection: close

--d5c6b2543ee94511943126c6a3c5d33a
Content-Disposition: form-data; name="asciicast"; filename="ascii.cast"
Content-Type: application/octet-stream

{"version": 2, "width": 203, "height": 32, "timestamp": 1650568938, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}}
[0.191182, "o", "\u001b]777;notify;Command completed;eve_log.py --format table --timestamp '2022-02-23T18:22:24.405139+0000' test/eve.json\u001b\\\u001b]777;precmd\u001b\\\u001b]0;josevnz@dmaf5:~/SuricataLog-Logging-features-branch\u001b\\"]
[0.19215, "o", "\u001b]7;file://dmaf5/home/josevnz/SuricataLog-Logging-features-branch\u001b\\"]
[0.192399, "o", "[josevnz@dmaf5 SuricataLog-Logging-features-branch]$ "]
[1.000538, "o", "Let me show you how you can filter your Suricata alerts, displaying the results in different formats"]
[4.506902, "o", "\r\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C"]
[4.921813, "o", "\u001b[1@#"]
[5.170393, "o", "\u001b[1@ "]
[5.538486, "o", "\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C\u001b[C"]
[6.914337, "o", "\r\n"]
[6.918708, "o", "\u001b]777;notify;Command completed;# Let me show you how you can filter your Suricata alerts, displaying the results in different formats\u001b\\\u001b]777;precmd\u001b\\\u001b]0;josevnz@dmaf5:~/SuricataLog-Logging-features-branch\u001b\\"]
[6.920219, "o", "\u001b]7;file://dmaf5/home/josevnz/SuricataLog-Logging-features-branch\u001b\\"]
[6.920352, "o", "[josevnz@dmaf5 SuricataLog-Logging-features-branch]$ "]
[8.202111, "o", "1"]
[8.658197, "o", ")"]
[8.962176, "o", " "]
[10.153862, "o", "A"]
[10.409632, "o", " "]
[10.61679, "o", "n"]
[10.777002, "o", "i"]
[10.881112, "o", "c"]
[10.952884, "o", "e"]
[11.088641, "o", " "]
[11.201045, "o", "t"]
[11.466022, "o", "a"]
[11.553785, "o", "b"]
[11.818412, "o", "l"]
[11.961808, "o", "e"]
[13.51443, "o", "\r\n"]
[13.514675, "o", "bash: syntax error near unexpected token `)'\r\n"]
[13.518913, "o", "\u001b]777;notify;Command completed;1) A nice table\u001b\\\u001b]777;precmd\u001b\\\u001b]0;josevnz@dmaf5:~/SuricataLog-Logging-features-branch\u001b\\"]
[13.520551, "o", "\u001b]7;file://dmaf5/home/josevnz/SuricataLog-Logging-features-branch\u001b\\"]
[13.52072, "o", "[josevnz@dmaf5 SuricataLog-Logging-features-branch]$ "]
[22.176716, "o", "eve_log.py --format table --timestamp '2022-02-23T18:22:24.405139+0000' test/eve.jso"]
[24.202009, "o", "n"]
[26.097822, "o", "\r\n"]
[26.098024, "o", "\u001b]777;preexec\u001b\\"]
[26.312676, "o", "\u001b[?1049h\u001b[H\u001b[?1000h\u001b[?1003h\u001b[?1015h\u001b[?1006h\u001b[?25l\u001b[?1003h\r\n"]
[26.314059, "o", "\u001bP=1s\u001b\\\u001b[H\u001b[H                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                               "]
[26.314299, "o", "            \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                              "]
[26.314387, "o", "             \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                             "]
[26.314455, "o", "              \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                            "]
[26.314502, "o", "               \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                           "]
[26.31456, "o", "                \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                          "]
[26.314616, "o", "                 \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \u001bP=2s\u001b\\"]
[26.31467, "o", "\u001bP=1s\u001b\\\u001b[H\u001b[H                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                               "]
[26.314714, "o", "            \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                              "]
[26.314781, "o", "             \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                             "]
[26.314843, "o", "              \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                            "]
[26.314902, "o", "               \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                           "]
[26.314957, "o", "                \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                          "]
[26.315012, "o", "                 \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \u001bP=2s\u001b\\"]
[26.316033, "o", "\u001b[?25l"]
[26.318086, "o", "\r\u001b[2KParsing test/eve.json \u001b[38;5;237m........................................................................................................................\u001b[0m \u001b[35m  0%\u001b[0m \u001b[36m-:--:--\u001b[0m"]
[26.378123, "o", "\r\u001b[2KParsing test/eve.json \u001b[38;2;114;156;31m........................................................................................................................\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m0:00:00\u001b[0m\r\n\u001b[?25h"]
[26.390312, "o", "\u001bP=1s\u001b\\\u001b[H\u001b[H                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                               "]
[26.39044, "o", "            \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                              "]
[26.390499, "o", "             \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                             "]
[26.390559, "o", "              \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                            "]
[26.390615, "o", "               \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                           "]
[26.39064, "o", "                \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                          "]
[26.390719, "o", "                 \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \u001bP=2s\u001b\\"]
[26.390868, "o", "\u001bP=1s\u001b\\\u001b[H\u001b[H                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                               "]
[26.390893, "o", "            \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                              "]
[26.390944, "o", "             \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                             "]
[26.391027, "o", "              \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                            "]
[26.391091, "o", "               \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                           "]
[26.391116, "o", "                \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                          "]
[26.391172, "o", "                 \r\n                                                                                                                                                                                                           \r\n                                                                                                                                                                                                           \u001bP=2s\u001b\\"]
[26.431391, "o", "\u001bP=1s\u001b\\\u001b[H\u001b[3m                                                                                      Suricata alerts for 2022-02-23 18:22:24.405139, logs=test/eve.json                                                   \u001b[0m\r\n.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................\r\n...\u001b[1;35m \u001b[0m\u001b[1;35mTimestamp                      \u001b[0m\u001b[1;35m \u001b[0m...\u001b[1;35m \u001b[0m\u001b[1;35mSeverity\u001b[0m\u001b[1;35m \u001b[0m...\u001b[1;35m \u001b[0m\u001b[1;35mSignature                                           \u001b"]
[26.431543, "o", "[0m\u001b[1;35m \u001b[0m...\u001b[1;35m \u001b[0m\u001b[1;35mProtocol\u001b[0m\u001b[1;35m \u00HTTP/1.1 413 Request Entity Too Large
Content-Length: 176
Content-Type: text/html
Date: Fri, 22 Apr 2022 11:17:19 GMT
Server: nginx
Connection: close

<html>
<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx</center>
</body>
</html>

What is next for you?

So next time you have an issue with a program that is installed on your system you will know what to check.

We covered 3 ways to investigate an issue with an application that uploads a file to a remote website using HTTPS:

1. Using strace

2. If the program is a Python script, then there is a good chance you can read the code yourself and run the script through the debugger, step by step, to understand the issue. This is probably the most time-consuming way, but also it is the most rewarding as you learn how other good developers think!

3. And finally, we captured the encrypted traffic between us and the remote site and analyzed the upload. By enabling certain special features we were able to decrypt and replay the traffic, confirming our findings from the previous two interactions.

This list of techniques is not exhaustive, but for some cases like this they will give you a good start.

As usual please share your feedback! Let's have a conversation so everybody learns a little.

Kismet is perfect for detecting anomalies and certain types of attack – but what if I want to analyze the traffic and look for abnormal patterns or patterns that could indicate an attack?

And Intrusion Detection System (IDS) is:

...a device or software application that monitors a network or systems for malicious activity or policy violations.

I used a good IDS in the past called Snort V2, I'm aware than Snort 3 is out. But there is a pretty clear warning about running it on a machine without much memory:

While Snort can compile on almost all *nix based machines, it is not recommended that you compile Snort on a low power or low RAM machine. Snort requires memory to run and to properly analyze as much traffic as possible.

And

Snort does not officially support any particular OS.

Not exactly a reason to dislike it, but I feel more confident when a vendor tells me than my OS is in their supported platform list. I do also have more recent experience setting up with the open source tool Suricata, so I decided to give it a more serious try to keep tabs on my local network and alert me if any suspicious activity was detected.

Poking around I found than for my local network, 8 GB of RAM will be sufficient along with my Linux distribution:

josevnz@raspberrypi:~$ lsb_release --release
Release:    20.04

My version of Ubuntu is supported out of the box.

The choice is yours. In my case it felt better to use Suricata than Snort. As usual, you need to plan around your hardware, your use cases, and the features offered by the tools (including commercial support).

Table of Contents

• Quick Installation

• Where you should connect your Raspberry Pi 4 with Suricata

• How to Set Up Suricata

• How to Tune Up Suricata

• Making Sense of All the Alerts

• What Did We Learn and What is Next?

Quick Installation

Installation is explained in detail here, so I will only put here the quick installation steps I used on my machine:

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:oisf/suricata-stable
sudo apt-get update
sudo apt-get install suricata

Suricata is a Complex Beast

You can use Suricata to detect and alert you about anomalies in your network traffic (IDS) or you can proactively drop suspicious connections when working in Intrusion Prevention System (IPS).

It can also capture network traffic and store it in PCAP format for later analysis (be careful as you can eat your disk space pretty fast).

We will keep things simple, and for now will take a more passive approach and get alerts when an intrusion is detected (sticking to IDS mode) in this tutorial.

Where you should connect your RaspBerryPI 4 with Suricata?

Ideally you want to put your Suricata sensor close to your home router. One way to do it is to connect all the devices (including your home router) to a common switch, and then mirror the traffic that goes into/out from the home router into a port on the switch. Suricata will be connected to that port, listening to all the traffic.

If you wanted to run Suricata as an IPS then the connectivity would have to be different, but this is not the intended use in this tutorial.

How to Set Up Suricata

Ideally the best place to put Suricata is between a firewall and the rest of the servers in your home network.

In this scenario let's assume than it is not possible because there is no firewall (OK, that will be your ISP router, but you cannot run Suricata there). So the next best thing is the wired network interface connected to it (in my case eth0).

The /etc/suricata/suricata.yaml file contains the defaults. I'll show here what I overrode:

root@raspberrypi:~# grep -in1 af-p /etc/suricata/suricata.yaml 
580-# Linux high speed capture support
581:af-packet:
582-  - interface: eth0
root@raspberrypi:~# grep -in 'HOME_NET: "' /etc/suricata/suricata.yaml |grep -v '#'
15:    HOME_NET: "[192.168.1.0/24]"

Start Suricata:

root@raspberrypi:~# systemctl start suricata.service
root@raspberrypi:~# systemctl status suricata.service
● suricata.service - LSB: Next Generation IDS/IPS
     Loaded: loaded (/etc/init.d/suricata; generated)
     Active: active (running) since Sun 2022-04-10 23:49:00 UTC; 24h ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 10 (limit: 9257)
     CGroup: /system.slice/suricata.service
             └─1834983 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -vvv

Apr 10 23:49:00 raspberrypi systemd[1]: Starting LSB: Next Generation IDS/IPS...
Apr 10 23:49:00 raspberrypi suricata[1834973]: Starting suricata in IDS (af-packet) mode... done.
Apr 10 23:49:00 raspberrypi systemd[1]: Started LSB: Next Generation IDS/IPS.

The important details go into the file '/var/log/suricata/eve.json'. Mine started to grow surprisingly fast after starting Suricata:

{"timestamp":"2022-04-10T23:49:32.527488+0000","event_type":"stats","stats":{"uptime":32,"capture":{"kernel_packets":113,"kernel_drops":0,"errors":0},"decoder":{"pkts":126,"bytes":17986,"invalid":0,"ipv4":30,"ipv6":74,"ethernet":126,"chdlc":0,"raw":0,"null":0,"sll":0,"tcp":4,"udp":30,"sctp":0,"icmpv4":0,"icmpv6":70,"ppp":0,"pppoe":0,"geneve":0,"gre":0,"vlan":0,"vlan_qinq":0,"vxlan":0,"vntag":0,"ieee8021ah":0,"teredo":0,"ipv4_in_ipv6":0,"ipv6_in_ipv6":0,"mpls":0,"avg_pkt_size":142,"max_pkt_size":392,"max_mac_addrs_src":0,"max_mac_addrs_dst":0,"erspan":0,"event":{"ipv4":{"pkt_too_small":0,"hlen_too_small":0,"iplen_smaller_than_hlen":0,"trunc_pkt":0,"opt_invalid":0,"opt_invalid_len":0,"opt_malformed":0,"opt_pad_required":0,"opt_eol_required":0,"opt_duplicate":0,"opt_unknown":0,"wrong_ip_version":0,"icmpv6":0,"frag_pkt_too_large":0,"frag_overlap":0,"frag_ignored":0},"icmpv4":{"pkt_too_small":0,"unknown_type":0,"unknown_code":0,"ipv4_trunc_pkt":0,"ipv4_unknown_ver":0},"icmpv6":{"unknown_type":0,"unknown_code":0,"pkt_too_small":0,"ipv6_unknown_version":0,"ipv6_trunc_pkt":0,"mld_message_with_invalid_hl":0,"unassigned_type":0,"experimentation_type":0},"ipv6":{"pkt_too_small":0,"trunc_pkt":0,"trunc_exthdr":0,"exthdr_dupl_fh":0,"exthdr_useless_fh":0,"exthdr_dupl_rh":0,"exthdr_dupl_hh":0,"exthdr_dupl_dh":0,"exthdr_dupl_ah":0,"exthdr_dupl_eh":0,"exthdr_invalid_optlen":0,"wrong_ip_version":0,"exthdr_ah_res_not_null":0,"hopopts_unknown_opt":0,"hopopts_only_padding":0,"dstopts_unknown_opt":0,"dstopts_only_padding":0,"rh_type_0":0,"zero_len_padn":21,"fh_non_zero_reserved_field":0,"data_after_none_header":0,"unknown_next_header":0,"icmpv4":0,"frag_pkt_too_large":0,"frag_overlap":0,"frag_invalid_length":0,"frag_ignored":0,"ipv4_in_ipv6_too_small":0,"ipv4_in_ipv6_wrong_version":0,"ipv6_in_ipv6_too_small":0,"ipv6_in_ipv6_wrong_version":0},"tcp":{"pkt_too_small":0,"hlen_too_small":0,"invalid_optlen":0,"opt_invalid_len":0,"opt_duplicate":0},"udp":{"pkt_too_small":0,"hlen_too_small":0,"hlen_invalid":0},"sll":{"pkt_too_small":0},"ethernet":{"pkt_too_small":0},"ppp":{"pkt_too_small":0,"vju_pkt_too_small":0,"ip4_pkt_too_small":0,"ip6_pkt_too_small":0,"wrong_type":0,"unsup_proto":0},"pppoe":{"pkt_too_small":0,"wrong_code":0,"malformed_tags":0},"gre":{"pkt_too_small":0,"wrong_version":0,"version0_recur":0,"version0_flags":0,"version0_hdr_too_big":0,"version0_malformed_sre_hdr":0,"version1_chksum":0,"version1_route":0,"version1_ssr":0,"version1_recur":0,"version1_flags":0,"version1_no_key":0,"version1_wrong_protocol":0,"version1_malformed_sre_hdr":0,"version1_hdr_too_big":0},"vlan":{"header_too_small":0,"unknown_type":0,"too_many_layers":0},"ieee8021ah":{"header_too_small":0},"vntag":{"header_too_small":0,"unknown_type":0},"ipraw":{"invalid_ip_version":0},"ltnull":{"pkt_too_small":0,"unsupported_type":0},"sctp":{"pkt_too_small":0},"mpls":{"header_too_small":0,"pkt_too_small":0,"bad_label_router_alert":0,"bad_label_implicit_null":0,"bad_label_reserved":0,"unknown_payload_type":0},"vxlan":{"unknown_payload_type":0},"geneve":{"unknown_payload_type":0},"erspan":{"header_too_small":0,"unsupported_version":0,"too_many_vlan_layers":0},"dce":{"pkt_too_small":0},"chdlc":{"pkt_too_small":0}},"too_many_layers":0},"flow":{"memcap":0,"tcp":1,"udp":20,"icmpv4":0,"icmpv6":15,"tcp_reuse":0,"get_used":0,"get_used_eval":0,"get_used_eval_reject":0,"get_used_eval_busy":0,"get_used_failed":0,"wrk":{"spare_sync_avg":100,"spare_sync":4,"spare_sync_incomplete":0,"spare_sync_empty":0,"flows_evicted_needs_work":0,"flows_evicted_pkt_inject":0,"flows_evicted":0,"flows_injected":0},"mgr":{"full_hash_pass":1,"closed_pruned":0,"new_pruned":0,"est_pruned":0,"bypassed_pruned":0,"rows_maxlen":1,"flows_checked":4,"flows_notimeout":4,"flows_timeout":0,"flows_timeout_inuse":0,"flows_evicted":0,"flows_evicted_needs_work":0},"spare":9600,"emerg_mode_entered":0,"emerg_mode_over":0,"memuse":11668608},"defrag":{"ipv4":{"fragments":0,"reassembled":0,"timeouts":0},"ipv6":{"fragments":0,"reassembled":0,"timeouts":0},"max_frag_hits":0},"flow_bypassed":{"local_pkts":0,"local_bytes":0,"local_capture_pkts":0,"local_capture_bytes":0,"closed":0,"pkts":0,"bytes":0},"tcp":{"sessions":0,"ssn_memcap_drop":0,"pseudo":0,"pseudo_failed":0,"invalid_checksum":0,"no_flow":0,"syn":0,"synack":0,"rst":0,"midstream_pickups":0,"pkt_on_wrong_thread":0,"segment_memcap_drop":0,"stream_depth_reached":0,"reassembly_gap":0,"overlap":0,"overlap_diff_data":0,"insert_data_normal_fail":0,"insert_data_overlap_fail":0,"insert_list_fail":0,"memuse":2424832,"reassembly_memuse":393216},"detect":{"engines":[{"id":0,"last_reload":"2022-04-10T23:49:00.377030+0000","rules_loaded":0,"rules_failed":0}],"alert":0},"app_layer":{"flow":{"http":0,"ftp":0,"smtp":0,"tls":0,"ssh":0,"imap":0,"smb":0,"dcerpc_tcp":0,"dns_tcp":0,"nfs_tcp":0,"ntp":1,"ftp-data":0,"tftp":0,"ikev2":0,"krb5_tcp":0,"dhcp":0,"snmp":0,"sip":0,"rfb":0,"mqtt":0,"rdp":0,"failed_tcp":0,"dcerpc_udp":0,"dns_udp":0,"nfs_udp":0,"krb5_udp":0,"failed_udp":19},"tx":{"http":0,"ftp":0,"smtp":0,"tls":0,"ssh":0,"imap":0,"smb":0,"dcerpc_tcp":0,"dns_tcp":0,"nfs_tcp":0,"ntp":1,"ftp-data":0,"tftp":0,"ikev2":0,"krb5_tcp":0,"dhcp":0,"snmp":0,"sip":0,"rfb":0,"mqtt":0,"rdp":0,"dcerpc_udp":0,"dns_udp":0,"nfs_udp":0,"krb5_udp":0},"expectations":0},"http":{"memuse":0,"memcap":0},"ftp":{"memuse":0,"memcap":0},"file_store":{"open_files":0}}}

Holy Priceless Collection of Etruscan Snoods!, Batman. How do we tune Suricata to avoid this overwhelming amount of information?

For now let's stop it while we figure it out.

How to Tune Up Suricata

Make sure the settings of suricata.yaml make sense for a home network:

sudo -i
# And a YAML linter so we can make sure our Suricata configuration files are good
apt-get install yamllint
cp -v -p  /etc/suricata/suricata.yaml /etc/suricata/suricata.yaml.orig

Note that I provide here a linted and clean version of my [suricata.yaml](file:///home/josevnz/SuricataLog/etc/suricata/suricata.yaml) file.

How to tame the /var/log/suricata/eve.json file

This is the file were we can learn in detail what triggered an alert. But it can grow VERY fast, depending on your traffic and event rules configuration.

So using logrotate (comes installed as part of Ubuntu), do this:

# Keep a week of logs, 1 GB of size.
# Always test your config: logrotate -vdf /etc/logrotate.d/suricata
/var/log/suricata/*.log /var/log/suricata/*.json {
    daily
    maxsize 1G
    rotate 7
    missingok
    nocompress
    create
    sharedscripts
    postrotate
        systemctl restart suricata.service
    endscript
}

How to help Suricata to do its job using emerging threats rules

We can tune Suricata using the ET OPEN Ruleset. Because threats change all the time, you need to automate their download and updating.

So install it first:

sudo -i
python3 -m venv ~/virtualenv/suricata
. ~/virtualenv/suricata/bin/activate
pip install --upgrade pip
pip install --upgrade suricata-update
suricata-update
# Also, install jq so we can see the contents of the eve.json file nicely formatted
apt-get install jq

Let's run it by hand and see how the rules are updated by the tool:

For our home network, we will download these rules once a day. A simple Cron job will do the trick:

crontab -e
# Run Suricata update once a day, 
# per https://rules.emergingthreats.net/OPEN_download_instructions.html
# Also will update at a different time than the log rotation, to avoid a race condition
# while rotating the logs. Note than we do not need to restart suricata
0 30 * * * . ~/virtualenv/suricata/bin/activate && suricata-update && suricatasc -c reload-rules

Let's start Suricata again, so we can test some rules:

What is inside the /var/log/suricata/eve.json file?

The file packs quite a bit of information, which is described in detail here:

{"timestamp":"2022-04-15T20:52:05.026189+0000","flow_id":1378250082748552,"in_iface":"eth0","event_type":"flow","src_ip":"192.168.1.1","src_port":59317,"dest_ip":"239.255.255.250","dest_port":1900,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":378,"bytes_toclient":0,"start":"2022-04-15T20:50:32.264328+0000","end":"2022-04-15T20:50:32.264328+0000","age":0,"state":"new","reason":"timeout","alerted":false}}
{"timestamp":"2022-04-15T20:52:05.026418+0000","flow_id":2222739437411106,"in_iface":"eth0","event_type":"flow","src_ip":"192.168.1.1","src_port":60890,"dest_ip":"239.255.255.250","dest_port":1900,"proto":"UDP","app_proto":"failed","flow":{"pkts_toserver":1,"pkts_toclient":0,"bytes_toserver":376,"bytes_toclient":0,"start":"2022-04-15T20:50:32.482082+0000","end":"2022-04-15T20:50:32.482082+0000","age":0,"state":"new","reason":"timeout","alerted":false}}

If you are casually inspecting the contents of the file in real time, I suggest you use jq (test your filters on jqplay.org) and show a few fields of interest:

Going forward we will focus on the alerts, so we can just filter out by that type of event:

jq 'select(.event_type=="alert")' /var/log/suricata/eve.json

The Suricata folks have put together a nice page with examples that you should check out.

How to test Suricata installation

Tools of the trade: Wireshark, tcpreplay, and PCAP files

We will use some traffic capture files, in PCAP format. So what is a PCAP file?

In the late 1980's, Van Jacobson, Steve McCanne, and others at the Network Research Group at Lawrence Berkeley National Laboratory developed the tcpdump program to capture and dissect network traces.

The code to capture traffic, using low-level mechanisms in various operating systems, and to read and write network traces to a file was later put into a library named libpcap.

And we will use a tool to inspect the contents of the PCAP file. Wireshark is a powerful traffic analysis tool, and we will use tcpreplay to trigger the Suricata alerts by playing a PCAP file with suspicious activity:

# On Ubuntu, Debian: sudo apt-get install wireshark tcpreplay
sudo dnf install -y wireshark tcpreplay

The best way to learn how the bad actors operate is to see their footprints. You should definitely head to https://www.malware-traffic-analysis.net/ and download some samples, an even better practice with their PCAP analysis exercises.

WARNING: You will be downloading files that are dangerous:

Use this website at your own risk! If you download or use of any information from this website, you assume complete responsibility for any resulting loss or damage.

So be careful and responsible when using this network traffic capturer.

No rules are enabled by default?

How we can check if that is the case? I'll show you next:

Once you enable the rules (suricata-update list-sources --free; uricata-update enable-source source; suricata-update list-enabled-sources) you can tell Suricata to reload the rules without a reboot:

root@raspberrypi:~# suricatasc -c reload-rules
{"message": "done", "return": "OK"}

2022-02-23 - TRAFFIC ANALYSIS EXERCISE - SUNNYSTATION

Let's see if we can trigger Suricata using this specific threat (it is relative new).

Start by downloading 2022-02-23-traffic-analysis-exercise.pcap.zip (the password is on the [about page](file:///home/josevnz/SuricataLog/)).

insta_dir="$HOME/Downloads/malware/"
mkdir --parent --verbose "$insta_dir"
url="https://www.malware-traffic-analysis.net/2022/02/23/2022-02-23-traffic-analysis-exercise.pcap.zip"
exercise=$(basename $url)
curl --fail --location --output "$insta_dir/$exercise" $url
# Be ready to put the password :-)
cd $insta_dir && unzip $exercise

What is inside? We can check with capinfos to get some insight on the file we just downloaded:

[josevnz@dmaf5 malware]$ capinfos 2022-02-23-traffic-analysis-exercise.pcap
File name:           2022-02-23-traffic-analysis-exercise.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
File timestamp precision:  microseconds (6)
Packet size limit:   file hdr: 65535 bytes
Number of packets:   30k
File size:           19MB
Data size:           19MB
Capture duration:    2680.736661 seconds
First packet time:   2022-02-23 13:22:24.405139
Last packet time:    2022-02-23 14:07:05.141800
Data byte rate:      7,191 bytes/s
Data bit rate:       57kbps
Average packet size: 642.09 bytes
Average packet rate: 11 packets/s
SHA256:              eefc7e61b50e7846f5a3282d7645539d7b2b4b85aa08a09d0b823896c9449d1f
RIPEMD160:           a8d84d262e37563c179e9ca52cdc6aae271efd9c
SHA1:                fdfa0d0edfe0cbcc0c1400fbe6ac61ff40942755
Strict time order:   True
Number of interfaces in file: 1
Interface #0 info:
                     Encapsulation = Ethernet (1 - ether)
                     Capture length = 65535
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Number of stat entries = 0
                     Number of packets = 30023

Will use a [small wrapper](file:///home/josevnz/SuricataLog/scripts/replay_pcap_file.sh) around tcpreplay to replay our PCAP file:

#!/bin/bash
:<<DOC
Script to replay a PCAP file at accelerated pace on the default network interface
Author: Jose Vicente Nunez (kodegeek.com@protonmail.com)
DOC
default_dev=$(ip route show| grep default| sort -n -k 9| head -n 1| cut -f5 -d' ')|| exit 100
if [ "$(id --name --user)" != "root" ]; then
  echo "ERROR: I need to be root to inject the PCAP contents into '$default_dev'"
  echo "Maybe 'sudo $0 $*'?"
  exit 100
fi
for util in tcpreplay ip; do
  if ! type -p $util > /dev/null 2>&1; then
    echo "Please put $util on the PATH and try again!"
    exit 100
  fi
done
:<<DOC
We may have more than one 'default' route, so we sort by priority and pick the one with the
preferred metric:
default via 192.168.1.1 dev eno1 proto dhcp metric 100 <----- PICK ME!!!
default via 192.168.1.1 dev wlp4s0 proto dhcp metric 600
DOC
for pcap in "$@"; do
  if [ -f "$pcap" ]; then
    if ! tcpreplay --stats 5 --intf1 "$default_dev" --multiplier 24 "$pcap"; then
      echo "ERROR: Will not try to replay any pending PCAP files due previous errors"
      exit 100
    fi
  fi
done

Let it replay until it reaches the end of the file:

root@raspberrypi:~# tcpreplay --stats 5 --intf1 eth0 --multiplier 24 ~josevnz/Downloads/malware/2022-02-23-traffic-analysis-exercise.pcap 
Test start: 2022-04-16 17:51:40.673394 ...
Actual: 3783 packets (1075843 bytes) sent in 5.03 seconds
Rated: 213624.5 Bps, 1.70 Mbps, 751.17 pps
Actual: 6959 packets (3325918 bytes) sent in 10.04 seconds
Rated: 331191.4 Bps, 2.64 Mbps, 692.96 pps
Actual: 8627 packets (4464002 bytes) sent in 15.14 seconds
Rated: 294744.2 Bps, 2.35 Mbps, 569.61 pps
Actual: 10975 packets (6331901 bytes) sent in 20.21 seconds
Rated: 313180.5 Bps, 2.50 Mbps, 542.83 pps
Actual: 13148 packets (7870783 bytes) sent in 25.26 seconds
Rated: 311561.9 Bps, 2.49 Mbps, 520.45 pps
Actual: 14500 packets (8612630 bytes) sent in 30.43 seconds
...
Actual: 24467 packets (14960314 bytes) sent in 110.83 seconds
Rated: 134978.5 Bps, 1.07 Mbps, 220.75 pps
Test complete: 2022-04-16 17:53:33.735188
Actual: 30023 packets (19277433 bytes) sent in 113.06 seconds
Rated: 170503.5 Bps, 1.36 Mbps, 265.54 pps
Statistics for network device: eth0
    Successful packets:        30023
    Failed packets:            0
    Truncated packets:         0
    Retried packets (ENOBUFS): 0
    Retried packets (EAGAIN):  0

And eventually we get a few alerts:

"2022-04-16T17:52:20.134763+0000,dns,1296231906414153,172.16.0.170:53806,172.16.0.52:53"
"2022-04-16T17:52:20.286785+0000,dns,293726410006593,172.16.0.170:50935,172.16.0.52:53"
"2022-04-16T17:52:20.290084+0000,dns,293726410006593,172.16.0.170:50935,172.16.0.52:53"
"2022-04-16T17:52:20.520858+0000,alert,1626224981242326,172.16.0.149:49795,172.16.0.52:139"
"2022-04-16T17:52:21.784804+0000,alert,1992149752477936,172.16.0.149:49796,172.16.0.52:139"
"2022-04-16T17:52:22.142041+0000,flow,1739064507071469,172.16.0.149:5353,224.0.0.251:5353"
"2022-04-16T17:52:22.351091+0000,dns,2078727703255923,172.16.0.149:51367,172.16.0.52:53"
"2022-04-16T17:52:22.351260+0000,dns,181632058678300,172.16.0.149:64943,172.16.0.52:53"
"2022-04-16T17:52:22.351129+0000,dns,2078727703255923,172.16.0.149:51367,172.16.0.52:53"
"2022-04-16T17:52:23.037637+0000,alert,282956779721256,172.16.0.149:49798,172.16.0.52:139"
"2022-04-16T17:52:23.901721+0000,dns,556717995180633,172.16.0.170:51164,172.16.0.52:53"
"2022-04-16T17:52:23.904764+0000,dns,556717995180633,172.16.0.170:51164,172.16.0.52:53"
"2022-04-16T17:52:24.293356+0000,alert,2006941620009246,172.16.0.149:49799,172.16.0.52:139"
"2022-04-16T17:52:25.322102+0000,dns,1671081620007478,172.16.0.170:51909,172.16.0.52:53"

For sake of example, zoom in alert id '282956779721256':

// root@raspberrypi:~# grep 282956779721256 /var/log/suricata/eve.json|jq
{
  "timestamp": "2022-04-16T17:52:23.037637+0000",
  "flow_id": 282956779721256,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "172.16.0.149",
  "src_port": 49798,
  "dest_ip": "172.16.0.52",
  "dest_port": 139,
  "proto": "TCP",
  "metadata": {
    "flowints": {
      "applayer.anomaly.count": 1
    }
  },
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2260002,
    "rev": 1,
    "signature": "SURICATA Applayer Detect protocol only one direction",
    "category": "Generic Protocol Command Decode",
    "severity": 3
  },
  "smb": {
    "id": 1,
    "dialect": "NT LM 0.12",
    "command": "SMB1_COMMAND_NEGOTIATE_PROTOCOL",
    "status": "STATUS_SUCCESS",
    "status_code": "0x0",
    "session_id": 0,
    "tree_id": 0,
    "client_dialects": [
      "PC NETWORK PROGRAM 1.0",
      "LANMAN1.0",
      "Windows for Workgroups 3.1a",
      "LM1.2X002",
      "LANMAN2.1",
      "NT LM 0.12"
    ],
    "server_guid": "a21b9552-a4a0-48cd-8abb-ea111498253d"
  },
  "app_proto": "smb",
  "app_proto_ts": "failed",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 3,
    "bytes_toserver": 579,
    "bytes_toclient": 387,
    "start": "2022-04-16T17:52:23.037416+0000"
  },
  "payload": "AAAAiv9TTUJzAAAAABgHyAAAQlNSU1BZTCAAAP////4AAEAADP8AAAAEQTIAAAAAAAAASgAAAAAA1AAAoE8AYEgGBisGAQUFAqA+MDygDjAMBgorBgEEAYI3AgIKoioEKE5UTE1TU1AAAQAAAJeCCOIAAAAAAAAAAAAAAAAAAAAACgBhSgAAAA8AAAAAAA==",
  "payload_printable": ".....SMBs.........BSRSPYL ........@.......A2.......J.........O.`H..+......>0<..0..\n+.....7..\n.*.(NTLMSSP.........................\n.aJ.........",
  "stream": 0,
  "packet": "AB5PDqh0ABv8e9HACABFAAC2t+tAAIAG6WysEACVrBAANMKGAIthfGQf7GIEdVAYIBP6YwAAAAAAiv9TTUJzAAAAABgHyAAAQlNSU1BZTCAAAP////4AAEAADP8AAAAEQTIAAAAAAAAASgAAAAAA1AAAoE8AYEgGBisGAQUFAqA+MDygDjAMBgorBgEEAYI3AgIKoioEKE5UTE1TU1AAAQAAAJeCCOIAAAAAAAAAAAAAAAAAAAAACgBhSgAAAA8AAAAAAA==",
  "packet_info": {
    "linktype": 1
  },
  "host": "ras[berripi"
}
{
  "timestamp": "2022-04-16T17:55:42.050329+0000",
  "flow_id": 282956779721256,
  "in_iface": "eth0",
  "event_type": "flow",
  "src_ip": "172.16.0.149",
  "src_port": 49798,
  "dest_ip": "172.16.0.52",
  "dest_port": 139,
  "proto": "TCP",
  "app_proto": "smb",
  "app_proto_ts": "failed",
  "flow": {
    "pkts_toserver": 13,
    "pkts_toclient": 12,
    "bytes_toserver": 1743,
    "bytes_toclient": 1963,
    "start": "2022-04-16T17:52:23.037416+0000",
    "end": "2022-04-16T17:52:23.488633+0000",
    "age": 0,
    "state": "closed",
    "reason": "timeout",
    "alerted": true
  },
  "metadata": {
    "flowbits": [
      "smb.tree.connect.ipc"
    ],
    "flowints": {
      "applayer.anomaly.count": 1
    }
  },
  "tcp": {
    "tcp_flags": "1b",
    "tcp_flags_ts": "1b",
    "tcp_flags_tc": "1b",
    "syn": true,
    "fin": true,
    "psh": true,
    "ack": true,
    "state": "closed"
  },
  "host": "raspberrypi"
}

That's quite a bit to process. Keep in mind that while we are tuning Suricata, we can also ask it to replay one or more PCAP file directly.

Ask Suricata to run in offline mode using PCAP file for SUNNYSTATION

It is a very convenient way to test Suricata, as we do not inject any traffic in our network and instead let Suricata 'ingest' the contents of the PCAP file directly, to test the rules.

Also, we redirect the logs to a separate location (by default the directory where you are running the 'offline' mode), so we don't pollute a live installation.

Another example: EMOTET WITH COBALT STRIKE

Let's try another malware capture, in this case 2022-02-08 (TUESDAY) - FILES FOR AN ISC DIARY (EMOTET WITH COBALT STRIKE):

cd ~/Downloads/malware/ && \
curl --remote-name https://www.malware-traffic-analysis.net/2022/02/08/2022-02-08-Emotet-epoch4-infection-start-and-spambot-traffic.pcap.zip && \
unzip 2022-02-08-Emotet-epoch4-infection-start-and-spambot-traffic.pcap.zip && \
sudo suricata -r ~josevnz/Downloads/malware/2022-02-08-Emotet-epoch4-infection-start-and-spambot-traffic.pcap -k none --runmode autofp -c /etc/suricata/suricata.yaml -l ~josevnz/Downloads/malware/

Here is a sample session:

Making Sense of All the Alerts

Suricata will save lots of details when it detects an anomaly. You can tell that using jq to go through the alerts may not be desirable.

For a bigger setup, you may want to use an Elastic Stack (Filebeat, Logstash, Elastic Search, Kibana):

• Get the logs

• Store historically and normalize the logs

• Visualize their contents

But that feels overkill for a home setup, so I will roll out a few scripts to help me with what I need.

Show me what happened in the last 10 minutes

This is a script that assumes most of the defaults, so I don't have to type a jq expression. If there are any alerts then I dive deeper into the eve.json file.

A simple Python 3 script will do the trick for us:

#!/usr/bin/env python
"""
Show Suricata alerts
Author: Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import argparse
import json
from datetime import datetime, timedelta
from json import JSONDecodeError
from pathlib import Path
from typing import Callable, Any, Dict

DEFAULT_EVE = [Path("/var/log/suricata/eve.json")]
DEFAULT_TIMESTAMP_10M_AGO = datetime = datetime.now() - timedelta(minutes=10)


def _parse_timestamp(candidate: str) -> datetime:
    """
    Expected something like 2022-02-08T16:32:14.900292+0000
    :param candidate:
    :return:
    """
    if isinstance(candidate, str):
        try:
            iso_candidate = candidate.split('+', 1)[0]
            return datetime.fromisoformat(iso_candidate)
        except ValueError:
            raise ValueError(f"Invalid date passed: {candidate}")
    elif isinstance(candidate, datetime):
        return candidate


def alert_filter(
        *,
        timestamp: datetime = DEFAULT_TIMESTAMP_10M_AGO,
        data: Dict[str, Any]
) -> bool:
    if 'event_type' not in data:
        return False
    if data['event_type'] != 'alert':
        return False
    try:
        event_timestamp = _parse_timestamp(data['timestamp'])
        if event_timestamp > timestamp:
            return False
    except ValueError:
        return False
    return True


def get_alerts(
        *,
        eve_files=None,
        row_filter: Callable = alert_filter,
        timestamp: datetime = DEFAULT_TIMESTAMP_10M_AGO
) -> str:
    if eve_files is None:
        eve_files = DEFAULT_EVE
    for eve_file in eve_files:
        with open(eve_file, 'rt') as eve:
            for line in eve:
                try:
                    data = json.loads(line)
                    if row_filter(data=data, timestamp=timestamp):
                        yield data
                except JSONDecodeError:
                    continue  # Try to read the next record


if __name__ == "__main__":
    PARSER = argparse.ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "--timestamp",
        type=_parse_timestamp,
        default=DEFAULT_TIMESTAMP_10M_AGO,
        help=f"Minimum timestamp in the past to use when filtering events ({DEFAULT_TIMESTAMP_10M_AGO})"
    )
    PARSER.add_argument(
        'eve',
        type=Path,
        nargs="+",
        help=f"Path to one or more {DEFAULT_EVE[0]} file to parse."
    )
    OPTIONS = PARSER.parse_args()
    try:
        for alert in get_alerts(eve_files=OPTIONS.eve, timestamp=OPTIONS.timestamp):
            print(json.dumps(alert, indent=6, sort_keys=True))
    except KeyboardInterrupt:
        pass

It is a big improvement over jq as at least we can filter by timestamp, but it would be nice if our script could do the following:

1. Support pagination

2. Colorize output

3. Let you show between a table format or raw JSON output

What Did We Learn and What is Next?

Suricata is a complex piece of software. It takes time to tame it and more time to make sense of the information it presents. But it is very rewarding to see how you can tackle a tool that will allow you to secure your network from threats.

• The OISF Suricata YouTube channel has many interesting resources about this tool and a thriving community.

• Want to learn how to analyze PCAP files for bad traffic? malware-traffic-analysis has perfect material for you.

• Writing complex software is hard. For example, older versions of Snort are vulnerable to an attack that can disable it, CVE-2022-20685. Suricata also had CVE-2019-1010279 .These issues were fixed but illustrates the need to keep your software current, specially the one you use to protect your network.

• I did not touch the IPS mode, or even hybrid modes for Suricata. Please read the official documentation to get up to speed.

• Finally, do yourself a favor and read this Suricata Tutorial from FloCon 2016. It is very complete and will have you looking for more.

You can leave your comments on the Git repository and report any bugs. But more important get Suricata, get the code of this tutorial, and start securing your home wireless infrastructure in no time.

[josevnz@dmaf5 ~]$ sudo nmap -v -n -p- -sT -sV -O --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24

So I started to wonder:

• Is my wireless network secure?

• How long would it take to an attacker to get in?

I have a Raspberry 4 with Ubuntu (focal) installed and decided to use the well-known Kismet to find out.

In this article you will learn:

• How to get a whole picture of the networks nearby you with Kismet

• How to customize Kismet using Python and the REST-API

If you are curious, this is my home Raspberry PI 4, tiny monitor and all

Table of contents

• The saying 'Ask for forgiveness, not permission' doesn't apply here

• Getting to know your hardware

• kismet

• REST-API

• What did we learn?

The saying 'Ask for forgiveness, not permission' doesn't apply here

And by that I mean that you should not be trying to eavesdrop or infiltrate a wireless network that is not yours. It is relatively easy to detect if a new unknown client joined your wireless network, and it is also illegal.

So do the right thing – use this tutorial to learn and not to break into someone else's network, OK?

Getting to know your hardware

I will jump a little ahead to show you a small issue with the Raspberry 4 integrated Wireless interface.

The Raspberry PI 4 onboard wireless card will not work out of the box as the firmware doesn't support monitor mode.

There are works to support this. Instead, I took the easy way out and ordered an external Wi-Fi dongle from CanaKit.

The CanaKit wireless card worked out of the box, and we'll see it shortly. But first let's install and play around with Kismet.

Make sure the interface is running in monitor mode

By default, the network interface will have monitor mode off:

root@raspberrypi:~# iwconfig wlan1
wlan1     IEEE 802.11  ESSID:off/any  
          Mode:Managed  Access Point: Not-Associated   Tx-Power=0 dBm   
          Retry short  long limit:2   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:off

I know I will always set up my Ralink Technology, Corp. RT5370 Wireless Adapter in monitor mode, but I need to be careful as Ubuntu can swap wlan0 and wlan1 (The Broadcom adapter I want to skip is a PCI device).

The Ralink adapter is a USB adapter, so we can find out where it is:

josevnz@raspberrypi:/etc/netplan$ /bin/lsusb|grep Ralink
Bus 001 Device 004: ID 148f:5370 Ralink Technology, Corp. RT5370 Wireless Adapter

Now we need to find out what device was mapped to the Ralink adapter. With a little bit of help of the Ubuntu community I found than the Ralink adapter uses the rt2800usb driver 5370 Ralink Technology

The answer I seek is here:

josevnz@raspberrypi:~$ ls /sys/bus/usb/drivers/rt2800usb/*:1.0/net/
wlan1

So the code that does the wireless card detection looks like this:

root@raspberrypi:~#/bin/cat<<RC_LOCAL>/etc/rc.local
#!/bin/bash
usb_driver=rt2800usb
wlan=\$(/bin/ls /sys/bus/usb/drivers/\$usb_driver/*/net/)
if [ $? -eq 0 ]; then
        set -ex
        /usr/sbin/ifconfig "\$wlan" down
        /usr/sbin/iwconfig "\$wlan" mode monitor
        /usr/sbin/ifconfig "\$wlan" up
        set +ex
fi
RC_LOCAL
root@raspberrypi:~# chmod u+x /etc/rc.local && shutdown -r now "Enabling monitor mode"

Make sure the card is on monitor mode:

root@raspberrypi:~# iwconfig wlan1
iw        iwconfig  iwevent   iwgetid   iwlist    iwpriv    iwspy     
root@raspberrypi:~# iwconfig wlan1
wlan1     IEEE 802.11  Mode:Monitor  Frequency:2.412 GHz  Tx-Power=20 dBm   
          Retry short  long limit:2   RTS thr:off   Fragment thr:off
          Power Management:off

Good, let's move on with the tool setup

What is Kismet?

Kismet is:

a wireless network and device detector, sniffer, wardriving tool, and WIDS (wireless intrusion detection) framework.

Kismet installation and setup

The version that comes with the Ubuntu RaspberryPI by default is from 2016, way too old.

Instead, get an updated binary as explained here (I have Ubuntu focal, check with lsb_release --all).

wget -O - https://www.kismetwireless.net/repos/kismet-release.gpg.key | sudo apt-key add -
echo 'deb https://www.kismetwireless.net/repos/apt/release/focal focal main' | sudo tee /etc/apt/sources.list.d/kismet.list
sudo apt update
sudo apt install kismet

Do not run as root, use a SUID binary and a unix group access

Kismet needs elevated privileges to run. And deals with possibly hostile data. So running with minimized permissions is the safest approach.

The right way to set it up is by using a Unix group and set user id (SUID) binary. My user is 'josevnz' so I did this:

sudo apt-get install kismet
sudo usermod --append --groups kismet josevnz

Encrypt your access to Kismet with a self-signed certificate

I will enable SSL for my Kismet installation by using a self-signed certificate. I will use for that the Cloudflare CFSSL tools:

sudo apt-get update -y
sudo apt-get install -y golang-cfssl

Next step is to create the self-signed certificates. There is a lot of boilerplate steps here, so I will show you how you can jump through them (but please read the man pages to see what each command does):

Initial certificate

sudo /bin/mkdir --parents /etc/pki/raspberrypi
sudo /bin/cat<<CA>/etc/pki/raspberrypi/ca.json
{
   "CN": "Nunez Barrios family Root CA",
   "key": {
     "algo": "rsa",
     "size": 2048
   },
   "names": [
   {
     "C": "US",
     "L": "CT",
     "O": "Nunez Barrios",
     "OU": "Nunez Barrios Root CA",
     "ST": "United States"
   }
  ]
}
CA
cfssl gencert -initca ca.json | cfssljson -bare ca

SSL profile config

root@raspberrypi:/etc/pki/raspberrypi# /bin/cat<<PROFILE>/etc/pki/raspberrypi/cfssl.json
{
   "signing": {
     "default": {
       "expiry": "17532h"
     },
     "profiles": {
       "intermediate_ca": {
         "usages": [
             "signing",
             "digital signature",
             "key encipherment",
             "cert sign",
             "crl sign",
             "server auth",
             "client auth"
         ],
         "expiry": "17532h",
         "ca_constraint": {
             "is_ca": true,
             "max_path_len": 0, 
             "max_path_len_zero": true
         }
       },
       "peer": {
         "usages": [
             "signing",
             "digital signature",
             "key encipherment", 
             "client auth",
             "server auth"
         ],
         "expiry": "17532h"
       },
       "server": {
         "usages": [
           "signing",
           "digital signing",
           "key encipherment",
           "server auth"
         ],
         "expiry": "17532h"
       },
       "client": {
         "usages": [
           "signing",
           "digital signature",
           "key encipherment", 
           "client auth"
         ],
         "expiry": "17532h"
       }
     }
   }
}
PROFILE

Intermediate certificate

root@raspberrypi:/etc/pki/raspberrypi# /bin/cat<<INTERMEDIATE>/etc/pki/raspberrypi/intermediate-ca.json
{
  "CN": "Barrios Nunez Intermediate CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C":  "US",
      "L":  "CT",
      "O":  "Barrios Nunez",
      "OU": "Barrios Nunez Intermediate CA",
      "ST": "USA"
    }
  ],
  "ca": {
    "expiry": "43830h"
  }
}
INTERMEDIATE
cfssl gencert -initca intermediate-ca.json | cfssljson -bare intermediate_ca
cfssl sign -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile intermediate_ca intermediate_ca.csr | cfssljson -bare intermediate_ca

Configuration for the SSL certificate on the Raspberry PI 4 machine

Here we put the name and IP address of the machine that will run our Kismet web application:

/bin/cat<<RASPBERRYPI>/etc/pki/raspberrypi/raspberrypi.home.json
{
  "CN": "raspberrypi.home",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "L": "CT",
    "O": "Barrios Nunez",
    "OU": "Barrios Nunez Hosts",
    "ST": "USA"
  }
  ],
  "hosts": [
    "raspberrypi.home",
    "localhost",
    "raspberrypi",
    "192.168.1.11"
  ]               
}
RASPBERRYPI
cd /etc/pki/raspberrypi
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=peer raspberrypi.home.json| cfssljson -bare raspberry-peer
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=server raspberrypi.home.json| cfssljson -bare raspberry-server
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=client raspberrypi.home.json| cfssljson -bare raspberry-client

Adding SSL support is then as easy as adding the following overrides:

/bin/cat<<SSL>>/etc/kismet/kismet_site.conf
httpd_ssl=true
httpd_ssl_cert=/etc/pki/raspberrypi/raspberry-server.csr
httpd_ssl_key=/etc/pki/raspberrypi/raspberry-server-key.pem
SSL

Putting everything together, with a Kismet 'site' overrides file

Kismet has a really nice feature: it can use a file that overrides some defaults, without the need to edit multiple files. In this case my installation will override the SSL settings, Wifi interface, and log location. So time to update our /etc/rc.local file:

#!/bin/bash
# Kismet setup
usb_driver=rt2800usb
wlan=$(ls /sys/bus/usb/drivers/$usb_driver/*/net/)
if [ $? -eq 0 ]; then
    set -ex
    /usr/sbin/ifconfig "$wlan" down
    /usr/sbin/iwconfig "$wlan" mode monitor
    /usr/sbin/ifconfig "$wlan" up
    set +ex
    /bin/cat<<KISMETOVERR>/etc/kismet/kismet_site.conf
server_name=Nunez Barrios Kismet server
logprefix=/data/kismet
source=$wlan
httpd_ssl=true
httpd_ssl_cert=/etc/pki/raspberrypi/raspberry-server.csr
httpd_ssl_key=/etc/pki/raspberrypi/raspberry-server-key.pem
KISMETOVERR
fi

Finally, it is time to start Kismet (in my case as the non-root user josevnz):

# If you know which interface is the one in monitoring mode, then 
josevnz@raspberrypi:~$ kismet

Now let's log on for the first time to the web interface (In my case http://raspberripi.home:2501)

You will get a prompt the first time you try to log in your Kismet installation

In here you set up your admin user and password.

Example of the wireless networks detected

After a little time, Kismet will populate the main Dashboard with the list of wireless networks and devices it can detect. You will be surprised not just how many neighboring devices are out there but how many you have in your own house.

In my example, the wireless devices around me look pretty normal, except one that doesn't have a name:

A device with suspicious characteristics

The web interface provides all sorts of useful information, but is there an easy way to filter all the mac addresses on my networks?

Kismet has a REST API, so it is time to see what we can automate from there.

REST-API in Python

The developer documentation contains examples of how to extend Kismet, specifically the one related to the official Kismet REST-API in Python.

But it seems to be missing a feature to use API keys, instead of user/password. And the interaction with the end points doesn't seem to be complicated, so I will write my (less rich feature) wrapper.

You can download and install the code for a small application I wrote (kismet_home to illustrate how to work with Kismet (also has a copy of this tutorial) like this:

python3 -m venv ~/virtualenv/kismet_home
. ~/virtualenv/kismet_home/bin/activate
python -m pip install --upgrade pip
git clone git@github.com:josevnz/kismet_home.git
python setup.py bdist_wheel
pip install kismet_home-0.0.1-py3-none-any.whl

And then run the unit tests/ integration tests and even the third party vulnerability scanner:

. ~/virtualenv/kismet_home/bin/activate
# Unit/ integration tests
python -m unittest test/unit_test_config.py
python -m unittest /home/josevnz/kismet_home/test/test_integration_kismet.py
# Third party vulnerability scanner
pip-audit  --requirement requirements.txt

You will find more details on the README.md and DEVELOPER.md files.

Let's move on with the code.

How to Interact with Kismet using Python

First I'll write a generic HTTP client I can use to query or send commands to Kismet, that is the KismetWorker class:

import json
from datetime import datetime
from typing import Any, Dict, Set, List, Union
import requests


class KismetBase:

    def __init__(self, *, api_key: str, url: str):
        """
        Parametric constructor
        :param api_key: The Kismet generated API key
        :param url: URL where the Kismet server is running
        """
        self.api_key = api_key
        if url[-1] != '/':
            self.url = f"{url}/"
        else:
            self.url = url
        self.cookies = {'KISMET': self.api_key}

    def __str__(self):
        return f"url={self.url}, api_key=XXX"

class KismetWorker(KismetBase):

    def check_session(self) -> None:
        """
        Confirm if the session is valid for a given API key
        :return: None, throws an exception if the session is invalid
        """
        endpoint = f"{self.url}session/check_session"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()

    def check_system_status(self) -> Dict[str, Any]:
        """
        Overall status of the Kismet server
        :return: Nested dictionary describing different aspect of the Kismet system
        """
        endpoint = f"{self.url}system/status.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

    def get_all_alerts(self) -> Any:
        """
        You can get a description how the alert system is set up as shown here: /alerts/definitions.prettyjson
        This method returns the last N alerts registered by the system. Severity and meaning of the alert is explained
        here: https://www.kismetwireless.net/docs/devel/webui_rest/alerts/
        :return:
        """
        endpoint = f"{self.url}alerts/all_alerts.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

    def get_alert_by_hash(self, identifier: str) -> Dict[str, Any]:
        """
        Get details of a single alert by its identifier (hash)
        :return:
        """
        parsed = int(identifier)
        if parsed < 0:
            raise ValueError(f"Invalid ID provided: {identifier}")
        endpoint = f"{self.url}alerts/by-id/{identifier}/alert.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

    def get_alert_definitions(self) -> Dict[Union[str, int], Any]:
        """
        Get the defined alert types
        :return:
        """
        endpoint = f"{self.url}alerts/definitions.json"
        r = requests.get(endpoint, cookies=self.cookies)
        r.raise_for_status()
        return json.loads(r.text)

The way Kismet API works is that you make the API KEY part of the query, or you define it in the KISMET cookie. I choose to populate the cookie.

KismetWorker implements the following methods:

• check_session: It checks if your API KEY is valid. If not it will throw an exception.

• check_system_status: Validates if the administrator (you most likely) defined an administrator for the Kismet server. If not, then all the API queries will fail.

• get_all_alerts: Gets all the available alerts (if any) from your Kismet server.

• get_alert_by_hash: If you know the identifier (hash) of an alert, you can retrieve the details of that event only.

• get_alert_definitions: Get all the alert definitions. Kismet supports a wide range of alerts and a user will definitely be interested to find out what type of alerts they are.

You can see all the integration code here to see how the methods work in action.

I also wrote a class that requires admin privileges. I use it to define a custom alert type and to send alerts using that type to Kismet, as part of the integration tests. Right now I don't have much use of sending custom alerts to Kismet in real life, but that may change in the future, so here is the code:

class KismetAdmin(KismetBase):

    def define_alert(
            self,
            *,
            name: str,
            description: str,
            throttle: str = '10/min',
            burst: str = "1/sec",
            severity: int = 5,
            aclass: str = 'SYSTEM'

    ):
        """
        Define a new type of alert for Kismet
        :param aclass: Alert class
        :param severity: Alert severity
        :param throttle: Optional throttle
        :param name: Name of the new alert
        :param description: What does this mean
        :param burst: Optional burst
        :return:
        """
        endpoint = f"{self.url}alerts/definitions/define_alert.cmd"
        command = {
            'name': name,
            'description': description,
            'throttle': throttle,
            'burst': burst,
            'severity': severity,
            'class': aclass
        }
        r = requests.post(endpoint, json=command, cookies=self.cookies)
        r.raise_for_status()

    def raise_alert(
            self,
            *,
            name: str,
            message: str
    ) -> None:
        """
        Send an alert to Kismet
        :param name: A well-defined name or id for the alert. MUST exist
        :param message: Message to send
        :return: None. Will raise an error if the alert could not be sent
        """
        endpoint = f"{self.url}alerts/raise_alerts.cmd"
        command = {
            'name': name,
            'text': message
        }
        r = requests.post(endpoint, json=command, cookies=self.cookies)
        r.raise_for_status()

Getting the data is just part of the story. We need to normalize it, so it can be used by the final scripts.

How to Normalize the Kismet raw data

Kismet contains a lot of details about the alerts, but we do not require to show the user those details (think about the nice view you get with the web application). Instead we do a few transformations using the following class with static methods:

• parse_alert_definitions: Returns a simplified report of all the alert definitions

• process_alerts: Changes numeric alerts for more descriptive types and also returns dictionaries for the types and severity meaning of those alerts.

• pretty_timestamp: Converts the numeric timestamp into something we can use for comparisons and display

The code for the KismetResultsParser helper class:

class KismetResultsParser:
    SEVERITY = {
        0: {
            'name': 'INFO',
            'description': 'Informational alerts, such as datasource  errors, Kismet state changes, etc'
        },
        5: {
            'name': 'LOW',
            'description': 'Low - risk events such as probe fingerprints'
        },
        10: {
            'name': 'MEDIUM',
            'description': 'Medium - risk events such as denial of service attempts'
        },
        15: {
            'name': 'HIGH',
            'description': 'High - risk events such as fingerprinted watched devices, denial of service attacks, '
                           'and similar '
        },
        20: {
            'name': 'CRITICAL',
            'description': 'Critical errors such as fingerprinted known exploits'
        }
    }

    TYPES = {
        'DENIAL': 'Possible denial of service attack',
        'EXPLOIT': 'Known fingerprinted exploit attempt against a vulnerability',
        'OTHER': 'General category for alerts which don’t fit in any existing bucket',
        'PROBE': 'Probe by known tools',
        'SPOOF': 'Attempt to spoof an existing device',
        'SYSTEM': 'System events, such as log changes, datasource errors, etc.'
    }

    @staticmethod
    def parse_alert_definitions(
            *,
            alert_definitions: List[Dict[str, str]],
            keys_of_interest: Set[str] = None
    ) -> List[Dict[str, str]]:
        """
        Remove unwanted keys from full alert definition dump, to make it easier to read onscreen
        :param alert_definitions: Original Kismet alert definitions
        :param keys_of_interest: Kismet keys of interest
        :return: List of dictionaries with trimmed keys, description, severity and header for easy reading
        """
        if keys_of_interest is None:
            keys_of_interest = {
                'kismet.alert.definition.class',
                'kismet.alert.definition.description',
                'kismet.alert.definition.severity',
                'kismet.alert.definition.header'
            }
        parsed_alerts: List[Dict[str, str]] = []
        for definition in alert_definitions:
            new_definition = {}
            for def_key in definition:
                if def_key in keys_of_interest:
                    new_key = def_key.split('.')[-1]
                    new_definition[new_key] = definition[def_key]
            parsed_alerts.append(new_definition)
        return parsed_alerts

    @staticmethod
    def process_alerts(
            *,
            alerts: List[Dict[str, Union[str, int]]],

    ) -> Any:
        """
        Removed unwanted fields from alert details, also return extra data for severity and types of alerts
        :param alerts:
        :return:
        """
        processed_alerts = []
        found_types = {}
        found_severities = {}
        for alert in alerts:
            severity = alert['kismet.alert.severity']
            severity_name = KismetResultsParser.SEVERITY[severity]['name']
            severity_desc = KismetResultsParser.SEVERITY[severity]['description']
            found_severities[severity_name] = severity_desc
            text = alert['kismet.alert.text']
            aclass = alert['kismet.alert.class']
            found_types[aclass] = KismetResultsParser.TYPES[aclass]
            processed_alert = {
                'text': text,
                'class': aclass,
                'severity': severity_name,
                'hash': alert['kismet.alert.hash'],
                'dest_mac': alert['kismet.alert.dest_mac'],
                'source_mac': alert['kismet.alert.source_mac'],
                'timestamp': alert['kismet.alert.timestamp']
            }
            processed_alerts.append(processed_alert)
        return processed_alerts, found_severities, found_types

    @staticmethod
    def pretty_timestamp(timestamp: float) -> datetime:
        """
        Convert a Kismet timestamp (TIMESTAMP.UTIMESTAMP) into a pretty timestamp string
        :param timestamp:
        :return:
        """
        return datetime.fromtimestamp(timestamp)

If you run the integration tests with the admin role enabled, you will see than one or more (depending how many times you ran the test) alerts were added to the Web UI:

These alerts where generated using the Python client and the REST API

As a reminder, you can see how this is used by looking at the code here. Showing a sample run of all the integration tests against my installation (this one without publishing alerts, so some tests are skipped):

(kismet_home) [josevnz@dmaf5 kismet_home]$ python -m unittest /home/josevnz/kismet_home/test/test_integration_kismet.py 
[09:13:05] DEBUG    Starting new HTTP connection (1): raspberrypi.home:2501                                                                                                                                                        connectionpool.py:228
           DEBUG    http://raspberrypi.home:2501 "GET /session/check_session HTTP/1.1" 200 None                                                                                                                                    connectionpool.py:456
.           DEBUG    Starting new HTTP connection (1): raspberrypi.home:2501                                                                                                                                                        connectionpool.py:228
           DEBUG    http://raspberrypi.home:2501 "GET /system/status.json HTTP/1.1" 200 None                                                                                                                                       connectionpool.py:456
.           DEBUG    Starting new HTTP connection (1): raspberrypi.home:2501                                                                                                                                                        connectionpool.py:228
           DEBUG    http://raspberrypi.home:2501 "GET /alerts/definitions.json HTTP/1.1" 200 None                                                                                                                                  connectionpool.py:456
.[09:13:05] 'ADMIN_SESSION_API' environment variable not defined. Skipping this test                                                                                                                                       test_integration_kismet.py:105
....
----------------------------------------------------------------------
Ran 7 tests in 0.053s

OK

Where do we store our API key and other configuration details?

Details like this won't be hardcoded inside the scripts, but instead they will reside on an external configuration file:

(kismet_home) [josevnz@dmaf5 kismet_home]$ cat ~/.config/kodegeek/kismet_home/config.ini 
[server]
url = http://raspberrypi.home:2501
api_key = E41CAD466552810392D538FF8D43E2C5

The following classes handle all the access details (using a Reader and a Writer class for each type of operation):

"""
Simple configuration management for kismet_home settings
"""
import os.path
from configparser import ConfigParser
from pathlib import Path
from typing import Dict

from kismet_home import CONSOLE

DEFAULT_INI = os.path.expanduser('~/.config/kodegeek/kismet_home/config.ini')
VALID_KEYS = {'api_key', 'url'}


class Reader:

    def __init__(self, config_file: str = DEFAULT_INI):
        """
        Constructor
        :param config_file: Optional override of the ini configuration file
        """
        self.config = ConfigParser()
        if not self.config.read(config_file):
            raise ValueError(f"Could not read {config_file}")

    def get_api_key(self):
        """
        Get back the API key used to connect to Kismet
        :return:
        """
        return self.config.get('server', 'api_key')

    def get_url(self):
        """
        Get back URL of Kismet server
        :return:
        """
        return self.config.get('server', 'url')


class Writer:

    def __init__(
            self,
            *,
            server_keys: Dict[str, str]
    ):
        if not server_keys:
            raise ValueError("Configuration is incomplete!, aborting!")
        self.config = ConfigParser()
        self.config.add_section('server')
        valid_keys_cnt = 0
        for key in server_keys:
            value = server_keys[key]
            if key not in VALID_KEYS:
                CONSOLE.log(f"Ignoring invalid key: {key} = {value}")
                continue
            self.config.set('server', key, value)
            CONSOLE.log(f"Added: server: {key} = {value}")
        for valid_key in VALID_KEYS:
            if not self.config.get('server', valid_key):
                raise ValueError(f"Missing required key: {valid_key}")

    def save(
            self,
            *,
            config_file: str = DEFAULT_INI
    ):
        basedir = Path(config_file).parent
        basedir.mkdir(exist_ok=True, parents=True)
        with open(config_file, 'w') as config:
            self.config.write(config, space_around_delimiters=True)
        CONSOLE.log(f"Configuration file {config_file} written")

The first time you set up your kismet_home installation, you can create the configuration files like this:

[josevnz@dmaf5 kismet_home]$ python3 -m venv ~/virtualenv/kismet_home
[josevnz@dmaf5 kismet_home]$ . ~/virtualenv/kismet_home/bin/activate
(kismet_home) [josevnz@dmaf5 kismet_home]$ python -m pip install --upgrade pip
(kismet_home) [josevnz@dmaf5 kismet_home]$ git clone git@github.com:josevnz/kismet_home.git
(kismet_home) [josevnz@dmaf5 kismet_home]$ python setup.py bdist_wheel
(kismet_home) [josevnz@dmaf5 kismet_home]$ pip install kismet_home-0.0.1-py3-none-any.whl

(kismet_home) [josevnz@dmaf5 kismet_home]$ kismet_home_config.py 
Please enter the URL of your Kismet server: http://raspberrypi.home:2501/
Please enter your API key: E41CAD466552810392D538FF8D43E2C5
[13:02:35] Added: server: url = http://raspberrypi.home:2501/                                                                                 config.py:44
           Added: server: api_key = E41CAD466552810392D538FF8D43E2C5                                                                          config.py:44
           Configuration file /home/josevnz/.config/kodegeek/kismet_home/config.ini written

Please note the use of the virtual environment here. This will allow us to keep the application's libraries self-contained.

Putting everything together: How to Write our CLI for kismet_home

The kismet_home_alerts.py script will support two modes:

• Show the alert definitions

• Show all the alerts

Also, it will allow filtering alerts based on the level (INFO, MEDIUM, HIGH, ...).

Showing all the definitions, filtered by CRITICAL:

You can see here the alert definitions filtered by level

Or showing all the alerts received so far, with anonymous MAC address (great for screenshots like this):

Alerts for my local network, with anonymous MAC addresses and filtered

How you can generate these tables with ease? There is a dedicated class for the text user interface (TUI):

from typing import List, Dict, Any

from rich.layout import Layout
from rich.table import Table

from kismet_home.kismet import KismetResultsParser


def create_alert_definition_table(
        *,
        alert_definitions: List[Dict[str, Any]],
        level_filter: str = 0
) -> Table:
    """
    Create a table showing the alert definitions
    :param alert_definitions: Alert definitions from Kismet
    :param level_filter: User can override the level of the alerts shown. But default is 0 (INFO)
    :return: A Table with the alert definitions
    """
    definition_table = Table(title="Alert definitions")
    definition_table.add_column("Severity", justify="right", style="cyan", no_wrap=True)
    definition_table.add_column("Description", style="magenta")
    definition_table.add_column("Header", justify="right", style="yellow")
    definition_table.add_column("Class", justify="right", style="green")
    filter_level = KismetResultsParser.get_level_for_security(level_filter)
    filtered_definitions = 0
    for definition in alert_definitions:
        int_severity: int = definition['severity']
        if int_severity < filter_level:
            continue
        severity = KismetResultsParser.SEVERITY[int_severity]['name']
        if 0 <= int_severity < 5:
            severity = f"[bold blue]{severity}[/ bold blue]"
        if 5 <= int_severity < 10:
            severity = f"[bold yellow]{severity}[/ bold yellow]"
        if 10 <= int_severity < 15:
            severity = f"[bold orange]{severity}[/ bold orange]"
        else:
            severity = f"[bold red]{severity}[/ bold red]"
        filtered_definitions += 1
        definition_table.add_row(
            severity,
            definition['description'],
            definition['header'],
            definition['class']
        )
    definition_table.caption = f"Total definitions: {filtered_definitions}"
    return definition_table


def create_alert_layout(
        *,
        alerts: List[Dict[str, Any]],
        level_filter: str = 0,
        anonymize: bool = False,
        severities: Dict[str, str]
):
    """
    :param severities:
    :param alerts:
    :param level_filter:
    :param anonymize:
    :return:
    """
    alerts_table = Table(title="Alert definitions")
    alerts_table.add_column("Timestamp", no_wrap=True)
    alerts_table.add_column("Severity", justify="right", style="cyan", no_wrap=True)
    alerts_table.add_column("Text", style="magenta")
    alerts_table.add_column("Source MAC", justify="right", style="yellow", no_wrap=True)
    alerts_table.add_column("Destination MAC", justify="right", style="yellow", no_wrap=True)
    alerts_table.add_column("Class", justify="right", style="green", no_wrap=True)
    filter_level = KismetResultsParser.get_level_for_security(level_filter)

    filtered_definitions = 0
    for alert in alerts:
        int_severity: int = KismetResultsParser.get_level_for_security(alert['severity'])
        if int_severity < filter_level:
            continue
        severity = KismetResultsParser.SEVERITY[int_severity]['name']
        if 0 <= int_severity < 5:
            severity = f"[bold blue]{severity}[/ bold blue]"
        if 5 <= int_severity < 10:
            severity = f"[bold yellow]{severity}[/ bold yellow]"
        if 10 <= int_severity < 15:
            severity = f"[bold orange]{severity}[/ bold orange]"
        else:
            severity = f"[bold red]{severity}[/ bold red]"
        filtered_definitions += 1
        if anonymize:
            s_mac = KismetResultsParser.anonymize_mac(alert['source_mac'])
            d_mac = KismetResultsParser.anonymize_mac(alert['dest_mac'])
        else:
            s_mac = alert['source_mac']
            d_mac = alert['dest_mac']
        alerts_table.add_row(
            str(KismetResultsParser.pretty_timestamp(alert['timestamp'])),
            severity,
            alert['text'],
            s_mac,
            d_mac,
            alert['class']
        )
    alerts_table.caption = f"Total alerts: {filtered_definitions}"

    severities_table = Table(title="Severity legend")
    severities_table.add_column("Severity")
    severities_table.add_column("Explanation")
    for severity in severities:
        explanation = f"[green]{severities[severity]}[/green]"
        severities_table.add_row(f"[yellow]{severity}[/yellow]", explanation)

    layout = Layout()
    layout.split(
        Layout(ratio=2, name="alerts"),
        Layout(name="severities"),
    )
    layout["alerts"].update(alerts_table)
    layout["severities"].update(severities_table)
    return layout, filtered_definitions

And now with all the ingredients ready, we can see how the final script looks:

#!/usr/bin/env python
"""
# kismet_home_alerts.py
# Author
Jose Vicente Nunez Zuleta (kodegeek.com@protonmail.com)
"""
import logging
import sys

from requests import HTTPError
import argparse

from kismet_home import CONSOLE
from kismet_home.config import Reader
from kismet_home.kismet import KismetWorker, KismetResultsParser
from kismet_home.tui import create_alert_definition_table, create_alert_layout

if __name__ == '__main__':

    arg_parser = argparse.ArgumentParser(
        description="Display alerts generated by your local Kismet installation",
        prog=__file__
    )
    arg_parser.add_argument(
        '--debug',
        action='store_true',
        default=False,
        help="Enable debug mode"
    )
    arg_parser.add_argument(
        '--anonymize',
        action='store_true',
        default=False,
        help="Anonymize MAC addresses"
    )
    arg_parser.add_argument(
        '--level',
        action='store',
        default='INFO',
        help="Enable debug mode"
    )
    arg_parser.add_argument(
        'mode',
        action='store',
        choices=['alert_type', 'alerts'],
        help="Operation mode"
    )

    try:
        args = arg_parser.parse_args()
        conf_reader = Reader()
        kw = KismetWorker(
            api_key=conf_reader.get_api_key(),
            url=conf_reader.get_url()
        )
        if args.mode == 'alert_type':
            alert_definitions = KismetResultsParser.parse_alert_definitions(
                alert_definitions=kw.get_alert_definitions()
            )
            table = create_alert_definition_table(alert_definitions=alert_definitions, level_filter=args.level)
            if table.columns:
                CONSOLE.print(table)
            else:
                CONSOLE.print(f"[b]Could not get alert definitions![/b]")
        elif args.mode == 'alerts':
            alerts, severities, types = KismetResultsParser.process_alerts(
                alerts=kw.get_all_alerts()
            )
            layout, found = create_alert_layout(
                alerts=alerts,
                level_filter=args.level,
                anonymize=args.anonymize,
                severities=severities
            )
            if found:
                CONSOLE.print(layout)
            else:
                CONSOLE.print(f"[b]No alerts to show for level={args.level}[/b]")
    except (ValueError, HTTPError):
        logging.exception("There was an error")
        sys.exit(100)
    except KeyboardInterrupt:
        CONSOLE.log("Scan interrupted, exiting...")
    sys.exit(0)

A few things to note:

• This is not a long-running application. Instead, is a snapshot of all the alerts. If you wanted, for example, to forward these alerts by email or to a framework like grafana, you are better off using Websockets and one of the methods that retrieves only the last changes.

• The layout is crude, and there is plenty of room for improvement. But our little tui is displaying relevant information without too many distractions

• And if was fun to code!

What did we learn?

• How to install Kismet and secure it with a self-signed SSL certificate

• How to write a simple Bash script to set up the correct Wireless interface in monitor mode, after the RaspBerryPI reboots

• How to add an API KEY with read-only access to use it instead of the legacy user/ password schema for authentication and authorization

• How to write classes in Python that can communicate with Kismet using its REST-API

• How to add unit and integration tests to the code to make sure everything works and new code changes do not break existing functionality

Please leave your comments on the git repository and report any bugs. But more important get Kismet, get the code of this tutorial, and start securing your home wireless infrastructure in no time.

Nmap ("Network Mapper") is a free and open source utility for network discovery and security auditing.

Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime.

You can also use it to bypass weak protections, find hidden or mis-configured services, or just to give you a better understanding how networks work.

Table of contents:

• What you will learn from this article

• Nmap 101: Identify all the public services in our network

• How to Write an 'easy button' Network Scanner that Uses Nmap

• How to Make a Home Network Scanner a Web Service

• What did we learn?

What you will learn from this article

We will cover the following to illustrate how you can enhance Nmap with Python:

• Write a small script that can scan all the hosts on the local network, making sure it runs with the proper privileges.

• Enhance Nmap by correlating services with security advisories.

• Convert our scripts into a web-service. Will add basic security (authorization and encryption).

Things you should know and do before starting

Don't worry too much, as I will guide you through the steps. This will be a fun experience, and you'll have all the source code to follow along:

• Be familiar with basic network concepts like Classless inter-domain routing (CIDR)

• Be able to write a program in a scripting language like Python.

• The code from can be installed using a virtual environment. If you are not familiar with a virtual environment, you can read the following: Packaging applications to install on other machines with Python.

What tools you will need for this tutorial?

I won't cover the installation of any of these tools, but there is plenty of documentation out there to get you started.

• Install the whole code from this tutorial by following the instructions as explained on the main README document file on my GitHub official repository site. You will need to install Git to clone the code.

• A Linux distribution. Fedora, Ubuntu, Kali, feel free to use the one you feel most comfortable with (I used Fedora 35.)

• Python interpreter. A good Linux distribution will come with Python pre-installed or at least will make it easier for you to install. I used Python 3.9 here.

Last two things:

• I skipped some imports in the code snippets as they do not enhance the code demonstrations. To get the most accurate code, please do clone the public Git repository for this tutorial and open the source code.

• Only run this examples against your local network. You can be curious, have fun, and learn new things about existing tools without affecting others.

Hacking is about learning!

Nmap 101: Identify all the public services in our network

Word of caution: The saying 'b_etter ask for forgiveness than permission'_ doesn't apply here

We do not care about being 'stealth' or triggering an Intrusion Detection System (IDS) because of our port scanning activity. An IDS normally looks for abnormal network patterns and if it sees a machine opening and closing ports on rapid succession across many hosts that would be considered a port scan attack. Again that won't be the case in our home network because, well, we know it is us running such a scan.

For the same reason you should not launch a port scan on a network you don't own, as Nmap is not 100% stealth (you can always play with randomizing frequency, type of TCP handshake, number of ports opened, use a proxy, and so on and yet you most likely will miss something).

So better behave, OK? :-)

What do we need to run Nmap and OS fingerprinting?

The goal here is to see what services are running in our network using a command line interface (CLI) script.