Before you touch a single exploit or send a single payload, you need to know what services are running, what ports are open, what technologies are in play, and where the weak spots might be.

This phase is called reconnaissance. It can eat up hours – sometimes even days – if you’re doing it manually.

That’s where Autorecon comes in.

What is AutoRecon?

Autorecon is a tool that automates most of the initial recon work. It’s not a magic box, but it’s close.

Autorecon takes a list of IPs or domain names and runs a series of predefined scans. Then it organizes the output neatly so you don’t waste time parsing through raw Nmap files or rerunning missed commands.

If you’re just starting out with pentesting – whether you’re on your first TryHackMe box or your tenth OSCP practice lab – Autorecon can save you a ton of time. Let’s break down how it works.

What Exactly Does Autorecon Do?

At its core, Autorecon does three things:

1. Runs Nmap scans on each target IP or hostname.

2. Identifies services running on open ports.

3. Runs specific enumeration tools based on those services.

Let’s say you run it against an IP that has ports 22 (SSH), 80 (HTTP), and 139/445 (SMB) open. Autorecon will:

• Use Nmap to check versions and scripts for each port.

• Run nikto or gobuster on port 80.

• Run enum4linux or smbmap on SMB.

• Store everything in organized folders for later review.

That’s what you’d do manually – but faster, cleaner, and without forgetting steps.

How to Use Autorecon

Let’s walk through a quick example. Assume you have a target at 10.129.8.143.

Here’s the basic command:

autorecon 10.129.8.143

That’s it. No flags, no extra setup. Autorecon takes care of the rest. To understand what is going on behind the scenes, let's add the verbosity -v flag.

Here is a sample result.

Behind the scenes, it creates a folder structure like this:

results/
├── 10.129.8.143/
│   ├── scans/
│   │   ├── nmap/
│   │   └── gobuster/
│   ├── reports/
│   └── notes.txt

You’ll find full Nmap outputs, service-specific tool results, and even a place to jot down your own observations. All ready to go.

If you want to scan multiple targets, just pass a list:

autorecon targets.txt

Once Autorecon completes a scan, go to the results/<IP>/scans/ folder. Start with the Nmap outputs.

Look for open ports and services:

• Port 80 open? Check gobuster and nikto outputs in the HTTP folder.

• SMB ports open? Look in the enum4linux and smbmap results to find shared drives or user info.

• FTP anonymous login allowed? Use that access to explore directories.

These findings will give you the next steps – like browsing a web service, crafting a payload, or checking for known exploits.

Why It’s a Big Deal for Beginners

If you’re new to pentesting, one of the hardest parts is remembering everything you’re supposed to check. You pop open a port, and you think:

• “Wait… Should I run enum4linux on this?”

• “What was that flag for aggressive Nmap scanning again?”

• “Did I already check this web service with nikto?”

Autorecon takes that mental load off your shoulders. You can focus on analysis, not babysitting scans.

And here’s another benefit: it helps you learn the process.

While Autorecon automates recon, it shows you every tool and command it runs. You can open the raw output, read the flags, and understand why it ran those scans.

Example: You’ll see it runs nmap -sV -sC for version detection and scripts. This helps beginners understand which scans map to which services and why they matter.

As it runs, you’ll see all the tools and commands it’s using. You can look at the raw results, see what worked, and gradually build your own workflow.

What It Scans (By Default)

Here’s a quick overview of what Autorecon runs based on port and service:

Nmap:

• Quick scan

• Full TCP port scan

• Service/version detection

• NSE scripts

HTTP/HTTPS:

• gobuster (directory brute-forcing)

• nikto (vulnerability scanner)

• whatweb (tech detection)

SMB:

• enum4linux-ng

• smbmap

• Nmap SMB scripts

FTP:

• Anonymous login check

• Nmap FTP scripts

SSH:

• Banner grab

• SSH version check

And that’s just a slice. It handles other services too, like MySQL, SNMP, SMTP, and even RPC.

When Autorecon Is Most Useful

Autorecon shines in certain situations:

• Training labs: You get a clear view of your target with minimal setup.

• OSCP preparation: It runs the exact recon tools you’ll need to use on the OSCP exam.

• Time-limited pentests: When you need to hit multiple targets fast, Autorecon keeps your output consistent and saves you from retyping everything.

But it’s not just about speed. It’s about being thorough. With manual scanning, it’s easy to miss something small. Autorecon doesn’t forget.

What Autorecon Doesn’t Do

Autorecon isn’t an exploit tool. It doesn’t hack anything for you. It doesn’t guess credentials or bypass login pages.

It’s focused purely on reconnaissance. That means you still have to:

• Review scan results

• Analyze web services manually (for example, browse the site, test inputs)

• Decide which exploits or payloads to run

Also, it can be noisy. If you’re on a real engagement where stealth matters, some scans might raise alarms. In that case, you’d want to run more controlled commands manually.

Tips for Using Autorecon Effectively

Use flags to control scans:
To increase verbosity and skip previously scanned hosts:

autorecon -v --only-scans-dir 10.129.8.143

Customize wordlists for better results:
By default, Autorecon uses small wordlists. You can improve this:

autorecon --dirbuster.wordlist /usr/share/seclists/Discovery/Web-Content/raft-medium-words.txt 10.129.8.143

This makes directory brute-forcing more effective, especially on web targets.

Don’t skip the output: Read the Nmap files, check the HTML reports. Tools don’t think like humans. You still have to connect the dots.

Final Thoughts

Autorecon doesn’t replace your skills – but it helps supercharge them. Instead of spending 30 minutes typing out scan commands, you can run one command and start analyzing in minutes. This helps beginners stay focused, and it helps pros save time.

So if you’re tired of rerunning the same Nmap scans over and over, or you just want cleaner results and fewer mistakes, let Autorecon do the heavy lifting – so you can focus on the part that really matters: breaking stuff.

For more cybersecurity tutorials, join our newsletter. To learn the basics of Offensive Cybersecurity, check out our Security Starter Course.

Nmap allows network admins to identify devices running on their network, discover open ports and services, and detect vulnerabilities.

Here is the basic syntax to use nmap:

nmap <ip/url>

Let’s do a quick scan and see what we can find. We can use the URL scanme.nmap.org to try out a scan. Nmap allows us to use this server to practice scans.

As you can see, we have found some open ports and services. These act as entry points for further analysis or exploitation.

Nmap is usually the first tool that ethical hackers learn. Here is a full tutorial if you want to learn more about Nmap.

Nmap Scripting Engine

A key feature is the Nmap Scripting Engine (NSE). It lets users run scripts to do detailed network scans and gather specific information.

Scripts help you perform a list of actions automatically instead of performing them step by step.

These scripts cover a range of functionalities, from service detection to vulnerability scanning. In this article, we’ll look at a few useful Nmap scripts.

I’ll walk you through each script, explain what it does, and show you how to use it. By the end, you’ll have a solid understanding of how to use these scripts as an ethical hacker.

Note: This tutorial is to help you understand network security. Hacking or even scanning another server without permission is illegal.

HTTP-Enum

Imagine you’re tasked with checking a website’s security and want to see if there are any hidden pages or directories. You suspect there might be admin panels, login pages, or test files that aren’t linked on the main site.

Finding these hidden areas could reveal critical security weaknesses, such as unprotected admin pages or old files that might still hold sensitive information.

The http-enum script is used to scan a web server and find common directories and files that might be hidden from the main site navigation.

Think of it like opening doors in a building to see what’s behind each one. It searches for paths like login pages, admin panels, config files, and other directories that aren’t typically linked on the main website.

For example, a login page or an admin section may exist at specific paths but aren’t visible to regular users. This information is useful because knowing these locations can help you identify security weak points.

Here is the command to run the http-enum script:

nmap - script http-enum -p 80 <target-ip>

As you can see, the above sample result shows /login.php, /docs and other exposed URL paths. These can be entry points to find restricted information in a web server.

SMB-OS-Discovery

Suppose you’re exploring a company’s network to understand what kind of systems they have in place, specifically in a Windows environment.

Knowing the exact operating system and version of each server helps you assess vulnerabilities. For example, an older version of Windows might have unpatched flaws that need attention.

The smb-os-discovery script targets servers that use the SMB protocol, mainly found in Windows environments, to gather information about the server’s operating system. It can reveal details like the Windows version, the server name, and its domain.

This script helps you understand what type of system you’re dealing with, which is key for checking security flaws specific to that OS.

Here is the syntax to run the smb-os-discovery script.

nmap - script smb-os-discovery -p 445 <target-ip>

As you can see in the above sample result, the script connects to the SMB service on the target and retrieves OS information. This can help you quickly identify the Windows version and other details about the server.

HTTP-Headers

Imagine you’re evaluating a website’s configuration and security settings. You want to know what kind of server it’s running, what methods are allowed, and if it’s enforcing HTTPS connections.

These details give you insights into whether the server’s configuration aligns with best practices, helping you spot any missing security settings.

The http-headers script checks the headers sent by a web server when a user connects to it it. Headers tell you the server type (like Apache or NGINX), security settings (like HTTPS requirements), allowed methods, and caching rules.

These details are like the server’s blueprint for communication, often revealing if the server has certain protections enabled.

Here is the syntax to run the http-headers script:

nmap - script http-headers -p 80 <target-ip>

You can see that the sample response shows headers like like X-Powered-By, Set-Cookie, and so on. These headers can help to find security issues such as cross-site scripting (XSS) and clickjacking.

SSH-Brute

Let’s say you’re testing a server’s defenses against unauthorized access through SSH. You know that weak passwords are a common risk, so you need a way to check if any accounts have easily guessable credentials.

This test will help you identify weak SSH logins that need stronger passwords to protect the server.

The ssh-brute script tries to log into an SSH server by guessing usernames and passwords. SSH, or Secure Shell, is often used for remote logins.

If the usernames and passwords are easy to guess, this script might find a way in. It’s a useful test to see if login credentials are strong enough to prevent unauthorized access.

Here is the syntax to run the ssh-brute script:

nmap - script ssh-brute -p 22 <target-ip>

As you can see, this script tries different username-password combinations on the SSH server. If successful, it will display the correct credentials.

DNS-Brute

Imagine you’re mapping out a company’s network and want to see if they have any subdomains that aren’t publicly listed. Each subdomain might serve a different purpose, such as hosting email servers or internal testing sites.

Discovering these subdomains helps you check if any of them are exposing sensitive services.

The dns-brute script helps you find subdomains associated with a given domain by trying out common names, like “www,” “mail,” or “ftp.” Subdomains can host separate services and applications, each with its own set of vulnerabilities.

Here is the syntax to run the dns-brute script:

nmap - script dns-brute <target-domain>

As you can see, the script attempts to resolve a list of common subdomains, and finds one internal hostname. Using this script can reveal subdomains that aren’t listed in public records, helping you to gain a fuller picture of an organization’s network layout.

Conclusion

These Nmap scripts provide a powerful way to audit, troubleshoot, and secure networks. By understanding what each script does and how to use it, you’ll be able to uncover hidden issues and safeguard your infrastructure.

To learn how to build a career in Cybersecurity, check out The Hacker’s Handbook. To practice hacking real systems and get help from other hackers, join The Hacker’s Hub.

Nmap ("Network Mapper") is a free and open source utility for network discovery and security auditing.

Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime.

You can also use it to bypass weak protections, find hidden or mis-configured services, or just to give you a better understanding how networks work.

Table of contents:

• What you will learn from this article

• Nmap 101: Identify all the public services in our network

• How to Write an 'easy button' Network Scanner that Uses Nmap

• How to Make a Home Network Scanner a Web Service

• What did we learn?

What you will learn from this article

We will cover the following to illustrate how you can enhance Nmap with Python:

• Write a small script that can scan all the hosts on the local network, making sure it runs with the proper privileges.

• Enhance Nmap by correlating services with security advisories.

• Convert our scripts into a web-service. Will add basic security (authorization and encryption).

Things you should know and do before starting

Don't worry too much, as I will guide you through the steps. This will be a fun experience, and you'll have all the source code to follow along:

• Be familiar with basic network concepts like Classless inter-domain routing (CIDR)

• Be able to write a program in a scripting language like Python.

• The code from can be installed using a virtual environment. If you are not familiar with a virtual environment, you can read the following: Packaging applications to install on other machines with Python.

What tools you will need for this tutorial?

I won't cover the installation of any of these tools, but there is plenty of documentation out there to get you started.

• Install the whole code from this tutorial by following the instructions as explained on the main README document file on my GitHub official repository site. You will need to install Git to clone the code.

• A Linux distribution. Fedora, Ubuntu, Kali, feel free to use the one you feel most comfortable with (I used Fedora 35.)

• Python interpreter. A good Linux distribution will come with Python pre-installed or at least will make it easier for you to install. I used Python 3.9 here.

Last two things:

• I skipped some imports in the code snippets as they do not enhance the code demonstrations. To get the most accurate code, please do clone the public Git repository for this tutorial and open the source code.

• Only run this examples against your local network. You can be curious, have fun, and learn new things about existing tools without affecting others.

Hacking is about learning!

Nmap 101: Identify all the public services in our network

Word of caution: The saying 'b_etter ask for forgiveness than permission'_ doesn't apply here

We do not care about being 'stealth' or triggering an Intrusion Detection System (IDS) because of our port scanning activity. An IDS normally looks for abnormal network patterns and if it sees a machine opening and closing ports on rapid succession across many hosts that would be considered a port scan attack. Again that won't be the case in our home network because, well, we know it is us running such a scan.

For the same reason you should not launch a port scan on a network you don't own, as Nmap is not 100% stealth (you can always play with randomizing frequency, type of TCP handshake, number of ports opened, use a proxy, and so on and yet you most likely will miss something).

So better behave, OK? :-)

What do we need to run Nmap and OS fingerprinting?

The goal here is to see what services are running in our network using a command line interface (CLI) script.

Nmap requires elevated privileges to do the OS fingerprinting and scans using raw sockets. You will need to run the commands as root or su "do" (SUDO) to elevate your permissions. A SUDO rule to do this is similar to this (file /etc/sudoers):

## Same thing without a password
%wheel    ALL=(ALL)    NOPASSWD: ALL

This means that anyone on the 'wheel' unix group can run commands as root:

(2600) [josevnz@dmaf5 2600]$ grep wheel /etc/group
wheel:x:10:josevnz,services

# To confirm we can run commands as root
(2600) [josevnz@dmaf5 2600]$ sudo -l
Matching Defaults entries for josevnz on dmaf5:
    !visiblepw, always_set_home, match_group_by_gid, always_query_group_plugin, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS",
    env_keep+="MAIL QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", env_keep+="LC_MONETARY
    LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin\:/var/lib/snapd/snap/bin

User josevnz may run the following commands on dmaf5:
    (ALL) NOPASSWD: ALL

Next we'll do a quick scan of our local network (in this example is 192.168.1.0/24). I used the -v (verbose) flag to get some progress feedback while scanning for all the ports while also doing OS fingerprinting (-O).

I saved the execution of the Nmap run to an XML file (-oX), which Nmap can use to resume execution if it gets interrupted (--resume):

# In case the scan is interrupted: nmap --resume $HOME/home_scan.xml
[josevnz@dmaf5 docs]$ sudo nmap -v -n -p- -sT -sV -O --osscan-limit --max-os-tries 1 -oX $HOME/home_scan.xml 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2021-12-30 16:35 EST
NSE: Loaded 45 scripts for scanning.
Initiating ARP Ping Scan at 16:35
Scanning 254 hosts [1 port/host]
...
# After a while and several cups of Venezuelan coffee...
Network Distance: 1 hop
TCP Sequence Prediction: Difficulty=265 (Good luck!)
IP ID Sequence Generation: All zeros

Nmap scan report for 192.168.1.20
Host is up (0.0097s latency).
Not shown: 65530 closed ports
PORT      STATE    SERVICE      VERSION
36184/tcp filtered unknown
37309/tcp filtered unknown
49323/tcp open     unknown
49376/tcp filtered unknown
62078/tcp open     iphone-sync?
MAC Address: 9E:90:75:3A:D7:XX (Unknown)
...

The resulting XML format file is very verbose:

<host starttime="1640901327" endtime="1640902555"><status state="up" reason="arp-response" reason_ttl="0"/>
<address addr="192.168.1.1" addrtype="ipv4"/>
<address addr="38:5B:5E:1D:52:99" addrtype="mac"/>
<hostnames>
</hostnames>
<ports><extraports state="closed" count="65523">
<extrareasons reason="conn-refused" count="65523"/>
</extraports>
<port protocol="tcp" portid="139"><state state="open" reason="syn-ack" reason_ttl="0"/><service name="netbios-ssn" product="Samba smbd" version="3.X - 4.X" extrainfo="workgroup: ZZZ" method="probed" conf="10"><cpe>cpe:/a:samba:samba</cpe></service></port>
    ...

Time to do some coding. Parsing data in many formats is one of Python's strengths. Data is extracted and normalized for all the ports that are not 'closed' using lxml:

class OutputParser:
    """
    Parse Nmap raw XML output
    """

    @staticmethod
    def parse_nmap_xml(xml: str) -> (str, Any):
        """
        Parse XML and return details for the scanned ports
        @param xml:
        @return: tuple nmaps arguments, port details
        """
        parsed_data = []
        root = ElementTree.fromstring(xml)
        nmap_args = root.attrib['args']
        for host in root.findall('host'):
            for address in host.findall('address'):
                curr_address = address.attrib['addr']
                data = {
                    'address': curr_address,
                    'ports': []
                }
                states = host.findall('ports/port/state')
                ports = host.findall('ports/port')
                for i in range(len(ports)):
                    if states[i].attrib['state'] == 'closed':
                        continue  # Skip closed ports
                    port_id = ports[i].attrib['portid']
                    protocol = ports[i].attrib['protocol']
                    services = ports[i].findall('service')
                    cpe_list = []
                    service_name = ""
                    service_product = ""
                    service_version = ""
                    for service in services:
                        for key in ['name', 'product', 'version']:
                            if key in service.attrib:
                                if key == 'name':
                                    service_name = service.attrib['name']
                                elif key == 'product':
                                    service_product = service.attrib['product']
                                elif key == 'version':
                                    service_version = service.attrib['version']
                        cpes = service.findall('cpe')
                        for cpe in cpes:
                            cpe_list.append(cpe.text)
                        data['ports'].append({
                            'port_id': port_id,
                            'protocol': protocol,
                            'service_name': service_name,
                            'service_product': service_product,
                            'service_version': service_version,
                            'cpes': cpe_list
                        })
                        parsed_data.append(data)
        return nmap_args, parsed_data

Once the data is collected we can create a nice table in the terminal with the help of Rich.

The table has the following columns:

• Internet Protocol (IP) address

• Protocol: On this script it will always be Transfer Control Protocol (TCP)

• Port ID: The port number where the service runs

• Service: An networked service like Secure Shell (SSH)

• Common Platform Enumeration (CPE): Is a structured naming scheme for information technology systems, software, and packages.

• Advisories: Any vulnerability related to the CPE identified by Nmap. Will need to correlate those ourselves.

def create_scan_table(*, cli: str) -> Table:
    """
    Create a table for the CLI UI
    :param cli: Full Nmap arguments used on the run
    :return: Skeleton table, no data
    """
    nmap_table = Table(title=f"NMAP run info: {cli}")
    nmap_table.add_column("IP", justify="right", style="cyan", no_wrap=True)
    nmap_table.add_column("Protocol", justify="right", style="cyan", no_wrap=True)
    nmap_table.add_column("Port ID", justify="right", style="magenta", no_wrap=True)
    nmap_table.add_column("Service", justify="right", style="green")
    nmap_table.add_column("CPE", justify="right", style="blue")
    nmap_table.add_column("Advisories", justify="right", style="blue")
    return nmap_table
...
def fill_simple_table(*, exec_data: str, parsed_xml: Dict[Any, Any]) -> Table:
    """
    Convenience method to create a simple UI table with Nmap XML output
    :param exec_data: Arguments and options used to run Nmap
    :param parsed_xml: Nmap data as a dictionary
    :return: Populated tabled
    """
    nmap_table = create_scan_table(cli=exec_data)
    for row_data in parsed_xml:
        address = row_data['address']
        ports = row_data['ports']
        for port_data in ports:
            nmap_table.add_row(
                address,
                port_data['protocol'],
                port_data['port_id'],
                f"{port_data['service_name']} {port_data['service_product']} {port_data['service_version']}",
                "\n".join(port_data['cpes']),
                ""
            )
    return nmap_table

The resulting script uses the code above to give the user the whole picture about the local network scan:

#!/usr/bin/env python
import sys
from rich.console import Console
from home_nmap.query import OutputParser
from home_nmap.ui import fill_simple_table

if __name__ == "__main__":
    console = Console()
    for nmap_xml in sys.argv[1:]:
        with open(nmap_xml, 'r') as xml:
            xml_data = xml.read()
            rundata, parsed = OutputParser.parse_nmap_xml(xml_data)
            nmap_table = fill_simple_table(exec_data=rundata, parsed_xml=parsed)
            console.print(nmap_table)

Scan for local network. Advisories column is empty

If you notice, the 'Advisories' column is left completely empty. We'll use the NIST cybersecurity website search engine to populate the missing advisories, by-passing the CPE that have version information to avoid false positives.

We use requests to help us with the HTTP communication:

from dataclasses import dataclass
import requests
IGNORED_CPES = {"cpe:/o:linux:linux_kernel"}
from cpe import CPE
from lxml import html

@dataclass
class NIDS:
    summary: str
    link: str
    score: str

class NDISHtml:

    def __init__(self):
        """
        Some CPE return too many false positives,
        so they are ignored right out the bat
        """
        self.raw_html = None
        self.parsed_results = []
        self.url = "https://nvd.nist.gov/vuln/search/results"
        self.ignored_cpes = IGNORED_CPES

    def get(self, cpe: str) -> str:
        """
        Run a CPE search on the NDIS website. If the CPE has no version then skip the search
        as it will return too many false positives
        @param cpe: CPE identifier coming from Nmap, like cpe:/a:openbsd:openssh:8.0
        @return:
        """
        params = {
            'form_type': 'Basic',
            'results_type': 'overview',
            'search_type': 'all',
            'isCpeNameSearch': 'false',
            'query': cpe
        }
        if cpe in self.ignored_cpes:
            return ""
        valid_cpe = CPE(cpe)
        if not valid_cpe.get_version()[0]:
            return ""
        response = requests.get(
            url=self.url,
            params=params
        )
        response.raise_for_status()
        return response.text

    def parse(self, html_data: str) -> list[NIDS]:
        """
        Parse NDIS web search. Not aware that they offer a REST API that doesn't require parsing.
        It is assumed that this method is never called directly by end users, so no further checks are done on the
        HTML file contents.
        @param html_data: RAW HTML used for scrapping
        @return: List of NDIS, if any
        """
        self.parsed_results = []
        if html_data:
            ndis_html = html.fromstring(html_data)
            # 1:1 match between 3 elements, use parallel array
            summary = ndis_html.xpath("//*[contains(@data-testid, 'vuln-summary')]")
            cve = ndis_html.xpath("//*[contains(@data-testid, 'vuln-detail-link')]")
            score = ndis_html.xpath("//*[contains(@data-testid, 'vuln-cvss2-link')]")
            for i in range(len(summary)):
                ndis = NIDS(
                    summary=summary[i].text,
                    link="https://nvd.nist.gov/vuln/detail/" + cve[i].text,
                    score=score[i].text
                )
                self.parsed_results.append(ndis)
        return self.parsed_results

Then we correlate the Nmap CPES in the results with each one of the advisories, if any:

from typing import Any
from dataclasses import dataclass
@dataclass
class NIDS:
    summary: str
    link: str
    score: str
class NDISHtml:
    def correlate_nmap_with_nids(self, parsed_xml: Any) -> dict[str, list[NIDS]]:
        correlated_cpe = {}
        for row_data in parsed_xml:
            ports = row_data['ports']
            for port_data in ports:
                for cpe in port_data['cpes']:
                    raw_ndis = self.get(cpe)
                    cpes = self.parse(raw_ndis)
                    correlated_cpe[cpe] = cpes
        return correlated_cpe

The new table speaks for itself:

Nmap scan results on a nice table

More complete, and we can see now that a few of our local services may have a vulnerability!

Can we do better? For example, it would be nice to be able to run Nmap directly from Python instead of parsing the results of a run, so let's code that.

How to Write an 'easy button' Network Scanner that Uses Nmap

How to wrap Nmap with Python (subprocess.run)

Nmap doesn't offer a formal API to interact with external programs. For that reason we will run it from Python and save the results into an XML file. We can then use the data any way we want (see the 'subprocess.run' call in method 'scan' from our class NmapRunner):

class NMapRunner:

    def __init__(self):
        """
        Create a Nmap executor
        """
        self.nmap_report_file = None
        found_sudo = shutil.which('sudo', mode=os.F_OK | os.X_OK)
        if not found_sudo:
            raise ValueError(f"SUDO is missing")
        self.sudo = found_sudo
        found_nmap = shutil.which('nmap', mode=os.F_OK | os.X_OK)
        if not found_nmap:
            raise ValueError(f"NMAP is missing")
        self.nmap = found_nmap

    def scan(
            self,
            *,
            hosts: str,
            sudo: bool = True
    ):
        command = []
        if sudo:
            command.append(self.sudo)
        command.append(self.nmap)
        command.extend(__NMAP__FLAGS__)
        command.append(hosts)
        completed = subprocess.run(
            command,
            capture_output=True,
            shell=False,
            check=True
        )
        completed.check_returncode()
        args, data = OutputParser.parse_nmap_xml(completed.stdout.decode('utf-8'))
        return args, data, completed.stderr

Security note: The named argument 'shell=False' tells us that we do not want to create a new shell when running our process. This will provide protection against shell injection attacks.

How to Speed up Nmap (remember all these flags in a single place)

Your local network has less latency than the Internet. It will also most likely be easier to scan for open ports and OS fingerprinting because there is no firewall between you and the hosts.

Additionally, we are not concerned of triggering an IDS detection, so you can use the following to reduce the amount of time required to complete the port scanning (Variable NMAP__FLAGS in package system):

import shlex
# Convert the args for proper usage on the CLI
NMAP_HOME_NETWORK_DEFAULT_FLAGS = {
    '-n': 'Never do DNS resolution',
    '-sS': 'TCP SYN scan, recommended',
    '-p-': 'All ports',
    '-sV': 'Probe open ports to determine service/version info',
    '-O': 'OS Probe. Requires sudo/ root',
    '-T4': 'Aggressive timing template',
    '-PE': 'Enable this echo request behavior. Good for internal networks',
    '--version-intensity 5': 'Set version scan intensity. Default is 7',
    '--disable-arp-ping': 'No ARP or ND Ping',
    '--max-hostgroup 20': 'Hostgroup (batch of hosts scanned concurrently) size',
    '--min-parallelism 10': 'Number of probes that may be outstanding for a host group',
    '--osscan-limit': 'Limit OS detection to promising targets',
    '--max-os-tries 1': 'Maximum number of OS detection tries against a target',
    '-oX -': 'Send XML output to STDOUT, avoid creating a temp file'
}
__NMAP__FLAGS__ = shlex.split(" ".join(NMAP_HOME_NETWORK_DEFAULT_FLAGS.keys()))

The Nmap documentation also suggests that you can split the total hostlist across several instances of Nmap (it can be no greater than the number of CPUs in the server running the tool) to increase parallelism. But that doesn't come for free. You will need to worry about issues like race conditions and synchronization in concurrent threads running Nmap.

For now we'll keep it simple and let Nmap take care of any optimizations by providing the flags showed above.

How to figure out the local networks on the machine where Nmap runs?

Our Python script can also check interfaces that are up, skip virtual interfaces,: and skip the special loopback interface. Luckily the kernel publishes all the information we need on /proc/net/dev file:

(2600) [josevnz@dmaf5 2600]$ cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo: 18303833  303389    0    0    0     0          0         0 18303833  303389    0    0    0     0       0          0
enp2s0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eno1: 1931173135 3908073    0    1    0     0          0    407486 274206691 3289566    0    0    0     0       0          0

We can parse it like this (class HostIface, method refresh_interfaces):

class HostIface:    
    ...

    def __refresh_interfaces__(self, *, skip_loopback: bool = True, only_alive: bool = True) -> Set[str]:
        """
        Alive means an interface that has shown any byte activity since the server is up
        Skips the loopback interface by default
        :param only_alive: Skip interfaces with zero bytes activity
        :param skip_loopback
        :return: Set with interface names
        """
        with open('/proc/net/dev', 'r') as dev:
            for line in dev:
                tokens = line.split()
                if tokens[0].find(":") != -1:
                    name = tokens[0].split(':')[0]
                    if re.search('virbr\\d+|docker', name):
                        continue  # Skip virtual interfaces
                    if only_alive and int(tokens[1].strip()) == 0:
                        continue
                    if skip_loopback and name == 'lo':
                        continue
                    self.interfaces.add(name)
        return self.interfaces

The class HostIface gets the IP address and network masks of each local interface using Socket programming. Then it maps each list of networks for these ip addresses + netmask combinations:

SIOCGIFADDR = 0x8915
SIOCGIFNETMASK = 0x891B

class HostIface:
    @staticmethod
    def get_iface_details(iface: str):
        """
        Get network interface IP using the network interface name
        :return: IP address and network mask
        :param iface: Interface name (like eth0, enp2s0, etc.)
        """
        with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
            iface_pack = struct.pack('256s', bytes(iface, 'ascii'))
            packed_ip = fcntl.ioctl(s.fileno(), SIOCGIFADDR, iface_pack)[20:24]
            packed_netmask = fcntl.ioctl(s.fileno(), SIOCGIFNETMASK, iface_pack)[20:24]
        return socket.inet_ntoa(packed_ip), socket.inet_ntoa(packed_netmask)

    def get_local_networks(self, *, refresh: bool = False) -> List[ipaddress.IPv4Network]:
        """
        Get the list of local networks, using all the local IP addresses
        :param refresh: If true, re-read /proc to get list of interfaces
        :return: List of IPv4Network addresses
        """
        local_networks: List[ipaddress.IPv4Network] = []
        for iface in self.get_alive_interfaces(refresh=refresh):
            ip, netmask = self.get_iface_details(iface)
            network: ipaddress.IPv4Network = ipaddress.ip_network(f"{ip}/{netmask}", strict=False)
            if network not in local_networks:
                local_networks.append(network)
        return local_networks

Note that this is not portable across other OS's like BSD and specially Windows.

How to put together the new Nmap CLI frontend

Now, creating a new CLI for Nmap is straightforward. As a plus, the new frontend also allows you to save your scanning results as a json file (--report optional argument):

#!/usr/bin/env python
"""
# home_scan.py - A simple host discovery script
This script can scan your home network to show information from all the connected devices.

## References:
* [Nmap reference](https://nmap.org/book/man.html)

# Author
Jose Vicente Nunez Zuleta (kodegeek.com@protonmail.com)
"""
import json
import logging
import re
import sys

from rich.layout import Layout
from rich.live import Live
from rich.console import Console
from rich.logging import RichHandler
from rich.text import Text
from rich.traceback import install
from rich.progress import TimeElapsedColumn, Progress, TextColumn
from typing import List
import argparse

from home_nmap.nmap import Scanner
from home_nmap.system import HostIface
from home_nmap.ui import create_scan_table, update_scan_table


def get_targets(target_list: List[str], cli_args: argparse.Namespace) -> str:
    if cli_args.target:
        for target in target_list:
            """
            This should not happen as the script has an alias for -oX
            """
            if re.search("-oX", target):
                raise ValueError(f"Cannot redirect the output to a file by passing -oX. Run this script with --help")
        return ','.join(target_list)
    return ','.join(HostIface().get_prefixed_local_networks())


if __name__ == '__main__':

    install()
    logging.basicConfig(
        level="NOTSET",
        format="%(message)s",
        datefmt="[%X]",
        handlers=[RichHandler(rich_tracebacks=True)]
    )

    console = Console()
    arg_parser = argparse.ArgumentParser(
        description="Identify my local networked devices, with open ports",
        prog=__file__
    )
    arg_parser.add_argument(
        '--debug',
        action='store_true',
        default=False,
        help="Enable debug mode"
    )
    arg_parser.add_argument(
        '--results',
        '-xO',
        action='store',
        help=f"If defined, save scan results into this file."
    )
    arg_parser.add_argument(
        'target',
        action='store',
        nargs='*',
        help=(f"One or more targets, in Nmap format (scanme.homenmap.org, microsoft.com/24, 192.168.0.1; "
              f"10.0.0-255.1-254). If not provided, then scan local networks")
    )
    args = arg_parser.parse_args()

    current_app_progress = Progress(
        TimeElapsedColumn(),
        TextColumn("{task.description}"),
    )
    scanning_task = current_app_progress.add_task("[yellow]Waiting[/yellow] for scan results... :hourglass:")

    try:
        scanner = Scanner()
        scan_targets = get_targets(args.target, args)
        if args.results:
            table_title = f"Targets: {scan_targets}, results file={args.results}"
        else:
            table_title = f"Targets: {scan_targets}"
        results_table = create_scan_table(cli=f"Targets: {table_title}")
        layout = Layout()
        layout.split(
            Layout(name="Scan status", size=1),
            Layout(name="Scan results"),
        )
        with Live(
                layout,
                console=console,
                screen=False,
                redirect_stderr=False,
        ) as live:
            layout['Scan results'].update(Text(
                text=f"No results yet ({scan_targets})", style="green", justify="center")),
            layout['Scan status'].update(current_app_progress)
            nmap_args, data, stderr = scanner.scan(hosts=scan_targets)
            update_scan_table(scan_result=data,
                              results_table=results_table,
                              main_layout=layout,
                              progress=current_app_progress,
                              task_id=scanning_task
                              )
        if args.results:
            report_data = {
                'args': nmap_args,
                'scan': data
            }
            with open(args.results, 'w') as report_file:
                json.dump(report_data, report_file, indent=True)

    except ValueError:
        logging.exception("There was an error")
        sys.exit(100)
    except KeyboardInterrupt:
        console.log("Scan interrupted, exiting...")
        pass
    sys.exit(0)

The code got a little more verbose due the argument parsing and the user interface updates handling, but not too much.

Let's see an example against 127.0.0.1:

Results of a live Nmap run, enriched with CVE advisories

If you are curious how the resulting JSON report looks like when passing the --report flag:

{
 "args": "/usr/bin/nmap -n -sS -p- -sV -O -T4 -PE --version-intensity 5 --disable-arp-ping --max-hostgroup 20 --min-parallelism 10 --osscan-limit --max-os-tries 1 -oX - 127.0.0.1",
 "scan": [
  {
   "addresses": [
    {   
     "ip": "127.0.0.1"
    }   
   ],  
   "ports": [
    {   
     "protocol": "tcp",
     "port_id": "22",
     "service_name": "ssh",
     "service_product": "OpenSSH",
     "service_version": "8.4",
     "cpe": "cpe:/o:linux:linux_kernel:2.6.32"
    },  
    {   
     "protocol": "tcp",
     "port_id": "631",
     "service_name": "ipp",
     "service_product": "CUPS",
     "service_version": "2.3",
     "cpe": "cpe:/o:linux:linux_kernel:2.6.32"
    },  
...]
}

What about a GUI?

Nmap has a very complete GUI called Zenmap, but the whole point was to show you that you can write a nice Text UI in Python as well to display the results.

You can achieve the same by using other popular frameworks like Tkinter, which has incredibly detailed documentation. For that reason, we'll not expand this topic any further.

Instead, let me show you how you can build a self-documenting REST-API for Nmap

How to Make a Home Network Scanner a Web Service

Sometimes you cannot install Nmap because you lack the elevated privileges to do so or the server has installation constraints (like space or memory).

Or it could be that you want to run the port scanner on a machine that is able to connect to a network not directly accessible from the server you are currently logged in (and bypassing network segregation imposed by firewall). In this case the webservice will act like a proxy to run our Nmap command.

This is also known as "pivoting", and it it is a common technique used to bypass firewalls and proxy servers.

Let's take a short detour to talk more about pivoting with Nmap

Can you run Nmap through a proxy?

Yes, you can use proxychains to run Nmap through a host with better connectivity or to bypass firewall restrictions:

Using pivoting with Nmap and Proxy-chains

Say for the sake of argument that host 'External Linux' doesn't have direct connectivity to the network 192.168.1.0/24 but 'Multi homed Linux' does, and it can run a SOCKS-5 proxy.

To gain access to the internal network, we run SSH forwarding port 9050 (as a SOCKS-5 proxy) under user 'josevnz':

josevnz@multihomed:~$ ssh  -N -D 9050 josevnz@192.168.1.11
The authenticity of host '192.168.1.11 (192.168.1.11)' can't be established.
ECDSA key fingerprint is SHA256:VIZCaCMb5rN2oL/xuv6CPrG1II+huW44x4TWhyKv8QM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.11' (ECDSA) to the list of known hosts.

Then we install proxychains on 'External Linux' if is not already there:

# You either install proxychains first with 
# RedHat: 'sudo dnf -y install proxychains'
# Debian: 'sudo apt-get install proxychains4'

And create a proxychains.conf file pointing to your SSH SOCKS-5 proxy server:

cat<<CFG>$HOME/proxychains.conf
strict_chain
proxy_dns
remote_dns_subnet 224
tcp_read_time_out 15000
tcp_connect_time_out 8000
[ProxyList]
socks5 192.168.1.11 9050
CFG

Finally, run Nmap, using a TCP scan:

[josevnz@external docs]$ proxychains -q -f $HOME/proxychains.conf sudo Nmap -sT 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2021-12-30 16:06 EST

Alternatively just tell Nmap itself to use our new SOCKS-5 proxy (documentation says this is still under development):

[josevnz@external docs]$ sudo nmap -v -sT --proxies socks4://192.168.1.11:9050 192.168.1.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2021-12-31 09:03 EST

Now lets go back to code our web service.

How to run Nmap as a web service

In any case, running Nmap as a service is not something new (Nmap-cgi). We'll make ours using FastAPI.

I put together a web service that shows the current version and also the available network interfaces (home_nmap/main.py):

"""
# Web service for home_nmap
# Author
Jose Vicente Nunez Zuleta (kodegeek.com@protonmail.com)
"""
from home_nmap import __version__
from fastapi import FastAPI

from home_nmap.system import HostIface

app = FastAPI()


@app.get("/version")
async def version():
    return {"version": __version__}


@app.get("/local_networks")
async def local_networks():
    hi = HostIface()
    return hi.get_local_networks()

In FastApi we define the web service endpoints with annotations it takes care of serializing our response back to the client.

Here is how you can start the service using the uvicorn web server with the '--reload' flag to detect changes in our code automatically:

(home_nmap) [josevnz@dmaf5 home_nmap]$ uvicorn home_nmap.main:app --reload
INFO:     Will watch for changes in these directories: ['/home/josevnz/Documents/home_nmap']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [122202] using watchgod
INFO:     Started server process [122204]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Getting the home_nmap API version using curl, JSON response pretty print with jq:

(home_nmap) [josevnz@dmaf5 rich]$ curl --fail --silent http://127.0.0.1:8000/version| jq '.'
{
  "version": "0.0.1"
}

Now get the list of local networks calling the '/local_networks' endpoint:

(home_nmap) [josevnz@dmaf5 rich]$ curl --fail --silent http://127.0.0.1:8000/local_networks| jq '.'
[
  "192.168.1.0/24"
]

One nice thing about FastApi is that you get automatic documentation for your REST endpoints (http://127.0.0.1:8000/docs#/):

Nmap self documenting REST API

Not bad for a few lines of code if you ask me.

How to implement the scanner service

On the 'main.py' file we implement the endpoint to scan the local network and to correlate the CPE with any possible advisories:

from typing import Optional
from home_nmap.system import NMapRunner
from home_nmap.query import NDISHtml, target_validator
from fastapi import FastAPI, HTTPException
app: FastAPI = FastAPI()

@app.get("/scan")
def scan(
        target: Optional[str] = None,
        full_advisories=True
):
    """
    Scan a target to get service information.
    Note, FastAPI has a query validator, but I decided to use my own as I look for bad targets:
    Query(None, min_length=MIN_LEN_TARGET, max_length=MAX_LEN_TARGET)
    @param target: Override local network with custom targets, in Nmap format.
    @param full_advisories: If false, skip the summary information from the advisories
    @return: JSON containing the results of the scan
    """
    try:
        scanner = NMapRunner()
        args, scan_results, stderr = scanner.scan(hosts=target_validator(target))
        enriched_results = {
            'args': args,
            'hosts': []
        }
        if not scan_results:
            raise HTTPException(status_code=404, detail=f"Got no results from scanning target={target}")
        cpe_details = NDISHtml().correlate_nmap_with_nids(scan_results)
        for host_data in scan_results:
            enriched_host_data = {
                'address': host_data['address'],
                'ports': []
            }
            ports = host_data['ports']
            for port_data in ports:
                advisories = []
                # Unroll the advisories, if any ...
                for cpe in port_data['cpes']:
                    if cpe in cpe_details:  # Service may not have an advisory
                        for nids in cpe_details[cpe]:
                            if full_advisories:
                                advisories.append({
                                    'link': nids.link,
                                    'summary': nids.summary,
                                    'score': nids.score
                                })
                            else:
                                advisories.append({
                                    'link': nids.link,
                                    'summary': '',  # For consistency
                                    'score': nids.score
                                })
                enriched_host_data['ports'].append(
                    {
                        'cpes': port_data['cpes'],
                        'advisories': advisories,
                        'protocol': port_data['protocol'],
                        'port_id': port_data['port_id'],
                        'service': [
                            f"{port_data['service_name']},"
                            f"{port_data['service_product']},"
                            f"{port_data['service_version']}"
                        ]
                    }
                )
            enriched_results['hosts'].append(enriched_host_data)
        return enriched_results
    except (TypeError, ValueError) as exp:
        raise HTTPException(status_code=500, detail=str(exp))

The 'target_validator' function does a few checks on the target to ensure only valid scanning targets are passed (this is the same function we wrote for the CLI program):

import re
MIN_LEN_TARGET = 9
MAX_LEN_TARGET = 50
from typing import Optional
import shlex
def target_validator(target: Optional[str]) -> str:
    """
    Simple validator for Nmap target expressions
    @param target: (scanme.homenmap.org, microsoft.com/24, 192.168.0.1; 10.0.0-255.1-254). None or empty are valid
    @return:
    """
    if target:
        regexp_list = [
            '-[a-z-A-Z][A-Z]*',
            '-[a-zA-Z]\\d*',
            '--[a-z-]+'
        ]
        if len(target) < MIN_LEN_TARGET:
            raise ValueError(f"Provided length for target is too small < {MIN_LEN_TARGET}")
        if len(target) > MAX_LEN_TARGET:
            raise ValueError(f"Provided length for target is too big < {MAX_LEN_TARGET}")
        for arg in shlex.split(target):
            for regexp in regexp_list:
                if re.search(regexp, arg):
                    raise ValueError(f"You cannot override Nmap arguments: {arg}")
    return target

Time to put everything together.

What does a scan run look like (very verbose JSON)?

Here is what the scan result of 2 machines in my local network looks like (the web service is running on dmaf5.home on port 8000):

[josevnz@dmaf5 ~]$ curl http://dmaf5.home:8000/scan?target=192.168.1.10,23
{"args":"/usr/bin/nmap -n -sS -p- -sV -O -T4 -PE --version-intensity 5 --disable-arp-ping --max-hostgroup 20 --min-parallelism 10 --osscan-limit --max-os-tries 1 -oX - 192.168.1.10,23","hosts":[{"address":"192.168.1.10","ports":[{"cpes":["cpe:/a:openbsd:openssh:8.2p1"],"advisories":[{"link":"https://nvd.nist.gov/vuln/detail/CVE-2021-41617","summary":"sshd in OpenSSH 6.2 through 8.x before 8.8, when certain non-default configurations are used, allows privilege escalation because supplemental groups are not initialized as expected. Helper programs for AuthorizedKeysCommand and AuthorizedPrincipalsCommand may run with privileges associated with group memberships of the sshd process, if the configuration specifies running the command as a different user.","score":"4.4 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2016-20012","summary":"OpenSSH through 8.7 allows remote attackers, who have a suspicion that a certain combination of username and public key is known to an SSH server, to test whether this suspicion is correct. This occurs because a challenge is sent only when that combination could be valid for a login session.","score":"4.3 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2021-28041","summary":"ssh-agent in OpenSSH before 8.5 has a double free that may be relevant in a few less-common scenarios, such as unconstrained agent-socket access on a legacy operating system, or the forwarding of an agent to an attacker-controlled host.","score":"4.6 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2020-15778","summary":"** DISPUTED ** scp in OpenSSH through 8.3p1 allows command injection in the scp.c toremote function, as demonstrated by backtick characters in the destination argument. NOTE: the vendor reportedly has stated that they intentionally omit validation of \"anomalous argument transfers\" because that could \"stand a great chance of breaking existing workflows.\"","score":"6.8 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2020-14145","summary":"The client side in OpenSSH 5.7 through 8.4 has an Observable Discrepancy leading to an information leak in the algorithm negotiation. This allows man-in-the-middle attackers to target initial connection attempts (where no host key for the server has been cached by the client). NOTE: some reports state that 8.5 and 8.6 are also affected.","score":"4.3 MEDIUM"}],"protocol":"tcp","port_id":"22","service":[["ssh"],["OpenSSH"],["8.2p1 Ubuntu 4ubuntu0.3"]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"2377","service":[["swarm"],[""],[""]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"7946","service":[["unknown"],[""],[""]]},{"cpes":["cpe:/a:influxdata:influxdb:2.1.1"],"advisories":[],"protocol":"tcp","port_id":"8086","service":[["http"],["InfluxDB http admin"],["2.1.1"]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"9100","service":[["jetdirect"],[""],[""]]},{"cpes":["cpe:/a:protocol_labs:go-ipfs"],"advisories":[],"protocol":"tcp","port_id":"9323","service":[["http"],["Golang net/http server"],[""]]}]},{"address":"DC:A6:32:F9:47:48","ports":[{"cpes":["cpe:/a:openbsd:openssh:8.2p1"],"advisories":[{"link":"https://nvd.nist.gov/vuln/detail/CVE-2021-41617","summary":"sshd in OpenSSH 6.2 through 8.x before 8.8, when certain non-default configurations are used, allows privilege escalation because supplemental groups are not initialized as expected. Helper programs for AuthorizedKeysCommand and AuthorizedPrincipalsCommand may run with privileges associated with group memberships of the sshd process, if the configuration specifies running the command as a different user.","score":"4.4 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2016-20012","summary":"OpenSSH through 8.7 allows remote attackers, who have a suspicion that a certain combination of username and public key is known to an SSH server, to test whether this suspicion is correct. This occurs because a challenge is sent only when that combination could be valid for a login session.","score":"4.3 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2021-28041","summary":"ssh-agent in OpenSSH before 8.5 has a double free that may be relevant in a few less-common scenarios, such as unconstrained agent-socket access on a legacy operating system, or the forwarding of an agent to an attacker-controlled host.","score":"4.6 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2020-15778","summary":"** DISPUTED ** scp in OpenSSH through 8.3p1 allows command injection in the scp.c toremote function, as demonstrated by backtick characters in the destination argument. NOTE: the vendor reportedly has stated that they intentionally omit validation of \"anomalous argument transfers\" because that could \"stand a great chance of breaking existing workflows.\"","score":"6.8 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2020-14145","summary":"The client side in OpenSSH 5.7 through 8.4 has an Observable Discrepancy leading to an information leak in the algorithm negotiation. This allows man-in-the-middle attackers to target initial connection attempts (where no host key for the server has been cached by the client). NOTE: some reports state that 8.5 and 8.6 are also affected.","score":"4.3 MEDIUM"}],"protocol":"tcp","port_id":"22","service":[["ssh"],["OpenSSH"],["8.2p1 Ubuntu 4ubuntu0.3"]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"2377","service":[["swarm"],[""],[""]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"7946","service":[["unknown"],[""],[""]]},{"cpes":["cpe:/a:influxdata:influxdb:2.1.1"],"advisories":[],"protocol":"tcp","port_id":"8086","service":[["http"],["InfluxDB http admin"],["2.1.1"]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"9100","service":[["jetdirect"],[""],[""]]},{"cpes":["cpe:/a:protocol_labs:go-ipfs"],"advisories":[],"protocol":"tcp","port_id":"9323","service":[["http"],["Golang net/http server"],[""]]}]},{"address":"192.168.1.23","ports":[{"cpes":["cpe:/a:openbsd:openssh:8.4"],"advisories":[{"link":"https://nvd.nist.gov/vuln/detail/CVE-2021-41617","summary":"sshd in OpenSSH 6.2 through 8.x before 8.8, when certain non-default configurations are used, allows privilege escalation because supplemental groups are not initialized as expected. Helper programs for AuthorizedKeysCommand and AuthorizedPrincipalsCommand may run with privileges associated with group memberships of the sshd process, if the configuration specifies running the command as a different user.","score":"4.4 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2016-20012","summary":"OpenSSH through 8.7 allows remote attackers, who have a suspicion that a certain combination of username and public key is known to an SSH server, to test whether this suspicion is correct. This occurs because a challenge is sent only when that combination could be valid for a login session.","score":"4.3 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2021-28041","summary":"ssh-agent in OpenSSH before 8.5 has a double free that may be relevant in a few less-common scenarios, such as unconstrained agent-socket access on a legacy operating system, or the forwarding of an agent to an attacker-controlled host.","score":"4.6 MEDIUM"},{"link":"https://nvd.nist.gov/vuln/detail/CVE-2020-14145","summary":"The client side in OpenSSH 5.7 through 8.4 has an Observable Discrepancy leading to an information leak in the algorithm negotiation. This allows man-in-the-middle attackers to target initial connection attempts (where no host key for the server has been cached by the client). NOTE: some reports state that 8.5 and 8.6 are also affected.","score":"4.3 MEDIUM"}],"protocol":"tcp","port_id":"22","service":[["ssh"],["OpenSSH"],["8.4"]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"5355","service":[["llmnr"],[""],[""]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"8443","service":[["https-alt"],[""],[""]]},{"cpes":[],"advisories":[],"protocol":"tcp","port_id":"9100","service":[["jetdirect"],[""],[""]]}]}]}[josevnz@dmaf5 ~]$

Is this web-service secure?

We exposed our Nmap scanner with no authorization, which means anyone who knows where the service is running can use it. This may not be a big issue on the local network, but it would be good to control who uses our precious resources.

How to add authentication and authorization

Right now anyone can call our service. It is a good idea to control who can run Nmap against our home network

There are several ways to make sure our web service can only be used by authorized clients. One way to do it is by requesting a client to provide a key that is also known to the server. This is the approach we'll follow here.

NOTE: As you might have guessed, if someone finds out the key then your service is compromised. To make it more secure you should:

• Stored the key in a safe place, encrypted

• Have an expiration date, to purge stale ones

• And transit of those keys should go over an encrypted channel, like HTTPS (we'll see about that soon)

We will take advantage of fastapi_simple_security to implement the API security access to our web application. It only requires a few new imports and that we declare a dependency on our REST API endpoints:

from fastapi import FastAPI, Depends
from fastapi_simple_security import api_key_router, api_key_security
from fastapi.responses import JSONResponse
from fastapi.encoders import jsonable_encoder
import typing
from home_nmap.system import HostIface
...
app: typing.Union[FastAPI] = FastAPI()
app.include_router(api_key_router, prefix="/auth", tags=["_auth"])

# Then add a 'dependencies' to each of the endpoints we want to secure
@app.get("/local_networks", dependencies=[Depends(api_key_security)])
def local_networks():
    """
    Get the available local networks where home_nmap runs
    @return: List with local networks in CIDR format
    """
    response = JSONResponse(jsonable_encoder(HostIface().get_local_networks()))
    return response
...

If we do not define a secret API key, the framework will provide us with one at startup (but you can override later through the documentation page):

(home_nmap) [josevnz@dmaf5 home_nmap]$ uuidgen 
23eb5572-1e63-4404-a64b-bcc18b62d4eb
(home_nmap) [josevnz@dmaf5 home_nmap]$ export FASTAPI_SIMPLE_SECURITY_SECRET="23eb5572-1e63-4404-a64b-bcc18b62d4eb"; uvicorn home_nmap.main:app --host 0.0.0.0 --port 8000 --reloadINFO:     Will watch for changes in these directories: ['/home/josevnz/Documents/home_nmap']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [134702] using watchgod
INFO:     Started server process [134704]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Now all the APIs that are protected by the keys have a different decoration in the documentation (a lock next to each endpoint):

Now documentation shows secured end points

What happens if we try to get the list of local networks, without our key?

josevnz@dmaf5 ~]$ curl 'http://127.0.0.1:8000/local_networks' --header 'accept: application/json'
{"detail":"An API key must be passed as query or header"}

In order to finish the setup, you need to enter your 'secret-key' (23eb5572-1e63-4404-a64b-bcc18b62d4eb) into the docs authentication page. Then go to the /auth/new to get the api-key, which is the one that your clients will use (header, cookie or part of the GET requests). In my case I got this:

curl 'http://127.0.0.1:8000/auth/new?never_expires=false' \
  --header 'accept: application/json' \
  --header 'secret-key: 23eb5572-1e63-4404-a64b-bcc18b62d4eb'
"e4c03730-02a1-4cb9-8e00-36a63930c064"

Now let's try again but passing our secret API key:

[josevnz@dmaf5 home_nmap]$ curl 'http://127.0.0.1:8000/local_networks'  --header 'accept: application/json' --header 'api-key: e4c03730-02a1-4cb9-8e00-36a63930c064'
["192.168.1.0/24"][josevnz@dmaf5 home_nmap]$

Still, we are not done yet. Assume that someone managed to run a sniffer on your network and is capturing all your HTTP traffic:

[josevnz@dmaf5 home_nmap]$ tshark -i eno1 -Px -Y http
Capturing on 'eno1'
   72 5.107984320 192.168.1.11 → 192.168.1.25 HTTP 219 GET /local_networks HTTP/1.1 

0000  1c 83 41 28 44 21 dc a6 32 f9 47 48 08 00 45 00   ..A(D!..2.GH..E.
0010  00 cd 7b ca 40 00 40 06 3a ec c0 a8 01 0b c0 a8   ..{.@.@.:.......
0020  01 19 b1 a6 1f 40 ce 1b 2a 22 ab b5 24 3c 80 18   .....@..*"..$<..
0030  01 f6 d0 3d 00 00 01 01 08 0a f3 07 ee 27 9d 96   ...=.........'..
0040  87 76 47 45 54 20 2f 6c 6f 63 61 6c 5f 6e 65 74   .vGET /local_net
0050  77 6f 72 6b 73 20 48 54 54 50 2f 31 2e 31 0d 0a   works HTTP/1.1..
0060  48 6f 73 74 3a 20 64 6d 61 66 35 2e 68 6f 6d 65   Host: dmaf5.home
0070  3a 38 30 30 30 0d 0a 55 73 65 72 2d 41 67 65 6e   :8000..User-Agen
0080  74 3a 20 63 75 72 6c 2f 37 2e 36 38 2e 30 0d 0a   t: curl/7.68.0..
0090  61 63 63 65 70 74 3a 20 61 70 70 6c 69 63 61 74   accept: applicat
00a0  69 6f 6e 2f 6a 73 6f 6e 0d 0a 61 70 69 2d 6b 65   ion/json..api-ke
00b0  79 3a 20 65 34 63 30 33 37 33 30 2d 30 32 61 31   y: e4c03730-02a1
00c0  2d 34 63 62 39 2d 38 65 30 30 2d 33 36 61 36 33   -4cb9-8e00-36a63
00d0  39 33 30 63 30 36 34 0d 0a 0d 0a                  930c064....

You can clearly see our not-so-secret-anymore API key. Time to add the next layer of protection.

We need encryption

The HTTP protocol is not encrypted. That means that someone using a sniffer (like tcpdump or wireshark) can capture the traffic. For example, if we request the home_nmap version using curl:

curl http://dmaf5.home:8000/version

It is possible for someone else running tshark to see all the traffic (look at the content-type: Application/ Json payload):

root@dmaf5 ~]# tshark -i eno1 -Px -Y http
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eno1'
  127 4.342379691 192.168.1.11 → 192.168.1.23 HTTP 152 GET /version HTTP/1.1 

0000  1c 83 41 28 44 21 dc a6 32 f9 47 48 08 00 45 00   ..A(D!..2.GH..E.
0010  00 8a c3 8a 40 00 40 06 f3 70 c0 a8 01 0b c0 a8   ....@.@..p......
0020  01 17 c7 68 1f 40 dc af 3c 37 c1 12 e6 69 80 18   ...h.@..<7...i..
0030  01 f6 ff a7 00 00 01 01 08 0a 08 94 d3 55 a8 7c   .............U.|
0040  ec df 47 45 54 20 2f 76 65 72 73 69 6f 6e 20 48   ..GET /version H
0050  54 54 50 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 64   TTP/1.1..Host: d
0060  6d 61 66 35 2e 68 6f 6d 65 3a 38 30 30 30 0d 0a   maf5.home:8000..
0070  55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c   User-Agent: curl
0080  2f 37 2e 36 38 2e 30 0d 0a 41 63 63 65 70 74 3a   /7.68.0..Accept:
0090  20 2a 2f 2a 0d 0a 0d 0a                            */*....

  129 4.344312849 192.168.1.23 → 192.168.1.11 HTTP/JSON 210 HTTP/1.1 200 OK , JavaScript Object Notation (application/json)

0000  dc a6 32 f9 47 48 1c 83 41 28 44 21 08 00 45 00   ..2.GH..A(D!..E.
0010  00 c4 36 78 40 00 40 06 80 49 c0 a8 01 17 c0 a8   ..6x@.@..I......
0020  01 0b 1f 40 c7 68 c1 12 e6 69 dc af 3c 8d 80 18   ...@.h...i..<...
0030  01 fd 84 29 00 00 01 01 08 0a a8 7c ec e1 08 94   ...).......|....
0040  d3 55 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f   .UHTTP/1.1 200 O
0050  4b 0d 0a 64 61 74 65 3a 20 4d 6f 6e 2c 20 31 37   K..date: Mon, 17
0060  20 4a 61 6e 20 32 30 32 32 20 32 30 3a 31 36 3a    Jan 2022 20:16:
0070  32 39 20 47 4d 54 0d 0a 73 65 72 76 65 72 3a 20   29 GMT..server: 
0080  75 76 69 63 6f 72 6e 0d 0a 63 6f 6e 74 65 6e 74   uvicorn..content
0090  2d 6c 65 6e 67 74 68 3a 20 31 39 0d 0a 63 6f 6e   -length: 19..con
00a0  74 65 6e 74 2d 74 79 70 65 3a 20 61 70 70 6c 69   tent-type: appli
00b0  63 61 74 69 6f 6e 2f 6a 73 6f 6e 0d 0a 0d 0a 7b   cation/json....{
00c0  22 76 65 72 73 69 6f 6e 22 3a 22 30 2e 30 2e 31   "version":"0.0.1
00d0  22 7d                                             "}

We can protect our traffic by encrypting it using Hypertext Transfer Protocol Secure (HTTPS).

How to create the Secure Socket Layer (SSL) certificates

Let me show you real quick how you can install a self-signed server certificate on Fedora using Cloudflare cfssl. First let's install the tools:

# On Fedora just do 
sudo dnf install -y golang-github-cloudflare-cfssl
# Or go get github.com/cloudflare/cfssl/cmd/...

Next step is to create a certificate authority (CA). We will use it to sign other certificates. For that let's create a definition in JSON format:

{
  "CN": "Nunez Barrios family Root CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "L": "CT",
    "O": "Nunez Barrios",
    "OU": "Nunez Barrios Root CA",
    "ST": "United States"
  }
 ]
}

Create the certificate:

cfssl gencert -initca ca.json | cfssljson -bare ca

Next we need to create a profile file (cfssl.json), that will specify certain features of the certificates, like expiration in 2 years:

{
  "signing": {
    "default": {
      "expiry": "17532h"
    },
    "profiles": {
      "intermediate_ca": {
        "usages": [
            "signing",
            "digital signature",
            "key encipherment",
            "cert sign",
            "crl sign",
            "server auth",
            "client auth"
        ],
        "expiry": "17532h",
        "ca_constraint": {
            "is_ca": true,
            "max_path_len": 0, 
            "max_path_len_zero": true
        }
      },
      "peer": {
        "usages": [
            "signing",
            "digital signature",
            "key encipherment", 
            "client auth",
            "server auth"
        ],
        "expiry": "17532h"
      },
      "server": {
        "usages": [
          "signing",
          "digital signing",
          "key encipherment",
          "server auth"
        ],
        "expiry": "17532h"
      },
      "client": {
        "usages": [
          "signing",
          "digital signature",
          "key encipherment", 
          "client auth"
        ],
        "expiry": "17532h"
      }
    }
  }
}

Now we create an intermediate certificate (intermediate-ca.json) that will expire in 5 years:

{
  "CN": "Barrios Nunez Intermediate CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C":  "US",
      "L":  "CT",
      "O":  "Barrios Nunez",
      "OU": "Barrios Nunez Intermediate CA",
      "ST": "USA"
    }
  ],
  "ca": {
    "expiry": "43830h"
  }
}

Here's the command to do it:

cfssl gencert -initca intermediate-ca.json | cfssljson -bare intermediate_ca
cfssl sign -ca ca.pem -ca-key ca-key.pem -config cfssl.json -profile intermediate_ca intermediate_ca.csr | cfssljson -bare intermediate_ca

Next step is to create the host certificates

You will need to put your fully-qualified host name (hostname -f) on the host-1.json file. Also, some software expects the IP address (ip address|grep inet) – we will do both:

{
  "CN": "dmaf5.home",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
  {
    "C": "US",
    "L": "CT",
    "O": "Barrios Nunez",
    "OU": "Barrios Nunez Hosts",
    "ST": "USA"
  }
  ],
  "hosts": [
    "dmaf5.home",
    "localhost",
    "dmaf5",
    "192.168.1.23",
    "192.168.1.26"
  ]
}

You can create three certificate types:

• client

• server

• peer

We'll use only the server certificate, but we'll create all three:

cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=peer host-1.json| cfssljson -bare host-1-peer  # Peer
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=server host-1.json | cfssljson -bare host-1-server  # Server
cfssl gencert -ca intermediate_ca.pem -ca-key intermediate_ca-key.pem -config cfssl.json -profile=client host-1.json | cfssljson -bare host-1-client  # Client

We are very close now. Install the intermediate certificate into the proper location so the clients on dmaf5 do not complain about the self-signed certificate:

# The path below is for Fedora, please check your OS documentation to find the right path for you
sudo /bin/cp --preserve --verbose tutorial/intermediate_ca.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

Restart uvicorn to listen now only on a secure port, using the host key and certificates we just created:

(home_nmap) [josevnz@dmaf5 home_nmap]$ uvicorn home_nmap.main:app --host 0.0.0.0 --port 8443 --reload --ssl-keyfile=$PWD/tutorial/host-1-server-key.pem --ssl-certfile=$PWD/tutorial/host-1-server.pem
INFO:     Will watch for changes in these directories: ['/home/josevnz/Documents/home_nmap']
INFO:     Uvicorn running on https://0.0.0.0:8443 (Press CTRL+C to quit)
INFO:     Started reloader process [166275] using watchgod
INFO:     Started server process [166277]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     192.168.1.23:47704 - "GET /version HTTP/1.1" 200 OK

And then test with curl (without the --insecure flag, no complaints from curl):

[josevnz@dmaf5 ~]$ curl --fail https://dmaf5.home:8443/version
{"version":"0.0.1"}[josevnz@dmaf5 ~]$

Try again to capture the version of our service using tshark:

# 'tshark -i eno1 -Px -Y http' doesn't work anymore as the payload is encrypted. So at least lets see how the SSL hello goes
tshark -i eno1 -Y ssl -Px
  343 59.344539258 192.168.1.11 → 192.168.1.23 TLSv1 583 Client Hello

0000  1c 83 41 28 44 21 dc a6 32 f9 47 48 08 00 45 00   ..A(D!..2.GH..E.
0010  02 39 8b 6b 40 00 40 06 29 e1 c0 a8 01 0b c0 a8   .9.k@.@.).......
0020  01 17 93 14 20 fb 10 10 d7 6f 7d ff f7 c1 80 18   .... ....o}.....
0030  01 f6 0b fe 00 00 01 01 08 0a 08 a5 00 20 a8 8d   ............. ..
0040  27 47 16 03 01 02 00 01 00 01 fc 03 03 39 03 ac   'G...........9..
0050  19 7c bd 38 dc e2 cf 72 8b 7e 00 e2 2d fc 68 7a   .|.8...r.~..-.hz
0060  cc af 9c d6 d5 1d ed 94 79 b2 0f c8 cf 20 a3 f8   ........y.... ..
0070  2a 8e 20 c0 d2 c1 57 ee 36 48 2e 8f 46 e7 da 76   *. ...W.6H..F..v
0080  69 67 d1 9d 5a 70 24 0e 7d ea ec 8b e2 a0 00 3e   ig..Zp$.}......>
0090  13 02 13 03 13 01 c0 2c c0 30 00 9f cc a9 cc a8   .......,.0......
00a0  cc aa c0 2b c0 2f 00 9e c0 24 c0 28 00 6b c0 23   ...+./...$.(.k.#
00b0  c0 27 00 67 c0 0a c0 14 00 39 c0 09 c0 13 00 33   .'.g.....9.....3
00c0  00 9d 00 9c 00 3d 00 3c 00 35 00 2f 00 ff 01 00   .....=.<.5./....
00d0  01 75 00 00 00 0f 00 0d 00 00 0a 64 6d 61 66 35   .u.........dmaf5
00e0  2e 68 6f 6d 65 00 0b 00 04 03 00 01 02 00 0a 00   .home...........
00f0  0c 00 0a 00 1d 00 17 00 1e 00 19 00 18 33 74 00   .............3t.
0100  00 00 10 00 0e 00 0c 02 68 32 08 68 74 74 70 2f   ........h2.http/
0110  31 2e 31 00 16 00 00 00 17 00 00 00 31 00 00 00   1.1.........1...
0120  0d 00 2a 00 28 04 03 05 03 06 03 08 07 08 08 08   ..*.(...........
0130  09 08 0a 08 0b 08 04 08 05 08 06 04 01 05 01 06   ................
0140  01 03 03 03 01 03 02 04 02 05 02 06 02 00 2b 00   ..............+.

Note that it is possible to capture the traffic and decrypt it later if you have access to the private key. That's why it is so important that you keep that file secure.

What about our authorized request using the API key + encryption?

josevnz@raspberrypi:~$ curl 'https://dmaf5.home:8443/local_networks' --header 'accept: application/json' --header 'api-key: e4c03730-02a1-4cb9-8e00-36a63930c064'
["192.168.1.0/24"]

Our application setup is now complete.

What did we learn?

In this article, we covered many topics and went from a very simple XML parser to a self documenting web service. Not bad for a single session!

You should know about the following topics now:

• How to parse an Nmap XML results file, and enrich it with security advisories from NIST

• How to enhance Nmap by mixing it with other scripts to automate its execution

• How to apply Nmap options to make our local network scan faster

• Understand what is pivoting and how you can use it to bypass firewall protections with the help of SSH and tcpproxy

• How to write a REST-API on top of our original CLI script and secure it with SSL and basic authentication

• How to add authorization to a web service using an API key

• How to use tshark to demonstrate how HTTP traffic can be captured, and show the data payload

• How to add encryption to a web service, by creating self-signed certificates

And what else could you learn? Here are some final suggestions:

• Check out the official Nmap documentation.

• The Operating system fingerprinting is fascinating. Figuring out what exactly runs behind a port is an art and a moving target.

• Integration with other great penetration testing tools like Metasploit, which you guessed, can also be scripted in Ruby!

• Also, as a bonus you have my code that can be installed using pip and can run some unit tests with unittest. I welcome pull requests and suggestions.

Feel free to reach out with your comments and bug-reports. I hope you enjoy it using it as much I enjoyed writing it.