That sounds simple enough, but even a minimal module produces a surprising amount of machinery around it: object files, metadata, exported and unresolved symbols, and a final .ko file that is quite different from an ordinary executable.

Here's a complete, working Linux kernel module. It's just twenty-two lines, seven of which are includes and metadata:

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Chris Roy");
MODULE_DESCRIPTION("A minimal loadable kernel module");
MODULE_VERSION("0.1");

static int __init hello_init(void)
{
    pr_info("hello: loaded, module at %pS\n", hello_init);
    return 0;
}

static void __exit hello_exit(void)
{
    pr_info("hello: unloaded\n");
}

module_init(hello_init);
module_exit(hello_exit);

Compiled on the machine I'm writing this on, that produces a file of about 106,000 bytes. Strip the debug information out and the same module is 4,864 bytes. Ninety-five percent of what the build gave you isn't code.

Your total will differ from mine, and not by a predictable amount. Part of it is where you built: the debug information records the directory you compiled in, so a deeply nested path costs a few hundred bytes that a short one doesn't. Your compiler version and kernel configuration move it further. The proportion is what holds. The exact byte count is only what this machine produced.

That gap is a good place to start, because most kernel module tutorials show you the listing above, tell you to run make, and stop.

This one follows what the build actually produced, what your module already depends on before you wrote anything useful, and why the tutorial you found from 2014 no longer compiles.

Table of Contents

• What You Need

• The Smallest Module That Works

• The Makefile is Stranger Than it Looks

• What the Build Actually Did

• What's Inside a .ko File

• Your Hello World Already Depends on Three Things

• vermagic, and Why Your Module Refuses to Load

• Passing Parameters at Load Time

• Loading it, and Where the Output Goes

• Four Build Errors and What They Mean

• Why the Tutorial You Found Doesn't Compile

• Conclusion

• Epilogue

What You Need

To follow along here, you'll need a Linux machine you're willing to load code into, the headers for the kernel you're running, and a compiler.

On Debian or Ubuntu:

sudo apt install build-essential linux-headers-$(uname -r)

On Fedora, the equivalent is kernel-devel and kernel-headers, and on Arch it's the linux-headers package matching your kernel.

Check that the headers landed where the build expects them:

ls -d /lib/modules/$(uname -r)/build

That path is a symlink into the headers package, and its absence is the single most common reason a module build fails with an error that mentions nothing about headers.

Two things will stop you from loading a module even after it builds. Secure Boot rejects unsigned modules, and kernel lockdown blocks loading in confidentiality mode. Check both:

mokutil --sb-state
cat /sys/kernel/security/lockdown

On the machine here, Secure Boot is disabled and lockdown reports [none] integrity confidentiality, with the brackets marking the active mode. If yours shows Secure Boot enabled, you'll need to sign the module or disable Secure Boot in firmware before it will load.

I'm on Ubuntu 22.04 with kernel 5.15.0-190-generic and gcc 11.4. Your versions will differ, and the article says where that matters.

The Smallest Module That Works

Save the code from the beginning of this article as hello.c. Here it is again for reference:

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Chris Roy");
MODULE_DESCRIPTION("A minimal loadable kernel module");
MODULE_VERSION("0.1");

static int __init hello_init(void)
{
    pr_info("hello: loaded, module at %pS\n", hello_init);
    return 0;
}

static void __exit hello_exit(void)
{
    pr_info("hello: unloaded\n");
}

module_init(hello_init);
module_exit(hello_exit);

Four things in it are doing real work.

module_init and module_exit register the functions the kernel calls when your module is loaded and unloaded. They aren't main. A module has no single entry point that runs and returns. It has hooks that fire on two specific events, and it does nothing in between unless something else calls into it.

__init and __exit are section markers. __init tells the kernel this code runs once and its memory can be freed afterward, which is why you'll see "Freeing unused kernel memory" in your boot log. __exit tells the build that this function is only needed if the module can be unloaded at all.

MODULE_LICENSE("GPL") isn't paperwork. The kernel checks it at load time, and a module declaring a non-GPL license is denied access to symbols marked EXPORT_SYMBOL_GPL, which is most of the interesting ones. Omit the macro entirely and the kernel taints itself and logs a complaint.

pr_info is the modern spelling of printk(KERN_INFO ...). It writes to the kernel ring buffer, not to your terminal, which trips up nearly everyone the first time.

The MODULE_AUTHOR, MODULE_DESCRIPTION, and MODULE_VERSION macros are metadata rather than behavior, and they end up in the file for modinfo to read. Leave them out and nothing breaks, but recent kernels warn at build time about a missing MODULE_DESCRIPTION, which is reason enough to write all three from the start.

The Makefile is Stranger Than it Looks

obj-m += hello.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

This looks like a Makefile, and mostly isn't one. obj-m += hello.o isn't a Make variable you invented. It's a declaration read by kbuild, the kernel's own build system.

The make -C line changes directory into the kernel headers and runs the kernel's build system there, passing M=$(PWD) to say "the module source is over here." Your Makefile is a thin wrapper that hands the job to a build system you didn't write and can't easily replace.

That indirection is why module builds fail in ways that seem unrelated to your code. You're not compiling against the kernel headers the way you compile against libc headers. You're running the kernel's build, on your file, with its flags and its rules.

What the Build Actually Did

Run make and read the output rather than skipping it:

make -C /lib/modules/5.15.0-190-generic/build M=/home/chris/lkm modules
make[1]: Entering directory '/usr/src/linux-headers-5.15.0-190-generic'
  CC [M]  /home/chris/lkm/hello.o
  MODPOST /home/chris/lkm/Module.symvers
  CC [M]  /home/chris/lkm/hello.mod.o
  LD [M]  /home/chris/lkm/hello.ko
  BTF [M] /home/chris/lkm/hello.ko
Skipping BTF generation for /home/chris/lkm/hello.ko due to unavailability of vmlinux

Five steps, and only the first is the compile you expected.

CC [M] hello.o compiles your source. Ordinary.

MODPOST is the step worth knowing about. It scans your object file for symbols you referenced but didn't define, checks each one against the kernel's table of exported symbols, and fails the build if you used something the kernel doesn't offer you. It also generates hello.mod.c, a small file of glue containing your module's metadata.

CC [M] hello.mod.o compiles that generated glue, and LD [M] links it together with your object into the final .ko.

BTF [M] would attach type information used by tracing tools. Here it was skipped, because generating BTF needs the uncompressed vmlinux image and Ubuntu doesn't ship it by default. The build warns and continues, which is correct: BTF is useful, not required.

What's Inside a .ko File

A .ko is an ordinary ELF object with kernel-specific sections bolted on. Look at its metadata:

modinfo ./hello.ko

version:        0.1
description:    A minimal loadable kernel module
author:         Chris Roy
license:        GPL
srcversion:     39D86510C9FF65D797EAF90
depends:        
retpoline:      Y
name:           hello
vermagic:       5.15.0-190-generic SMP mod_unload modversions

All of that lives in one ELF section, stored as null-separated strings. You can read it straight out of the file:

objcopy -O binary --only-section=.modinfo hello.ko /dev/stdout | tr '\0' '\n'

Which brings us back to the number from the opening. The module is about 106,000 bytes on disk:

ls -l hello.ko
cp hello.ko /tmp/ && strip --strip-debug /tmp/hello.ko && ls -l /tmp/hello.ko

105984  hello.ko
  4864  /tmp/hello.ko

The actual module is under five kilobytes. Everything else is DWARF debug information the build keeps so that tools like crash and gdb can make sense of your code if it panics. When you load the module, the kernel doesn't load the debug sections, so the memory cost is the small number rather than the large one.

Your Hello World Already Depends on Three Things

This is the part I'd have wanted someone to show me first. Ask the object what it needs from the kernel:

nm -u hello.ko

U __fentry__
U _printk
U __x86_return_thunk

U means undefined: symbols your module references and the kernel must supply at load time.

_printk you can account for, since pr_info expands to it.

__fentry__ is a call the compiler placed at the top of every one of your functions, because the kernel is built with function tracing support. Every function you write in a module gets that hook whether you asked for it or not, and it's what lets ftrace instrument your code later without recompiling anything.

__x86_return_thunk is a Spectre mitigation. Your compiler replaced ordinary return instructions with a call to a thunk that avoids the speculative execution path the vulnerability relies on. It appears in a module that prints one line, because the mitigation applies to all kernel code on this machine, module or not.

Two of the three symbols in your hello world are infrastructure the machine imposed on you. That's a fair picture of what writing kernel code is like.

MODPOST verified all three exist before the link succeeded. Had you called a function the kernel doesn't export, the build would have failed there with an "undefined symbol" error rather than producing a module that fails at load.

vermagic, and Why Your Module Refuses to Load

Look again at that line from modinfo:

vermagic: 5.15.0-190-generic SMP mod_unload modversions

The kernel compares that string against its own before loading anything, and refuses on a mismatch. It covers the release, whether the kernel is SMP, whether module unloading is compiled in, and whether symbol versioning is on.

This is why a module built on one machine usually won't load on another, and why upgrading your kernel means rebuilding your modules.

There's no ABI stability guarantee inside the Linux kernel. Internal structures change between releases, and a module compiled against one layout that ran against another would corrupt memory rather than fail cleanly. Refusing to load is the kernel being careful.

It's also why DKMS exists. If you have VirtualBox, ZFS, or an NVIDIA driver installed, you already have a module being rebuilt this way. On the machine here:

modinfo vboxdrv | head -3

filename:       /lib/modules/5.15.0-190-generic/updates/dkms/vboxdrv.ko
version:        6.1.50_Ubuntu r161033 (0x00320000)
license:        GPL

Note the updates/dkms/ in that path. DKMS keeps the source and rebuilds the module each time you install a new kernel, which is the maintenance cost the vermagic check makes unavoidable.

Passing Parameters at Load Time

A module that always does the same thing is rarely what you want. module_param exposes a variable so its value can be set at load time. Save this as param.c alongside hello.c:

#include <linux/init.h>
#include <linux/module.h>

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("A module that takes parameters");

static char *who = "world";
static int times = 1;

module_param(who, charp, 0444);
MODULE_PARM_DESC(who, "who to greet");
module_param(times, int, 0644);
MODULE_PARM_DESC(times, "how many times to greet");

static int __init param_init(void)
{
    int i;

    for (i = 0; i < times; i++)
        pr_info("param: hello, %s\n", who);
    return 0;
}

static void __exit param_exit(void)
{
    pr_info("param: unloaded\n");
}

module_init(param_init);
module_exit(param_exit);

Add it to the Makefile, which takes a list:

obj-m += hello.o param.o

The three arguments are the variable, its type, and the permissions on the file that will represent it under /sys/module/<name>/parameters/. A mode of 0444 makes it readable and fixed once loaded. 0644 lets root write to that file and change the value while the module is running, which is useful. It's also how you introduce a race if the module reads the variable without expecting it to change.

charp is a char pointer, and the other common types are int, bool, long and charp arrays via module_param_array. Pass a type that doesn't match the variable and the build fails rather than misbehaving later.

MODULE_PARM_DESC puts the description into the module metadata, where modinfo finds it:

name:           param
parm:           who:who to greet (charp)
parm:           times:how many times to greet (int)

Set them at load time as name=value pairs:

sudo insmod ./param.ko who=kernel times=3

Anyone can read what parameters a module accepts before loading it, which is the main reason to bother with MODULE_PARM_DESC at all.

Loading it, and Where the Output Goes

sudo insmod ./hello.ko
sudo dmesg | tail -2
lsmod | grep hello
sudo rmmod hello

The pr_info output goes to the kernel ring buffer, so it appears in dmesg rather than your terminal. If dmesg refuses without root, that's kernel.dmesg_restrict, and sudo journalctl -k | tail reads the same messages through the journal.

The %pS in the format string prints a kernel pointer as a symbol name and offset instead of a raw address, which is how you get something readable out of a log line rather than a hexadecimal number.

lsmod reads /proc/modules and shows three columns: the module name, its size in memory, and a use count with the names of anything depending on it. A module with a non-zero use count can't be unloaded, which is the most common reason rmmod refuses.

One caution before you load anything. A bug in userspace crashes your program. But a bug here can take the machine down or corrupt a filesystem. Do this in a virtual machine the first several times. The cost of a snapshot is far lower than the cost of a corrupted disk.

Four Build Errors and What They Mean

These four account for most of the time people lose, and each says something specific once you know what to read.

No rule to make target 'modules'. Stop. The kernel headers are missing or the symlink at /lib/modules/$(uname -r)/build points nowhere. Install the headers package matching the exact kernel you're running, which is often not the newest one installed if you haven't rebooted since an update.

ERROR: modpost: "some_function" [hello.ko] undefined! You referenced a symbol the kernel doesn't export. Here's what that looks like from a real build:

ERROR: modpost: "this_symbol_does_not_exist" [bad.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:133: Module.symvers] Error 1

Retrying won't help. Either the function is internal to the kernel and never exported, or it's exported with EXPORT_SYMBOL_GPL and your module declares a non-GPL license. Check with grep the_symbol /proc/kallsyms, where a capital T in the second column means it's a global text symbol.

insmod: ERROR: could not insert module: Invalid module format: The build succeeded but vermagic doesn't match the running kernel. Compare modinfo ./hello.ko | grep vermagic against uname -r. Rebuilding against the correct headers fixes it.

insmod: ERROR: could not insert module: Operation not permitted: Usually Secure Boot rejecting an unsigned module, or lockdown in confidentiality mode. Check mokutil --sb-state and cat /sys/kernel/security/lockdown before assuming your code is at fault.

One more thing, which is easier to learn now than to debug later. Loading any out-of-tree module sets a taint flag on the kernel, which is recorded and reported in any subsequent oops or panic:

cat /proc/sys/kernel/tainted

The value is a bitmask. It reads 4096 on the machine here, which is bit 12, TAINT_OOT_MODULE, meaning an out-of-tree module has been loaded at some point.

Bit 13 is the neighboring one people confuse it with, TAINT_UNSIGNED_MODULE, which is what Secure Boot cares about.

The full list is in include/linux/panic.h in the kernel source. Kernel developers will ask you to reproduce a bug on an untainted kernel before they look at it, and this is the file that tells them whether you did.

Why the Tutorial You Found Doesn't Compile

Most module tutorials on the web predate several changes, and these are the ones that bite.

printk(KERN_INFO "...") still works, but pr_info is the current spelling and carries the log level for you.

init_module and cleanup_module as bare function names were the old convention. Use module_init and module_exit with your own names instead, which lets a file define both without collisions.

MODULE_LICENSE was once optional in practice. It's now load-bearing, since it gates access to GPL-only exported symbols.

The M= argument used to be spelled SUBDIRS=. That spelling was removed, and a tutorial using it fails with an error that doesn't mention SUBDIRS anywhere.

Header paths moved. Anything referring to /usr/src/linux predates the split into per-kernel headers packages and is old enough that the rest of it needs checking too. A smaller sign: <linux/module.h> has included <linux/moduleparam.h> for years, so a tutorial that carefully includes both is copying from something old, even though including both is harmless.

If a tutorial builds without warnings on your kernel, it's current enough. If it doesn't, the kernel version it targeted is usually printed in the first error.

Conclusion

You can now build a kernel module, read what the build produced, and explain every symbol it depends on.

More usefully, you know why it fails in the specific ways it does. A missing /lib/modules/$(uname -r)/build is a headers problem. A vermagic mismatch is a rebuild. An undefined symbol at MODPOST means the kernel doesn't export what you asked for, and no amount of retrying will change that.

There are a few directions to go from here. Register a /proc entry with proc_create and read from it, which is the smallest useful thing a module can do.

Read the kernel's own Module.symvers under /usr/src/linux-headers-$(uname -r)/ to see the table MODPOST checked against, which is 26,420 exported symbols on this machine and marks each one EXPORT_SYMBOL or EXPORT_SYMBOL_GPL.

Or trace your own module's functions, which works because of the __fentry__ hook that was there from the first build. There's no ftrace command to run. It's an interface under /sys/kernel/tracing, so you drive it by writing to files:

sudo sh -c 'echo hello_init > /sys/kernel/tracing/set_ftrace_filter'
sudo sh -c 'echo function > /sys/kernel/tracing/current_tracer'
sudo cat /sys/kernel/tracing/trace

On older systems, that path is /sys/kernel/debug/tracing instead. If you would rather not write to files by hand, trace-cmd wraps the same interface.

Epilogue

Everything above assumes you're allowed to do it, and that assumption is the part I find interesting. A loaded module runs with the same authority as the kernel itself. It can read any memory, patch any function, and ignore any policy the system thought it was enforcing, because by the time it runs there's nothing left above it to say no.

That makes module loading the one operation a permission model can't contain, which is why the kernel guards it with signatures and lockdown rather than with permissions.

I ran into that floor while working on a capability-backed desktop OS, one where a program's manifest is the whole of what it may do, and the Debian ecosystem still has to work underneath it. Modules are where that model stops being expressible, so working out exactly what the kernel checks before accepting one stopped being a detail and became a design constraint.

I write about systems and their mysteries at thechris.in.

without landlock:
  read /etc/hostname            ok
  read /tmp/secret.txt          ok
with landlock, /etc allowed:
  read /etc/hostname            ok
  read /tmp/secret.txt          FAILED (Permission denied)

There's no root, no container, no configuration file, and no daemon. The program asked the kernel to take away its own access to most of the filesystem, and the kernel obliged.

That's Landlock, which has been in the kernel since 2021 without most people noticing. This article builds that program from nothing, runs it, and then walks into the four surprises that catch people the first time.

Table of Contents

• What You Need

• Where Landlock Sits

• Three System Calls and No Library

• A Program That Restricts Itself

• Why no_new_privs is Mandatory

• Rulesets Intersect, They Never Widen

• What Your Children Inherit

• The exec Trap

• Wrapping a Program You Didn't Write

• Finding the Paths a Program Needs

• Which ABI Version You Have

• Conclusion

• Epilogue

What You Need

To follow along, you'll need a kernel of 5.13 or newer, the standard headers, and a C compiler. Nothing else, and notably not root.

grep landlock /sys/kernel/security/lsm
ls /usr/include/linux/landlock.h

The first command matters. Landlock can be compiled into a kernel and still be inactive, because Linux Security Modules have to be enabled at boot. On this machine, that file reads lockdown,capability,landlock,yama,apparmor.

If landlock is missing from yours, add lsm=landlock, to the front of the existing list in your kernel command line and reboot. Ubuntu has shipped it enabled since 22.04, and current Fedora and Arch kernels carry it too, but the grep above is the only answer that counts for your machine.

Everything below was run on kernel 5.15.0-190-generic under Ubuntu 22.04.5, compiled with gcc 11.4, as an ordinary user with no sudo anywhere.

Where Landlock Sits

Linux Security Modules are a framework, not a policy. The kernel calls out to LSM hooks at decision points, before opening a file, creating a process, or mapping executable memory, and whatever modules are loaded get to say yes or no.

SELinux and AppArmor are the two most people have heard of, and both are administrator tools: someone with root writes a policy, the system loads it, and your program lives inside whatever that policy says.

Landlock inverts that. It's the first LSM a process can apply to itself, without privilege, at runtime. You don't need to convince an administrator that your program deserves a policy. The program asks for less than it currently has, and the kernel narrows it.

That "asks for less" is the whole design. Landlock can only ever remove access. There's no call that grants you something you didn't already have, which is precisely why it's safe to expose to unprivileged processes.

Three System Calls and No Library

Landlock is three syscalls and glibc wraps none of them, so you call them directly through syscall():

static int create_ruleset(const struct landlock_ruleset_attr *attr)
{ return syscall(__NR_landlock_create_ruleset, attr, sizeof(*attr), 0); }

static int add_rule(int fd, const struct landlock_path_beneath_attr *pb)
{ return syscall(__NR_landlock_add_rule, fd, LANDLOCK_RULE_PATH_BENEATH, pb, 0); }

static int restrict_self(int fd)
{ return syscall(__NR_landlock_restrict_self, fd, 0); }

landlock_create_ruleset declares which kinds of access you intend to govern and returns a file descriptor representing the ruleset. landlock_add_rule adds an exception in the form of a directory you want to keep. Finally, landlock_restrict_self applies the whole thing to the calling process, permanently.

The handled_access_fs field in the ruleset attribute is the part people get backwards. It doesn't list what you're allowing. It lists the access types this ruleset is responsible for, and anything in that list is denied everywhere except the paths you explicitly add. Handle read access and you lose read access to the entire filesystem until you add rules back.

A Program That Restricts Itself

Here is the whole thing:

#define _GNU_SOURCE
#include <linux/landlock.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>

#define READ_RIGHTS (LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR)

static int create_ruleset(const struct landlock_ruleset_attr *attr)
{ return syscall(__NR_landlock_create_ruleset, attr, sizeof(*attr), 0); }

static int add_rule(int fd, const struct landlock_path_beneath_attr *pb)
{ return syscall(__NR_landlock_add_rule, fd, LANDLOCK_RULE_PATH_BENEATH, pb, 0); }

static int restrict_self(int fd)
{ return syscall(__NR_landlock_restrict_self, fd, 0); }

static int allow_read(int ruleset_fd, const char *path)
{
    struct landlock_path_beneath_attr pb = { .allowed_access = READ_RIGHTS };
    int rc;

    pb.parent_fd = open(path, O_PATH | O_CLOEXEC);
    if (pb.parent_fd < 0) { perror(path); return -1; }
    rc = add_rule(ruleset_fd, &pb);
    close(pb.parent_fd);
    return rc;
}

static void try_read(const char *path)
{
    int fd = open(path, O_RDONLY);

    if (fd < 0)
        printf("  read %-24s FAILED (%s)\n", path, strerror(errno));
    else
        { printf("  read %-24s ok\n", path); close(fd); }
}

int main(void)
{
    struct landlock_ruleset_attr attr = { .handled_access_fs = READ_RIGHTS };
    int ruleset_fd = create_ruleset(&attr);

    if (ruleset_fd < 0) { perror("landlock_create_ruleset"); return 1; }
    if (allow_read(ruleset_fd, "/etc") < 0) return 1;

    if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)) { perror("prctl"); return 1; }
    if (restrict_self(ruleset_fd)) { perror("landlock_restrict_self"); return 1; }
    close(ruleset_fd);

    printf("with landlock, /etc allowed:\n");
    try_read("/etc/hostname");
    try_read("/tmp/secret.txt");
    return 0;
}

Build and run it:

gcc -Wall -o sandbox sandbox.c
echo "hunter2" > /tmp/secret.txt
./sandbox

with landlock, /etc allowed:
  read /etc/hostname            ok
  read /tmp/secret.txt          FAILED (Permission denied)

Two details in there matter. The rule refers to a directory by an open file descriptor rather than a path string, opened with O_PATH so you get a handle without needing read permission on the directory itself. And restrict_self takes effect immediately for the calling process, with no way to undo it.

Why no_new_privs is Mandatory

Take the prctl call out and the program stops working:

landlock_restrict_self -> Operation not permitted

That's EPERM, and it's deliberate. PR_SET_NO_NEW_PRIVS tells the kernel that this process and its descendants can never gain privileges through execve, which is what stops a sandboxed process from escaping by running a setuid binary.

Without that guarantee, a restricted process could exec sudo or any setuid program and step outside the restrictions you just applied. Landlock refuses to apply itself at all rather than offer a sandbox with that hole in it. Set no_new_privs first, every time.

Rulesets Intersect, They Never Widen

This is the property to get right. Apply a ruleset allowing /etc, then apply a second allowing /tmp, and ask what you can reach:

after first ruleset:  /etc=ok               /tmp=Permission denied
after second ruleset: /etc=Permission denied  /tmp=Permission denied

The second ruleset didn't add /tmp. It took away /etc, and left you with nothing.

Each restrict_self intersects with everything already applied. The first ruleset permitted /etc and denied the rest. The second permitted /tmp and denied the rest. What survives is the overlap of those two, which is empty.

So a Landlock sandbox is a ratchet. Every application can only tighten, never loosen, and there's no operation anywhere in the API that widens what a restricted process may do. If you need a process to have access to two directories, both rules go into one ruleset before you apply it.

That also means you can't change your mind. A long-running process that restricts itself early can't be granted more later, by itself or by anyone else, short of starting a new process.

What Your Children Inherit

Restrictions follow fork without asking:

parent:       /etc=ok   /tmp/secret.txt=Permission denied
forked child: /etc=ok   /tmp/secret.txt=Permission denied

The child inherits the parent's Landlock domain exactly and there's no flag to opt out. The same holds across execve, which is the point of no_new_privs: the new program starts already inside the sandbox the old one built.

This is what makes Landlock useful for wrapping something you didn't write. Restrict yourself, then exec the thing you want contained, and it runs inside your restrictions without knowing they exist.

The exec Trap

It also sets a trap. Take the program above, keep only /etc allowed, and try to exec anything:

allowed: /etc
execl(/usr/bin/cat) failed: Permission denied

Executing a binary requires reading it. This ruleset handles LANDLOCK_ACCESS_FS_READ_FILE, so the kernel checks whether /usr/bin/cat may be read, finds no rule covering /usr, and refuses before the program ever starts.

Add /usr to the same ruleset and it works:

allowed: /etc and /usr
devils-dell

The general lesson is that a Landlock sandbox has to include everything the process touches, and that set is larger than you think. Your binary, its interpreter, every shared library it loads, and any config it reads at startup. ldd on the binary is a good place to begin the list.

Wrapping a Program You Didn't Write

Inheritance across exec is what makes Landlock useful beyond your own code. Restrict yourself, then exec whatever you want contained, and it runs inside the sandbox without cooperating or even knowing.

A usable wrapper needs one addition to the program above. Handle LANDLOCK_ACCESS_FS_EXECUTE alongside the read rights, allow the system directories any binary needs, then allow whatever working directory the user asked for:

#define RIGHTS (LANDLOCK_ACCESS_FS_READ_FILE | LANDLOCK_ACCESS_FS_READ_DIR | \
                LANDLOCK_ACCESS_FS_EXECUTE)

static int add_path(int ruleset_fd, const char *path)
{
    struct landlock_path_beneath_attr pb = { .allowed_access = RIGHTS };
    int rc;

    pb.parent_fd = open(path, O_PATH | O_CLOEXEC);
    if (pb.parent_fd < 0)
        return -1;
    rc = syscall(__NR_landlock_add_rule, ruleset_fd,
                 LANDLOCK_RULE_PATH_BENEATH, &pb, 0);
    close(pb.parent_fd);
    return rc;
}

int main(int argc, char **argv)
{
    struct landlock_ruleset_attr attr = { .handled_access_fs = RIGHTS };
    const char *base[] = { "/usr", "/lib", "/lib64", "/bin", "/etc" };
    int fd, i;

    if (argc < 3) { fprintf(stderr, "usage: %s DIR CMD...\n", argv[0]); return 2; }

    fd = syscall(__NR_landlock_create_ruleset, &attr, sizeof(attr), 0);
    if (fd < 0) { perror("create_ruleset"); return 1; }

    for (i = 0; i < (int)(sizeof(base) / sizeof(*base)); i++)
        add_path(fd, base[i]);
    if (add_path(fd, argv[1]) < 0) { perror(argv[1]); return 1; }

    if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)) { perror("prctl"); return 1; }
    if (syscall(__NR_landlock_restrict_self, fd, 0)) { perror("restrict_self"); return 1; }
    close(fd);

    execvp(argv[2], &argv[2]);
    fprintf(stderr, "%s: %s\n", argv[2], strerror(errno));
    return 1;
}

The loop ignores what add_path returns, so the same binary works on systems where /lib64 or /bin are absent or symlinked somewhere else. The directory the user named is checked, because a typo there should be an error now rather than a puzzling denial later.

Now cat and ls can see the working directory and nothing else:

mkdir -p /tmp/work && echo "project data" > /tmp/work/notes.txt

./llrun /tmp/work cat /tmp/work/notes.txt
./llrun /tmp/work cat /tmp/secret.txt
./llrun /tmp/work ls /home

project data
cat: /tmp/secret.txt: Permission denied
ls: cannot open directory '/home': Permission denied

Neither program was modified, recompiled, or asked for consent. bwrap and similar tools reach that outcome by building mount and user namespaces around the program. This gets there by asking one LSM for less, in about sixty lines, with nothing to install.

Finding the Paths a Program Needs

The hard part of any sandbox isn't the API. It's the list. Programs open far more than you expect, and a path you forget shows up as a failure somewhere deep in a run.

Two tools build the list for you. ldd gives the shared libraries, which must be readable or the program never starts:

ldd /usr/bin/cat

linux-vdso.so.1 (0x00007fff85f39000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4f61bb5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4f61e0a000)

strace gives everything else. Run the program unrestricted first and collect what it opens:

strace -e trace=openat cat /etc/hostname 2>&1 | grep -oE '"/[^"]+"' | sort -u

"/etc/hostname"
"/etc/ld.so.cache"
"/lib/x86_64-linux-gnu/libc.so.6"
"/usr/lib/locale/locale-archive"

Four paths for a program that prints one file, and only one of them is the file you asked for. The linker cache, the C library, and the locale archive are all mandatory, which is why the wrapper allows /usr, /lib and /etc before it allows anything you chose.

Once restricted, strace also tells you exactly what a denial was:

strace -f -e trace=openat ./llrun /tmp/work cat /tmp/secret.txt 2>&1 | grep EACCES

openat(AT_FDCWD, "/tmp/secret.txt", O_RDONLY) = -1 EACCES (Permission denied)

That's the loop. Run it, read the EACCES line, decide whether the path belongs in the ruleset or the program shouldn't be reaching for it, then repeat. Landlock logs nothing of its own on this kernel, so strace is the debugger. Kernels from 6.15 report denials through the audit subsystem, so check your version before hunting for a log that isn't there.

Which ABI Version You Have

Landlock has grown since 5.13, and features you read about may not exist on your kernel. Ask it directly:

int v = syscall(__NR_landlock_create_ruleset, NULL, 0,
                LANDLOCK_CREATE_RULESET_VERSION);

This machine reports 1, which is the original from 5.13 and offers thirteen filesystem access rights:

grep -oE "LANDLOCK_ACCESS_FS_[A-Z_]+" /usr/include/linux/landlock.h | sort -u

LANDLOCK_ACCESS_FS_EXECUTE      LANDLOCK_ACCESS_FS_MAKE_BLOCK
LANDLOCK_ACCESS_FS_MAKE_CHAR    LANDLOCK_ACCESS_FS_MAKE_DIR
LANDLOCK_ACCESS_FS_MAKE_FIFO    LANDLOCK_ACCESS_FS_MAKE_REG
LANDLOCK_ACCESS_FS_MAKE_SOCK    LANDLOCK_ACCESS_FS_MAKE_SYM
LANDLOCK_ACCESS_FS_READ_DIR     LANDLOCK_ACCESS_FS_READ_FILE
LANDLOCK_ACCESS_FS_REMOVE_DIR   LANDLOCK_ACCESS_FS_REMOVE_FILE
LANDLOCK_ACCESS_FS_WRITE_FILE

Later versions added file reparenting, truncation, network rules covering TCP bind and connect, and control over device ioctls. Each arrived in its own ABI bump, so a program that wants a newer right should query the version and degrade rather than assume. Passing a right the running kernel doesn't know about makes landlock_create_ruleset fail with EINVAL, which is a confusing error to debug if you haven't checked the version first.

Conclusion

You can now sandbox a process from inside itself, with no privileges and no configuration, and you know the four surprises. no_new_privs comes first or nothing applies. Rulesets intersect rather than accumulate, so build one ruleset with everything in it. Children inherit, which is a feature. And read restrictions break exec unless the binary's path is allowed too.

There are a few directions to go from here. Add LANDLOCK_ACCESS_FS_WRITE_FILE to handled_access_fs and make a program that can read widely but write to exactly one directory. Wrap a program you didn't write by restricting yourself and then calling execve. Or look at how strace reports the denials, which is the fastest way to build the list of paths a real program actually needs.

Epilogue

The interesting question about Landlock isn't what it does. It's what it can't express. A ruleset names paths, so the unit of authority is a location in the filesystem rather than a particular file you were handed. You can say this process may read below /etc. You can't say this process may read the one file the user just picked, and nothing else.

That gap is what I've spent the last while on, building a capability-backed desktop OS in which authority arrives as a handle to one object rather than a rule about a location, with the Debian ecosystem still working underneath. Landlock does a great deal of the work, and the places it stops are where the design gets interesting.

I write about systems and their mysteries at thechris.in.

systemd-analyze

On the laptop I'm writing this on, it says:

Startup finished in 5.855s (firmware) + 8.469s (loader) + 3.106s (kernel)
+ 12.181s (userspace) = 29.613s
graphical.target reached after 12.175s in userspace

Add those four and you get 29.611, not the 29.613 printed on the last line. That gap is real and it isn't a mistake: systemd-analyze truncates each phase for display while summing the underlying microseconds, so a couple of milliseconds hide in the rounding. It's a small thing, and it will save you an evening of hunting for a bug that isn't there.

Four numbers, and most people's mental model accounts for one of them. The kernel took three seconds. The bootloader before it took eight and a half, and the firmware before that took nearly six. Fourteen seconds of this machine's boot happened before Linux was running at all, and I chose a good part of it without noticing.

This article follows the Linux boot process the whole way through one real boot, from the firmware handing control to a bootloader to a login prompt on your screen. Everything here you can run yourself, and almost none of it needs root.

Table of Contents

• What you need

• The Linux boot process is four handoffs, not one

• Firmware, and the part Linux never sees

• The bootloader, and the five seconds you chose

• The kernel phase: three seconds to a working machine

• What the initramfs is, and the chicken-and-egg problem it solves

• PID 1, and where the other twelve seconds go

• Why systemd-analyze blame misleads you about boot time

• Reading the critical chain

• Why your Linux boot time will be different

• The login screen, and the handoff to you

• Conclusion

• Epilogue

What You Need

Any Linux machine running systemd, which covers Ubuntu, Debian, Fedora, Arch, and most things people install in 2026. A terminal. That's it.

The machine I'm measuring is Ubuntu 22.04.5 LTS running kernel 5.15.0-190-generic, booting in UEFI mode from an NVMe disk with an ext4 root filesystem, with systemd 249 as PID 1. Yours will differ, sometimes by a lot, and I'll say where to expect that.

One command needs root and one file is restricted on many systems. I'll flag both when we get there.

The Linux Boot Process is Four Handoffs, Not One

The word "boot" suggests a single process. But it's four, and they barely know about each other.

Firmware runs first, from a chip on the motherboard, and its job is to find something bootable and start it.

It then hands control to a bootloader and stops. The bootloader's job is to find a kernel, load it into memory along with an initial filesystem, and jump to it. It then stops.

The kernel brings up hardware, mounts a root filesystem, and starts exactly one userspace process. Then it stops being in charge, though it keeps servicing that process forever afterward.

That first process, PID 1, starts everything else.

Each handoff is one way. The firmware isn't sitting underneath Linux waiting to help. The bootloader is gone from memory. Hold onto that, because it explains why the four numbers in systemd-analyze are measured by different things and mean different things.

Firmware, and the Part Linux Never Sees

The 5.855 seconds attributed to firmware is the only figure here that Linux didn't measure itself, and you can read the exact source it came from:

cat /sys/firmware/acpi/fpdt/boot/bootloader_launch_ns

5855973785

That's 5.855973785 seconds, which is the 5.855 systemd-analyze reported, truncated for display. The firmware wrote it into an ACPI table called the Firmware Performance Data Table before handing over, and the kernel exposes the table's fields under that directory. Its sibling fills in the rest:

cat /sys/firmware/acpi/fpdt/boot/exitbootservice_end_ns

On this machine that reads 14325507185, or 14.325 seconds, which is firmware and bootloader combined and matches the two phases added together.

Here's the part that trips people up. The gate isn't UEFI, it's whether your firmware publishes an FPDT at all. Plenty of UEFI machines don't, virtual machines especially, and on those systemd-analyze reports no firmware phase and no loader phase either, starting its accounting at the kernel. Check for the table rather than for UEFI:

ls /sys/firmware/acpi/fpdt/boot/ 2>/dev/null || echo "no FPDT, so no firmware timing"

If that prints nothing, your firmware isn't reporting, and there's no way to recover the number from inside Linux.

Almost six seconds is a lot, and there isn't much you can do about it from inside Linux. It's memory training and device enumeration, plus whatever your vendor decided to run before handing over. On a laptop with fast storage this is often the single largest phase, which surprises people who spend their optimization effort on services.

The Bootloader, and the Five Seconds You Chose

Here's the number that changes how you read the whole output. The loader phase took 8.469 seconds, nearly three times what the kernel took. Almost none of that was work.

grep ^GRUB_TIMEOUT /etc/default/grub

On this machine:

GRUB_TIMEOUT="5"

Five of those 8.469 seconds are GRUB counting down and waiting for a keypress that never comes. That's a configuration choice, made once, probably by the installer, and then never revisited. If you've ever wondered why your machine feels slow to start despite good hardware, this is the first place to look, and it's the cheapest fix in this entire article.

While GRUB is waiting, it already knows what it's going to do. You can read the instructions it passed along:

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-5.15.0-190-generic root=UUID=b4d0343e-9df4-40a7-be97-dcd51bdbf553
ro splash intel_iommu=on vt.handoff=7

That single line is the entire contract between the bootloader and the kernel. BOOT_IMAGE is which kernel got loaded. root=UUID=... names the filesystem to mount, by UUID rather than device name so it survives disks being renumbered. ro says mount it read-only at first. splash asks for a graphical boot screen instead of scrolling text. intel_iommu=on enables the IOMMU, and vt.handoff=7 is Ubuntu passing the virtual terminal to the graphical stack without a flicker.

GRUB loads two files into memory: the kernel, and an initial filesystem image we'll come back to shortly. Then it jumps into the kernel and ceases to exist.

The Kernel Phase: Three Seconds to a Working Machine

Now Linux is running. The kernel decompresses itself, sets up memory management, brings up the CPUs, initializes drivers, and looks for a root filesystem.

Watch it happen, with timestamps measured from the moment the kernel started:

journalctl -k -b -o short-monotonic | head

[    0.000000] devils-dell kernel: microcode: microcode updated early to revision 0x100
[    0.000000] devils-dell kernel: Linux version 5.15.0-190-generic
[    0.000000] devils-dell kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-190-generic
[    0.000000] devils-dell kernel: KERNEL supported cpus:

Zero is when the kernel began executing, not when you pressed the power button. Everything before this point, all fourteen seconds of firmware and bootloader, is outside this clock entirely. That's the first thing to understand about kernel boot timestamps, and it's why dmesg output makes some machines look far faster than they are.

You may have reached for dmesg there and been refused:

dmesg: read kernel buffer failed: Operation not permitted

That's deliberate, and you can confirm it:

sysctl kernel.dmesg_restrict

Ubuntu sets kernel.dmesg_restrict = 1, which limits the kernel ring buffer to root because it leaks kernel addresses useful to an attacker. Use journalctl -k instead, which reads the same messages through the journal and works as an ordinary user.

One more detail from /proc/cmdline explains something people notice and rarely investigate. The kernel was told to mount the root filesystem ro, read-only, yet the system you're using now clearly writes to disk. Check what it looks like today:

findmnt -n -o SOURCE,FSTYPE,OPTIONS /

/dev/nvme0n1p2 ext4 rw,relatime,errors=remount-ro

It's read-write now, so something changed it. The root filesystem is mounted read-only first so that a filesystem check can run safely against it, since checking a filesystem while processes are writing to it is how you turn a small problem into a large one.

Once that check passes, userspace remounts the same filesystem read-write in place, and the errors=remount-ro option you can see there is the reverse promise: if the kernel hits a filesystem error later, it drops back to read-only instead of writing to something it no longer trusts.

Now find the exact moment the kernel stopped being alone:

journalctl -b -o short-monotonic | grep -m1 'systemd\[1\]'

[    3.152109] devils-dell systemd[1]: Inserted module 'autofs4'

3.15 seconds in, PID 1 logged its first line. That matches the 3.106 seconds systemd-analyze attributes to the kernel, and it's the handoff. From here the kernel does nothing on its own initiative. It answers system calls, and every decision about what runs next belongs to userspace.

What the initramfs is, and the Chicken-and-Egg Problem it Solves

There's a step hidden inside that three seconds, and it's the part of Linux boot that confuses people most.

The kernel needs to mount your root filesystem. To do that, it needs a driver for your storage controller and a driver for the filesystem, and possibly code to assemble a RAID array, open an encrypted volume, or activate LVM. Those drivers live in modules. The modules live on the root filesystem. Which the kernel can't mount yet.

The way out is a small filesystem the bootloader loads into memory alongside the kernel, complete in itself. Look at it:

ls -l /boot/initrd.img-$(uname -r)
lsinitramfs /boot/initrd.img-$(uname -r) | wc -l
lsinitramfs /boot/initrd.img-$(uname -r) | grep -c '\.ko'

Those paths and that tool are Debian and Ubuntu conventions. On Fedora the image is /boot/initramfs-$(uname -r).img and the tool is lsinitrd. On Arch it's usually /boot/initramfs-linux.img, read with lsinitcpio.

On this machine the image is 79,945,887 bytes, roughly 76 MB, holding 2,318 files of which 1,386 are kernel modules. It's a real, working, if minimal Linux system that exists to solve one problem: find and mount the actual root filesystem.

Once it succeeds, it does something unusual. It never exits. It swaps the real root into place, moves itself out of the way, and executes the real /sbin/init without ever starting a new process tree. PID 1 changes identity mid-flight and keeps its process ID.

If you use Arch or a minimal Fedora install, your initramfs may be a tenth of this size, because Ubuntu builds a generic one containing drivers for hardware you don't own so that the same image boots on any machine. That's a deliberate trade of size against portability, and lsinitramfs will show you what you're carrying.

PID 1, and Where the Other Twelve Seconds Go

The kernel finished at 3.1 seconds. The login screen appeared at 12.175 seconds of userspace. So what happened in between?

systemctl get-default
systemctl list-unit-files --no-legend | wc -l
systemctl list-units --type=service --state=running --no-legend | wc -l

graphical.target
472
44

systemd's model is that you name a goal and it works out the order. The goal here is graphical.target, which wants multi-user.target, which wants a working network, filesystems, logging, and dozens of other things.

There are 472 unit files installed on this machine and 44 services actually running. systemd builds a dependency graph from those and starts everything it can in parallel, waiting only where a real dependency exists.

That parallelism is why boot analysis is harder than it looks. Dozens of things are happening at once, and the total isn't the sum of the parts.

Why systemd-analyze blame Misleads You About Boot Time

The obvious next command is the wrong one, and the trap is worth walking into deliberately:

systemd-analyze blame | head -5

3min 48.947s fstrim.service
     44.548s plocate-updatedb.service
     13.953s apt-daily.service
      4.875s docker.service
      4.106s NetworkManager-wait-online.service

Read that against the total. Userspace took 12.181 seconds. The top entry claims three minutes and forty-nine. Both numbers are correct, and the contradiction is the whole point.

blame lists how long each unit took to start, for every unit systemd has started, whenever it started. It says nothing about whether the unit was on the path to your login screen. Check the top three:

systemctl show fstrim.service -p TriggeredBy -p WantedBy
systemctl list-timers fstrim.timer

TriggeredBy=fstrim.timer
WantedBy=
NEXT                        LEFT        LAST                        PASSED
Mon 2026-09-14 00:59:36 IST 4 days left Mon 2026-09-07 00:33:35 IST 2 days ago

WantedBy is empty, so nothing pulls it in at boot. It's triggered by a timer. It last ran two days ago and runs again in four.

plocate-updatedb.service and apt-daily.service are the same shape, triggered by their own timers.

The top three entries in blame, nearly five minutes of apparent boot time, contributed exactly nothing to how long you waited for a login prompt.

The first entry that's genuinely on the boot path is docker.service, fourth in the list, at 4.875 seconds.

I'd rather you take the general lesson than the specific one. A measurement that reports on a superset of what you care about will mislead you in proportion to how much of that superset is irrelevant. blame isn't broken. It answers a different question than the one people ask it.

Reading the Critical Chain

The command that answers the actual question is this one:

systemd-analyze critical-chain

graphical.target @12.175s
└─multi-user.target @12.175s
  └─docker.service @7.297s +4.875s
    └─network-online.target @7.295s
      └─NetworkManager-wait-online.service @3.188s +4.106s
        └─NetworkManager.service @3.151s +35ms
          └─network-pre.target @3.150s
            └─netfilter-persistent.service @1.377s +1.772s
              └─local-fs.target @1.374s

The @ is when a unit became active. The + is how long it took. Now the twelve seconds make sense: docker.service at 4.875 and NetworkManager-wait-online.service at 4.106 account for nearly nine of them, and they're serialized because Docker wants a working network before it starts.

NetworkManager-wait-online is the one to look at first on most desktops. It does what its name says, which is block until the network is actually up, and on a laptop associating with Wi-Fi that can be seconds of doing nothing. It exists so that services needing a network don't start before there is one. Whether you need that guarantee is a real question with a real answer, and it depends on what you run.

Two cautions about this output. It shows one chain, not every slow thing, so a unit that was slow but off the critical path never appears. And the @ times aren't a causal sequence you can read top to bottom. My own output has a Docker network mount timestamped at 11 seconds nested underneath a target that completed at 650 milliseconds, which looks impossible until you realize the tree shows dependency edges and not a sequence of events. Read the + values for cost and the structure for ordering constraints, and don't read the nesting as a chronology.

Why Your Linux Boot Time Will Be Different

Everything above is one boot on one machine, and the specific figures are worth less than the method. Before you compare yours to mine, know which parts move and why.

Firmware time varies more than anything else here, and it has almost nothing to do with Linux. A desktop with lots of RAM to train and a dozen USB devices to enumerate can spend fifteen seconds where this laptop spends six. If your firmware has a fast boot option, that option is what it sounds like: skipping enumeration steps, at the cost of not noticing hardware you plugged in.

Loader time is mostly your timeout, so it's mostly your decision. Kernel time moves with how much hardware you have and how much of the initramfs has to be unpacked and searched. This is why a distribution-generic initramfs like Ubuntu's costs more here than a host-specific one built for your machine alone. If your root filesystem is encrypted, some of what looks like kernel time is actually you typing a passphrase.

Userspace time is where your machine differs from mine most, because it reflects what you installed rather than what you own. Docker costs me nearly five seconds and would cost you nothing if you don't run it.

Run it a few times before drawing conclusions. Boot timing varies between runs on the same machine, and a single reading tells you less than you'd like.

The Login Screen, and the Handoff to You

The last step is the one you see:

systemctl status display-manager --no-pager | head -1
loginctl show-session $(loginctl | awk 'NR==2{print $1}') -p Type -p Class

● lightdm.service - Light Display Manager
Type=x11
Class=user

On this machine, the display manager is LightDM, started as part of graphical.target. It opens a session on a virtual terminal, draws the greeter, and waits.

Behind it, systemd has already created a seat and a session slot for whoever logs in. When you type your password, the display manager authenticates through PAM, systemd assigns the session, and your desktop environment starts as a user process.

That vt.handoff=7 from the kernel command line pays off here. It hands the virtual terminal to the graphical stack without the screen blanking and redrawing, which is the difference between a smooth boot and a flickering one.

From this point, the kernel is doing what it always does, which is answering system calls. If you want to follow what happens next, I wrote about that boundary in detail.

Conclusion

You can now account for your own boot, phase by phase, with numbers instead of guesses. On this laptop, 29.6 seconds breaks down as almost six seconds of firmware you can't control, eight and a half seconds of bootloader that's mostly a countdown you can delete, three seconds of kernel, and twelve seconds of userspace dominated by two services waiting on the network.

More usefully, you have a way to tell a real measurement from a plausible one. systemd-analyze blame looks authoritative and answers a question nobody asked. The critical chain answers the right question and still needs care, because its tree maps dependencies, not chronology.

A few directions from here. Set GRUB_TIMEOUT=1 and regenerate the config, with sudo update-grub on Debian and Ubuntu or sudo grub2-mkconfig -o /boot/grub2/grub.cfg on Fedora, then reboot and watch five seconds vanish. Run systemd-analyze plot > boot.svg and open it in a browser for the parallel view the text output flattens. Or look at whether NetworkManager-wait-online.service is earning its four seconds on your machine, which for most desktops it isn't.

Epilogue

The reason I went looking at any of this is that I'm building a Linux distribution with an Android-style permission model, where a program gets only the authority its manifest asks for rather than everything its user happens to have. That turns the boot sequence into a security question. Every process started before you log in runs with more authority than anything you launch afterward, and until I could name each one and say why it was there, I had no real way to argue about which of them deserved it.

You can find more of what I write at thechris.in.

#include <stdio.h>
#include <time.h>
#include <unistd.h>

int main(void)
{
    struct timespec ts;

    for (int i = 0; i < 3; i++)
        clock_gettime(CLOCK_MONOTONIC, &ts);

    write(1, "done\n", 5);
    return 0;
}

Both of those look like system calls. Both of them ask the kernel for something your program can't get on its own: the current time, and access to a file descriptor.

Now run it under strace, which reports every system call a process makes:

gcc -O0 -o mystery mystery.c
strace ./mystery 2>&1 | grep -c clock_gettime

The answer is 0.

The write() shows up immediately. The three clock_gettime() calls don't appear at all. Same program, same libc, same machine, and one of them never reaches the kernel.

By the end of this article you'll know every step between your write() and the code that runs inside the kernel, why the return trip is stranger than the way in, and why clock_gettime() gets to skip the whole thing.

Table of Contents

• What You Need

• What a System Call Looks Like from Userspace

• The Crossing

• Inside the Kernel: Finding the Handler

• The Return Trip, and the Truth About errno

• The System Call That Never Happens

• What the Boundary Costs

• Conclusion

• Epilogue

What You Need

You need an x86-64 machine running Linux, gcc, strace, and objdump. On Debian or Ubuntu that's build-essential, strace, and binutils. You also need to be comfortable reading C. You don't need to have written kernel code, and you won't build or install a kernel here.

Everything below runs on a normal user account, except for one optional tracing step that needs sudo.

Two warnings about scope. First, this article is about x86-64 only. ARM64 does the same job with different instructions and different register rules, and hedging every sentence for both would double the length and halve the clarity. Second, kernel internals move. I ran everything here on Linux 5.15 (Ubuntu 22.04, Intel Core i7-10750H), and I'll flag the places where newer kernels differ. Check your own version with uname -r.

What a System Call Looks Like from Userspace

Let's start with a correction that matters: write() isn't a system call. It's an ordinary C function in your C library. That function makes a system call on your behalf, and the difference between those two things is where most confusion about the kernel begins.

You can prove it by cutting libc out and making the call yourself.

On x86-64, a system call has a fixed convention. You put the number of the call you want in rax, and its arguments in rdi, rsi, rdx, r10, r8, and r9, in that order. Then you execute a single instruction called syscall.

The numbers aren't something you memorise. They live in a header on your machine:

grep -E "__NR_(write|getpid|clock_gettime) " /usr/include/x86_64-linux-gnu/asm/unistd_64.h

On this machine:

#define __NR_write 1
#define __NR_getpid 39
#define __NR_clock_gettime 228

So write is call number 1. Here's that call written by hand, with no libc wrapper involved:

static long raw_write(int fd, const void *buf, unsigned long count)
{
    long ret;

    __asm__ volatile (
        "syscall"
        : "=a" (ret)                   /* the result comes back in rax */
        : "a" (1L),        /* rax = 1, the syscall number for write */
          "D" ((long)fd),  /* rdi = first argument                  */
          "S" (buf),       /* rsi = second argument                 */
          "d" (count)      /* rdx = third argument                  */
        : "rcx", "r11", "memory"
    );

    return ret;
}

Compile and run it and your bytes turn up on standard output, with nothing from libc anywhere in the path.

Look at that last line, the clobber list. It tells the compiler rcx and r11 are going to be destroyed. I didn't add that for safety. It's a fact about the hardware, and it quietly explains something odd about the convention above.

C functions on x86-64 pass their fourth argument in rcx. System calls pass theirs in r10 instead. Every explanation that says "because that's the convention" stops one step too early. The real reason is that the syscall instruction overwrites rcx as part of doing its job. The kernel couldn't receive a fourth argument there even if it wanted to, so the ABI routed around its own hardware.

This is the first sign that this boundary isn't a function call wearing a costume. Different mechanism, different rules, and the hardware got there first.

You can see the instruction itself in your compiled binary:

objdump -d --no-show-raw-insn raw_write | grep -B2 -A2 syscall

The instruction is right there:

    118b:	mov    -0x28(%rbp),%rdx
    118f:	syscall
    1191:	mov    %rax,-0x8(%rbp)

Three lines: load a register, execute one instruction, and store what came back. Everything else in this article happens between line two and line three.

The Crossing

When the CPU executes syscall, it does something no ordinary jump can do: it changes the privilege level of the processor. Your code runs in what x86 calls ring 3. Kernel code runs in ring 0.

The instruction does three things, in order:

1. It saves the address of the next instruction in your program into rcx. That's the return address, and it's why rcx gets clobbered.

2. It saves the CPU flags into r11.

3. It loads a new instruction pointer, code segment, and stack segment from three special CPU registers.

That third step is the important one. The new instruction pointer doesn't come from your program. It comes from a machine-specific register called LSTAR, and the kernel wrote that register during boot.

That's the security property the whole design rests on. Userspace triggers the transition. Userspace doesn't get to pick where it lands. There's one door, the kernel installed it, and it opens on entry_SYSCALL_64 in arch/x86/entry/entry_64.S.

Notice what the instruction does not do. It doesn't consult the interrupt descriptor table or push an exception frame. Older systems reached the kernel through int 0x80, a software interrupt with all of that machinery attached, and it was slow. The syscall instruction exists because this path was worth building dedicated hardware for.

Becoming the Kernel

Arriving at entry_SYSCALL_64 isn't the same as being ready to run kernel code. At the instant of arrival the CPU is in ring 0, but it's still using your stack and your register state. The kernel has to fix that before it can safely do anything.

Three things happen, and all three are the kernel establishing trust in a machine it is already running on:

1. swapgs

The kernel keeps a per-CPU pointer in the GS register so it can find its own data structures. While you were running, GS held whatever your program put there. A single instruction, swapgs, exchanges it for the kernel's value. The kernel documentation is unusually blunt about this one, calling it fragile and warning that it must nest perfectly. Get it wrong in either direction and you have a very bad afternoon ahead of you.

2. The stack switch

Your stack pointer is a value your program chose, so the kernel can't trust it. It stashes your rsp and switches to a kernel stack it allocated for this thread.

3. Building pt_regs

The kernel then pushes your saved registers onto that new stack in a specific order, forming a C struct called struct pt_regs. pt_regs is your process, frozen. Every debugger that inspects a stopped process, every signal handler that modifies the context it returns to, and every system call handler reads its arguments out of that struct.

There may be a fourth step. If your CPU is vulnerable to Meltdown, the kernel also swaps page tables here, because on those chips the kernel's memory can't safely stay mapped while your code runs.

That swap isn't free. It's why system calls got measurably slower in 2018, and why some of the numbers later in this article would look different on a machine three years older.

You can check whether your machine pays that cost:

cat /sys/devices/system/cpu/vulnerabilities/meltdown

The test machine here reports Not affected, because its generation of silicon has the fix in hardware. An older laptop will report Mitigation: PTI, and every system call it makes is doing extra work at exactly this point.

Look through the other files in that directory while you're there. Each one is a mitigation that this boundary may be paying for.

Inside the Kernel: Finding the Handler

The kernel is now running on its own stack with your registers safely captured. It calls a C function, do_syscall_64, and hands it two things: your pt_regs, and the system call number you left in rax.

Dispatch is short enough to describe completely. The kernel checks that your number is within range, clamps it, and jumps to the matching handler:

if (likely(nr < NR_syscalls)) {
    nr = array_index_nospec(nr, NR_syscalls);
    regs->ax = x64_sys_call(regs, nr);
}

Two things in there need explaining.

array_index_nospec is a Spectre mitigation. A plain bounds check isn't enough on a speculating CPU, because the processor may run ahead and touch memory past the end of the table before the check resolves. This helper forces the index to be clamped in a way speculation can't skip.

x64_sys_call is where a lot of older explanations are now wrong, including some still near the top of search results. They'll tell you the kernel indexes an array of function pointers called sys_call_table. That was true for many years and is no longer how dispatch works. Since kernel 6.9, x64_sys_call is a generated switch statement of direct calls.

The reason is a chain of consequences. Spectre mitigations made indirect calls through function pointers expensive, because each one has to go through a retpoline.

A switch of direct calls avoids that cost entirely. The table still exists, because tracing tools use it, but the hot path no longer reads it. On my 5.15 kernel the older table-based dispatch is still in place, which is exactly why naming your kernel version in an article like this one matters.

Where the Handler Comes From

The handler for write is named __x64_sys_write, and you won't find that name written anywhere in the kernel source. It's generated by a macro:

SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf, size_t, count)

SYSCALL_DEFINE3 means "a system call taking three arguments". The macro expands into two functions: the real implementation, and a thin wrapper named __x64_sys_write that takes a single struct pt_regs * and pulls the arguments out of it.

That indirection is deliberate. Rather than trusting whatever userspace happened to leave in the argument registers, the kernel unpacks exactly the values it expects from the frozen struct it built itself. It's the same defensive instinct as array_index_nospec, applied to the shape of the function call.

You don't have to take any of this on faith. ftrace, the kernel's built-in tracer, will show you the handler running.

This needs a root shell rather than sudo on each line, because the filter that keeps the output readable refers to the shell's own process ID:

sudo -i
cd /sys/kernel/tracing

echo 0 > tracing_on
echo $$ > set_ftrace_pid              # trace only this shell
echo function_graph > current_tracer
echo __x64_sys_write > set_graph_function

echo 1 > tracing_on
echo "trigger a write" > /dev/null    # the call we want to catch
echo 0 > tracing_on

head -40 trace

Without that set_ftrace_pid line, you'll trace every write on the machine, which on a running desktop is far too much output to read.

Here's the result on the test system, lightly trimmed:

 9)               |  __x64_sys_write() {
 9)               |    ksys_write() {
 9)               |      __fdget_pos() {
 9)   0.124 us    |        __fget_light();
 9)   0.363 us    |      }
 9)               |      vfs_write() {
 9)               |        rw_verify_area() {
 9)               |          security_file_permission() {
 9)               |            apparmor_file_permission() {
 9)   0.264 us    |              aa_file_perm();
 9)   0.457 us    |            }
 9)   0.644 us    |          }
 9)   0.857 us    |        }
 9)   0.083 us    |        write_null();
 9)               |        __fsnotify_parent() {
 9)   0.107 us    |          fsnotify();
 9)   1.383 us    |        }
 9)   2.813 us    |      }
 9)   3.449 us    |    }
 9)   3.720 us    |  }

Read that from the outside in and you have the whole descent in twenty lines.

__x64_sys_write is the generated wrapper. It calls ksys_write, the real implementation. That looks up your file descriptor with __fdget_pos, then hands off to vfs_write, the virtual filesystem layer, which is where the kernel stops caring what kind of thing you're writing to.

Then security_file_permission calls into AppArmor, because this machine runs Ubuntu. On a SELinux system something else sits there. Either way it's a security module deciding whether you're allowed to do this. On every write. Every one.

write_null is the payoff, and it's there by accident: the command above wrote to /dev/null, so that's the actual driver, the one whose whole job is throwing your bytes away. Point the same write at a file on disk and a filesystem function shows up in that slot instead. Nothing above it moves.

The whole thing took 3.7 microseconds, and the timings on the right tell you where it went.

When you're done, put the tracer back:

echo nop > current_tracer
echo > set_graph_function
echo > set_ftrace_pid

If /sys/kernel/tracing doesn't exist on your system, try /sys/kernel/debug/tracing instead.

The Return Trip, and the Truth About errno

The handler finishes and returns a number. That number goes into rax, and rax is the only thing your program gets back.

Which raises a question that is rarely asked directly: if the kernel can only return one value, how does it report what went wrong as well as that something went wrong?

The answer is that it doesn't have a separate channel. The kernel returns errors as small negative numbers in the same register as the result. A successful write of 24 bytes returns 24. A write to a closed descriptor returns -9, because EBADF is error number 9.

Now put that together with the fact that errno exists, and something doesn't add up. errno is a variable in your process. The kernel doesn't write to your variables.

Here's the test. Set errno to zero, make a raw system call that's guaranteed to fail, and look at both values:

#include <stdio.h>         /* fprintf, stderr */
#include <errno.h>         /* errno */

/* raw_write() is the function from the previous section */

errno = 0;

long ok  = raw_write(1, "written via raw syscall\n", 24);
long bad = raw_write(999, "x", 1);          /* not an open descriptor */

fprintf(stderr, "ok = %ld\n",  ok);
fprintf(stderr, "bad = %ld\n", bad);
fprintf(stderr, "errno = %d\n", errno);

Running it:

written via raw syscall
ok = 24
bad = -9
errno = 0

There it is. The kernel returned -9, and errno never moved.

errno is a libc invention. When you call the normal write(), the wrapper checks whether the return value is a small negative number. If it is, it negates it, stores the result in errno, and returns -1 to you. The -1-and-check-errno pattern every C programmer learns is a convention built entirely in userspace, on top of a kernel interface that works a completely different way.

Once you've seen this, a familiar bug class makes more sense. errno is only meaningful immediately after a failed call, because it's just a variable that the last wrapper to fail happened to write to.

Two Ways Out

Getting back to userspace has a fast path and a slow path.

The fast path is sysret, the mirror of syscall: it restores your instruction pointer from rcx and your flags from r11 and drops back to ring 3 in a few cycles.

The slow path is iret, the general-purpose return-from-interrupt instruction. It's significantly slower, and the kernel uses it when sysret can't be trusted. The entry code's own comments explain why: sysret has trouble with non-canonical addresses due to bugs in both AMD and Intel CPUs, so whenever something might have changed your saved state, the kernel forces the safe path. A debugger reaching in through ptrace and changing your registers is the usual culprit.

Before either instruction runs, the kernel does the housekeeping it deferred. It checks for pending signals and delivers them. It checks whether the scheduler wants the CPU back, and if so, your process stops here and something else runs.

Which means a system call isn't only a request for service. It's one of the main places your process can simply stop running. You asked to write five bytes. On the way back the kernel gets to reconsider everything about you, including whether you should continue at all.

The System Call That Never Happens

Now back to the mystery from the opening.

Look at your own process's memory map:

cat /proc/self/maps | tail -4

which ends with:

7fff32f97000-7fff32f9b000 r--p  [vvar]
7fff32f9b000-7fff32f9d000 r-xp  [vdso]

Two regions you never asked for. Neither came from your program or your libraries. The kernel put them there, in every process on the system.

[vdso] stands for virtual dynamic shared object. It's a small, complete shared library (real ELF, with a symbol table) that the kernel maps into every address space. And because the kernel tells each process where it put it, you can dump your own copy and take it apart:

#include <stdio.h>
#include <sys/auxv.h>      /* getauxval, AT_SYSINFO_EHDR */
#include <unistd.h>        /* getpagesize */

int main(void)
{
    void  *vdso = (void *)getauxval(AT_SYSINFO_EHDR);   /* the kernel tells us where */
    size_t len  = 2 * getpagesize();                    /* the mapping is two pages  */

    FILE *f = fopen("vdso.so", "wb");
    fwrite(vdso, 1, len, f);
    fclose(f);

    printf("vDSO was mapped at %p\n", vdso);
    return 0;
}

AT_SYSINFO_EHDR lives in <sys/auxv.h>. Leave that header out and you don't get a polite warning about it: the build stops with AT_SYSINFO_EHDR undeclared.

Run that, then read its symbol table like any other library:

./dump_vdso && objdump -T vdso.so | grep __vdso

and out comes:

__vdso_gettimeofday
__vdso_clock_gettime
__vdso_clock_getres
__vdso_time
__vdso_getcpu

There's the answer. clock_gettime is in that list.

When you call clock_gettime(), libc calls into the vDSO. That code is written by kernel developers and shipped with the kernel, but it executes in ring 3, as part of your process. It reads the current time out of the [vvar] page (a read-only page the kernel keeps updated) and returns.

There's no privilege change, syscall instruction, entry point, stack switch, or pt_regs. And that means there's nothing for strace to see, because strace works by watching the boundary, and this call never goes near it.

That's the whole trick. The kernel took a handful of operations that are called constantly, need no privileges to read, and only ever return information the kernel is willing to publish – and it published them.

That last constraint explains why the list is so short. write() can never work this way, because it has to change state that belongs to the kernel. Reading the clock does not. So the clock, the time of day, and the current CPU number moved out to where the caller already is.

What the Boundary Costs

Everything above is mechanism. Here's the price, measured.

The benchmark compares a call that definitely traps against one that definitely does not. For the first, use syscall(SYS_getpid). Going through the thin syscall() wrapper guarantees a real crossing:

/* Excerpt. Needs <unistd.h>, <sys/syscall.h> and <time.h>, plus a now()
   helper returning seconds as a double, and ITERATIONS defined above. */

double a = now();
for (long i = 0; i < ITERATIONS; i++)
    sink += syscall(SYS_getpid);

double b = now();
for (long i = 0; i < ITERATIONS; i++)
    clock_gettime(CLOCK_MONOTONIC, &ts);

On the test machine, two million iterations of each:

real system call (getpid):   106.9 ns/call
vDSO call (clock_gettime):    17.2 ns/call
ratio:                          6.2x

Roughly six times, and getpid is about as cheap as a system call gets. It reads one field and returns. Which means almost none of that 107 nanoseconds is the work. It's the privilege change, swapgs, the stack switch, pt_regs going up and coming back down, plus whatever mitigations your particular CPU insists on along the way.

Now the caveat, because it matters more than the number.

That 107 nanoseconds is close to a best case. Check what this machine reported earlier: Not affected for Meltdown, so it never does the page-table swap. Its Spectre mitigation is Enhanced IBRS, which is handled in silicon rather than by retpolines in software. This CPU is skipping two of the most expensive things a crossing can involve.

So run the benchmark yourself, and read your own mitigation files alongside it:

grep . /sys/devices/system/cpu/vulnerabilities/*

If yours says Mitigation: PTI, your crossings are doing strictly more work than the ones measured here, and your number should be higher. Older silicon can be dramatically worse.

Treat the ratio as the durable result and the absolute number as one reading from one machine. The figure moves with your CPU, your kernel, and whichever mitigations you happen to be carrying. Run it a few times while you're there. The spread between runs on this laptop was about fifteen percent, which tells you roughly how much to trust any single number, including mine.

One more result from the same run corrects a widely repeated claim. getpid() through normal libc costs the same as the raw syscall(SYS_getpid). glibc used to cache the process ID to avoid the trip, and stopped years ago, because keeping the cache correct across fork and namespace changes was worse than paying the hundred nanoseconds.

Six times sounds abstract until you attach it to something. A program making a million small read() calls spends about a tenth of a second on nothing but crossings. This is the pressure behind a lot of modern kernel interface design: io_uring exists so that submitting a thousand operations can cost one crossing instead of a thousand. Batching syscalls, buffering writes, and using sendfile() instead of a read-write loop are all the same optimisation: not doing less work, just crossing the boundary fewer times.

Conclusion

You can now follow a system call the whole way. You've seen the syscall instruction in your own binary, watched a bad file descriptor come back as -9 while errno stayed at zero, pulled the vDSO out of your own address space and read its symbol table, and measured what the crossing costs on your own CPU.

More usefully, you have a mental model that keeps paying out. When you read that io_uring reduces syscall overhead, you know exactly what overhead means. When a profile shows time in entry_SYSCALL_64, you know what that function does. When strace shows nothing, you know to check the vDSO before doubting the tool.

There are a few directions to go from here. Run the ftrace recipe and follow __x64_sys_write down into the filesystem layer. Read arch/x86/entry/entry_64.S: it's heavily commented and much more approachable than its reputation suggests. Or check /sys/devices/system/cpu/vulnerabilities/ on an older machine and work out what each mitigation is costing you at this boundary.

Epilogue

I'm currently experimenting with an OS design on top of the Linux kernel that would bring an Android-style permissions and capabilities model to a desktop OS while trying to be 100% compatible with the Debian ecosystem. This has led to some really interesting research lately. This article is a product of that research.

I'll be writing more about the Linux kernel before I move on to formal verification, as in the DO-178C and DO-333 world where avionics software has to qualify the tools that check it. Usually that means Viper, Why3, Z3 and friends.

In the meantime, I also write about systems that have to survive contact with reality at thechris.in, including a companion piece to this one, on what it means to build on an abstraction whose cost you can measure but never see.

One important yet sometimes overlooked tool available to engineers for querying and retrieving system information is the CPUID instruction.

What is the CPUID Instruction?

The CPUID instruction is a low level instruction, inside the heart of every modern x86 and x86-64 processor that allows the software to query the CPU for information about the processor and its supported features.

By invoking this instruction, you can gather information such as the processor’s model, family, internal cache sizes, and supported features like SIMD or hardware virtualization. This can help you optimize performance and dynamically enable or disable supported features.

For bootloader or kernel developers, understanding what features a processor supports—such as hardware virtualization, cache sizes, or SIMD instructions—can ensure that the system runs efficiently and that the code you write is compatible across different CPUs. By utilizing the CPUID instruction, you can dynamically adjust your kernel’s behavior based on the specific processor it is running on.

In this article you will learn how to check if the CPUID instruction is available for your system, how it works and what information you can get from using it.

Prerequisites

• Some knowledge of assembly language (for this example I use FASM)

• Some knowledge of operating systems/kernels

• Access to low-level debugging tools (for example, GDB) or hardware emulators like QEMU to test your bootloader/kernel on various platforms.

Step 1: Check for CPUID Availability

Before executing the CPUID instruction, it's important to determine whether the processor supports it, as not all CPUs are guaranteed to have this functionality. The following code checks the availability of the CPUID instruction by modifying and testing the ID bit (bit 21) in the EFLAGS register.

Here’s a picture from wiki.osdev.org that shows each bit of the EFLAGS register:

If the processor allows this bit to be toggled, CPUID is supported; otherwise, it is not. Here's how the detection process works:

(most people think that in Real mode 32 registers are not accessible. That is not true. All 32bit registers are usable)

cpuid_check:
    pusha                                ; save state
    pushfd                               ; Save EFLAGS
    pushfd                               ; Store EFLAGS
    xor dword [esp],0x00200000           ; Invert the ID bit in stored EFLAGS
    popfd                                ; Load stored EFLAGS (with ID bit inverted)
    pushfd                               ; Store EFLAGS again (ID bit may or may not be inverted)
    pop eax                              ; eax = modified EFLAGS (ID bit may or may not be inverted)
    xor eax,[esp]                        ; eax = whichever bits were changed
    popfd                                ; Restore original EFLAGS
    and eax,0x00200000                   ; eax = zero if ID bit can't be changed, else non-zero
    cmp eax,0x00
    je .cpuid_instruction_not_is_available
.cpuid_instruction_is_available:
    ;handle CPUID exists
.cpuid_instruction_not_is_available:
    ;handle CPUID isn't supported
.cpuid_check_end:
    popa                                  ; restore state
    ret

pusha: Saves all the general purpose registers to ensure the original state can be restored at the end.

pushfd: Saves the current EFLAGS register.

pushfd: Stores a copy of the EFLAGS.

xor dword [esp], 0x00200000: The code flips the ID bit (21) of the EFLAGS using the XOR operator.

popfd: Restores the modified EFLAGS with the ID bit inverted.

pushfd: Pushes the modified EFLAGS back to the stack.

pop eax: Puts the modified EFLAGS (ID bit may or may not be inverted) in the EAX register.

xor eax, [esp]: After the XOR operation, the EAX will contain the bits that were changed.

popfd: Restores the original EFLAGS.

and eax, 0x00200000: The and operation isolates the 21st bit (ID bit) by masking all other bits. After this operation the EAX register will contain either 0x00200000 (if 21 bit was changed which means CPUID is supported) or 0×00 (21 bit hasn’t changed, CPUID not supported).

cmp eax, 0x00: The CMP instruction checks the result of the previous operation. If EAX equals 0×00, it means that the ID bit cannot be modified and the processor doesn’t support the CPUID instruction. If it is not zero, it means that the ID bit was flipped and your processor supports the CPUID instruction.

Step2: How to Use The CPUID Instruction

Get CPU Features

The CPUID instruction will return different information with different values in the EAX register.

mov eax, 0x1
cpuid

With EAX set to 1, the CPUID will return a bitfield in EDX, which will contain the following values. Different brands may give different meaning to these (source https://wiki.osdev.org/CPUID)

enum {
    CPUID_FEAT_ECX_SSE3         = 1 << 0,
    CPUID_FEAT_ECX_PCLMUL       = 1 << 1,
    CPUID_FEAT_ECX_DTES64       = 1 << 2,
    CPUID_FEAT_ECX_MONITOR      = 1 << 3,
    CPUID_FEAT_ECX_DS_CPL       = 1 << 4,
    CPUID_FEAT_ECX_VMX          = 1 << 5,
    CPUID_FEAT_ECX_SMX          = 1 << 6,
    CPUID_FEAT_ECX_EST          = 1 << 7,
    CPUID_FEAT_ECX_TM2          = 1 << 8,
    CPUID_FEAT_ECX_SSSE3        = 1 << 9,
    CPUID_FEAT_ECX_CID          = 1 << 10,
    CPUID_FEAT_ECX_SDBG         = 1 << 11,
    CPUID_FEAT_ECX_FMA          = 1 << 12,
    CPUID_FEAT_ECX_CX16         = 1 << 13,
    CPUID_FEAT_ECX_XTPR         = 1 << 14,
    CPUID_FEAT_ECX_PDCM         = 1 << 15,
    CPUID_FEAT_ECX_PCID         = 1 << 17,
    CPUID_FEAT_ECX_DCA          = 1 << 18,
    CPUID_FEAT_ECX_SSE4_1       = 1 << 19,
    CPUID_FEAT_ECX_SSE4_2       = 1 << 20,
    CPUID_FEAT_ECX_X2APIC       = 1 << 21,
    CPUID_FEAT_ECX_MOVBE        = 1 << 22,
    CPUID_FEAT_ECX_POPCNT       = 1 << 23,
    CPUID_FEAT_ECX_TSC          = 1 << 24,
    CPUID_FEAT_ECX_AES          = 1 << 25,
    CPUID_FEAT_ECX_XSAVE        = 1 << 26,
    CPUID_FEAT_ECX_OSXSAVE      = 1 << 27,
    CPUID_FEAT_ECX_AVX          = 1 << 28,
    CPUID_FEAT_ECX_F16C         = 1 << 29,
    CPUID_FEAT_ECX_RDRAND       = 1 << 30,
    CPUID_FEAT_ECX_HYPERVISOR   = 1 << 31,

    CPUID_FEAT_EDX_FPU          = 1 << 0,
    CPUID_FEAT_EDX_VME          = 1 << 1,
    CPUID_FEAT_EDX_DE           = 1 << 2,
    CPUID_FEAT_EDX_PSE          = 1 << 3,
    CPUID_FEAT_EDX_TSC          = 1 << 4,
    CPUID_FEAT_EDX_MSR          = 1 << 5,
    CPUID_FEAT_EDX_PAE          = 1 << 6,
    CPUID_FEAT_EDX_MCE          = 1 << 7,
    CPUID_FEAT_EDX_CX8          = 1 << 8,
    CPUID_FEAT_EDX_APIC         = 1 << 9,
    CPUID_FEAT_EDX_SEP          = 1 << 11,
    CPUID_FEAT_EDX_MTRR         = 1 << 12,
    CPUID_FEAT_EDX_PGE          = 1 << 13,
    CPUID_FEAT_EDX_MCA          = 1 << 14,
    CPUID_FEAT_EDX_CMOV         = 1 << 15,
    CPUID_FEAT_EDX_PAT          = 1 << 16,
    CPUID_FEAT_EDX_PSE36        = 1 << 17,
    CPUID_FEAT_EDX_PSN          = 1 << 18,
    CPUID_FEAT_EDX_CLFLUSH      = 1 << 19,
    CPUID_FEAT_EDX_DS           = 1 << 21,
    CPUID_FEAT_EDX_ACPI         = 1 << 22,
    CPUID_FEAT_EDX_MMX          = 1 << 23,
    CPUID_FEAT_EDX_FXSR         = 1 << 24,
    CPUID_FEAT_EDX_SSE          = 1 << 25,
    CPUID_FEAT_EDX_SSE2         = 1 << 26,
    CPUID_FEAT_EDX_SS           = 1 << 27,
    CPUID_FEAT_EDX_HTT          = 1 << 28,
    CPUID_FEAT_EDX_TM           = 1 << 29,
    CPUID_FEAT_EDX_IA64         = 1 << 30,
    CPUID_FEAT_EDX_PBE          = 1 << 31
};

A brief explanation of the CPU features above:

• PCLMUL, AES: Cryptographic instruction sets for fast encryption and decryption.

• VMX, SMX: Virtualization support for running virtual machines.

• SSE3, SSSE3, SSE4.1, SSE4.2, AVX: SIMD instruction sets for faster multimedia, math, and vector processing.

• FMA: Fused Multiply-Add, improves performance in floating-point calculations.

• RDRAND: Random number generator.

• X2APIC: Advanced interrupt handling in multiprocessor systems.

• PCID: Optimizes memory management during context switches.

• FPU: Hardware floating-point unit for faster math operations.

• PAE: Physical Address Extension, allows addressing more than 4 GB of memory.

• HTT: Allows a single CPU core to handle multiple threads.

• PAT, PGE: Memory management features for controlling caching and page mapping.

• MMX, SSE, SSE2: Older SIMD instruction sets for multimedia processing.

Get CPU Vendor String

If you want to get the CPU vendor string, EAX should be set to 0×0 before invoking the CPUID instruction.

mov eax, 0x0
cpuid

The vendor string is a unique identifier that CPU vendors like AMD and Intel use. Examples are: GenuineIntel (for Intel processors) or AuthenticAMD (for AMD processors). It basically specifies the manufacturer of the CPU.

The vendor string allows the kernel to identify the CPU manufacturer which is very useful because different manufacturers implement certain features differently. Also, software or drivers can interact differently based on the CPU manufacturer to ensure compatibility.

When used like this, the vendor id string will be returned in EBX, EDX, ECX registers. You can write them to a buffer and get the full 12 character string.

Example code:

Step 1: The Buffer

Create a buffer that can hold 12 bytes:

buffer: db 12 dup(0), 0xA, 0xD, 0

Step 2: Print the Buffer

We start by creating a string printing function.

This assembly code reads a string character by character and prints it to the screen using BIOS interrupt 0x10. The print function loops through the string and uses the lodsb instruction to load each character in the al register.

Then the print_char function uses the interrupt 0×10 to print it on the screen. When the code reaches the end of the string (null terminator), the loop ends.

print_string:
    call print
    ret
print:
.loop:  
    lodsb   ;read character to al and then increment
    cmp al ,0 ;check if we reached the end
    je .done  ;we reached null terminator, finish
    call print_char ;print character
    jmp .loop   ;jump back into the loop
.done:
    ret
print_char:
    mov ah, 0eh
    int 0x10
    ret

Step 3: Fill the Buffer and Print it

Here, after saving the current state using the pusha instruction and calling cpuid with 0×0 passed in the EAX register, we can store the contents of ebx, edx, ecx to the buffer. Then we call print_string to print it.

get_cpu_vendor:
    pusha
    mov eax, 0x0
    cpuid
    mov [buffer], ebx
    mov [buffer + 4], edx
    mov [buffer + 8], ecx
    mov si, buffer 
    call print_string
    popa
    ret

A video from my YouTube channel where I implement and explain the code above in detail

More information about what information CPUID instruction can give you according to the value passed in the EAX register, can be found here: https://gitlab.com/x86-cpuid.org/x86-cpuid-db

Epilogue

By understanding and using the CPUID instruction, you can make your bootloader/kernel more adaptable to a wide range of processors. Knowing how to detect the instruction's availability and retrieve crucial system information—such as CPU features, cache sizes, and supported technologies—can significantly enhance performance and compatibility.

After reading this article, you should have the tools and knowledge to start exploring the CPUID instruction and how you can use it in your own project!

Happy coding!

But not all memory is freely available for use. Some memory sections are reserved for system functions and others may be occupied by hardware devices. That’s why it is very important to get the system’s memory map.

What is a Memory Map?

But what is a memory map? A memory map is a representation (think about it like a table) that shows how physical memory is organized in your system. It shows the address of each memory region, it’s length and it’s type.

Type 1 means that the region is available for you to use freely and type 2 means that it is reserved by your system. Type 3 means that the region is reserved for the Advanced configuration and power interface (ACPI 3.x). While a type 3 region might not be used by the system, it can be reclaimed later.

Using a memory map will allow you to manage memory resources successfully without any issues such as crashes or system instability.

There are some ways you can detect your system’s available memory. One is by using the BIOS and interrupt 15h. Another one is by doing memory probing.

In this article you will learn which tools are available to help you get a memory map of your system, which ones you should use, and which ones you should avoid and why. Then finally, you will see some assembly code that you can use in your own bootloader / kernel.

Prerequisites

if you want to follow along with the code shown in this article, you’ll need:

• A Linux operating system

• Some knowledge of assembly language

• A text editor of your choice

• An emulator installed. For this example I use QEMU.

• FASM assembler installed

• Git to be able to clone the repository (https://github.com/nikolaospanagopoulos/memoryMapBoot)

A Few Words about BIOS int 15h

In Real mode, the BIOS offers many interrupts that interact with the hardware and can give you information.

There are some interrupts that can help with getting a memory map, but the most powerful one is int15h with E820h function (hexadecimal numbers! very important to remember. Decimal numbers will not work). This method offers a detailed memory map that you can use to safely determine which areas of memory can be used for vital tasks like setting up paging, memory allocation, and more.

In this article you will see how you can use this interrupt to get a detailed memory map of your system.

Now, before we go deeper, I would like to add a few things about memory probing and why you should avoid it.

Memory probing and why you should avoid it

Memory probing is the process of manually accessing physical memory and determining whether it is available or not. The issue is that not all memory is designed to be accessed directly.

Accessing parts of memory that you shouldn’t can cause unpredictable behavior like:

• System Crashes: some memory is reserved for BIOS structures, hardware devices etc. Accessing those areas can lead to system crashes or system instability.

• Memory Corruption: accessing reserved memory areas can lead to corruption of those areas. This can cause again crashes, instability, malfunctions etc

So, you should avoid memory probing because it’s an unnecessary risk to your kernel development process.

The Code

Step 1: Prepare to Call int 15h

In this part, you will basically setup the environment needed to invoke int 15h. The general purpose registers need to be stored so that no important data on them is lost during the interrupt invocation. Then the registers bp, ebx are cleared so that they can be set to their initial values.

The “SMAP” value is stored in the edx register to ensure the correct format that the BIOS will return. Finally, we setup the 0xe820 function and request memory map data.

pusha
mov di, 0x0504        ; Set DI register for memory storage
xor ebx, ebx          ; EBX must be 0
xor bp, bp            ; BP must be 0 (to keep an entry count)
mov edx, 0x534D4150   ; Place "SMAP" into edx | The "SMAP" signature ensures that the BIOS provides the correct memory map format
mov eax, 0xe820       ; Function 0xE820 to get memory map
mov dword [es:di + 20], 1 ; force a valid ACPI 3.X entry | allows us to get additional information (extended attributes)
mov ecx, 24           ; Request 24 bytes of data

• The pusha command pushed all general purpose registers to the stack to save their values during the interrupt call. They can be restored after the interrupt call to avoid corruption of other areas.

• The mov di, 0x0504 instruction sets the di register to 0×0504 (where the memory map entries will be stored).

• xor ebx, ebx the xor instruction uses the xor operator to clear the ebx register. It must be set to 0 to start retrieving entries.

• xor bp, bp use of the same xor operator here to set bp to 0. This will keep track of your memory entries.

• mov edx, 0x534D4150 this instruction will store 0x534D4150 (ASCII string “SMAP”) into the edx register. It makes certain that the BIOS will return the correct format for your memory map.

• mov eax, 0xe820 this instruction sets the function 0xe280 which will get the memory map along with int15h.

• mov dword [es:di + 20], 1 this instruction forces a valid ACPI (Advanced Configuration and Power Interface) 3.x entry. This way the BIOS provides extra information in the form of extra attributes.

• mov ecx, 24 this instruction asks the BIOS for 24 bytes of memory data. This is the size that ACPI 3.x entries need to include extra information.

Step 2: Call int15h

Here, you can finally invoke the interrupt to fetch the memory map. You need to check that the function is supported by the BIOS and that valid data is being fetched. You also need to ensure that the correct format is being fetched by setting again the “SMAP” into the edx register.

    int 0x15                 ; using interrupt
    jc short .failed         ; carry set on first call means "unsupported function"
    mov edx, 0x534D4150      ; Some BIOSes apparently trash this register? lets set it again
    cmp eax, edx             ; on success, eax must have been reset to "SMAP"
    jne short .failed
    test ebx, ebx            ; ebx = 0 implies list is only 1 entry long (worthless)
    je short .failed

• int 0x15 this instruction invokes the interrupt 0×15.

• jc short .failed is the carry flag that is set. It means the function is unsupported and the call has failed. It jumps to our error handler.

• mov edx, 0x534D4150 set again the “SMAP” because some BIOSes corrupt this register after the call.

• cmp eax, edx if the call is successfull, on success the BIOS will return the “SMAP” value in eax.

• jne short .failed if it doesn’t, it means the call has failed and it jumps to our error handling label.

• test ebx, ebx this instruction checks if ebx is 0 after the first call. This means that the memory map only contains one entry. This entry is probably invalid, so it jumps to the error handling label.

Step 3: Loop Through Memory Entries

After a successful first invocation, you need to loop through each entry of the memory map.

In the loop, you will invoke again int 15h to get all subsequent memory entries while checking each entry’s length and other attributes. If it meets the criteria, you increment the counter and you store the entry. This continues until there are no entries left to process.

    jmp short .jmpin
.e820lp:
    mov eax, 0xe820          ; eax, ecx get trashed on every int 0x15 call
    mov dword [es:di + 20], 1 ; force a valid ACPI 3.X entry
    mov ecx, 24              ; ask for 24 bytes again
    int 0x15
    jc short .e820f          ; carry set means "end of list already reached"
    mov edx, 0x534D4150      ; repair potentially trashed register
.jmpin:
    jcxz .skipent            ; skip any 0 length entries (If ecx is zero, skip this entry (indicates an invalid entry length))
    cmp cl, 20               ; got a 24 byte ACPI 3.X response?
    jbe short .notext
    test byte [es:di + 20], 1 ;if bit 0 is clear, the entry should be ignored
    je short .skipent         ; jump if bit 0 is clear 
.notext:
    mov eax, [es:di + 8]     ; get lower uint32_t of memory region length
    or eax, [es:di + 12]     ; "or" it with upper uint32_t to test for zero and form 64 bits (little endian)
    jz .skipent              ; if length uint64_t is 0, skip entry
    inc bp                   ; got a good entry: ++count, move to next storage spot
    add di, 24               ; move next entry into buffer
.skipent:
    test ebx, ebx            ; if ebx resets to 0, list is complete
    jne short .e820lp

• .e820lp is a label for looping through each memory map entry.

The next lines are used to call int15h to get the next memory entry:

• jc short .e820f if the carry flag is set, it means that we have reached the end of the list.

• jcxz .skipent if ecx register is 0, it means the length of the memory entry is invalid. So the code skips it.

• cmp cl, 20 checks if the memory entry is a valid ACPI 3.x entry. (It would be 24 bytes long). If it is not, the code jumps to .notext.

• test byte [es:di + 20], 1 checks if bit 0 is set in the memory entry's extended attributes, indicating a valid entry. If it's clear, the entry is skipped.

• mov eax, [es:di + 8] gets the lower 32 bits of the memory region length and then we combine it using the or operator, with the upper 32 bits. If the total length is 0, then the entry is skipped.

• inc bp increments entry count.

• add di, 24 moves the pointer di forward to the next memory entry. Each entry is 24 bytes long.

Step 4: End of Memory Entries Handling

Finally, you can store the entry count. And by using the popa instruction, you will restore all general purpose registers to their previous values. If an error occurs during the process, the code jumps to .failed label which is our error handling function.

.e820f:
    mov [mmap_ent], bp       ; store the entry count
    clc                      ; there is "jc" on end of list to this point, so the carry must be cleared

    popa
    ret
.failed:
    stc                      ; "function unsupported" error exit
    ret

• mov [mmap_ent], bp stores the entry count.

• clc clears the carry flag because it is already set.

• popa pops all general purpose registers back from the stack.

• .failed we use this label for error handling.

Here is a video from my YouTube account where I implement and explain the above code:

Epilogue

In kernel development, one of the most important tasks is managing memory. The above is a reliable way to detect your system’s memory layout information. This means that you can make safe decisions when allocating resources, implementing paging, and so on.

It might appear to be complex and it maybe is, but if you follow the code line by line you will be able to understand it. These techniques will allow you to build a robust kernel capable of running on different hardware configurations.

Keep Coding!