The cryptographic operation returns the right ciphertext. The parser rejects malformed input. The cache survives its benchmark. The release candidate passes every unit and integration test.

But somewhere below those correct results, a wrapper still believes it owns a handle that's already transferred. A success path returns without initializing an output field. A finalizer is waiting to free an object whose lifetime is now controlled by another runtime. Two threads mutate the same state, but the scheduler hasn't chosen the interleaving that makes the race visible.

Nothing in the expected output necessarily reveals any of this.

The first observable failure may arrive hours later, after an allocator reuses a freed region, a garbage-collection cycle changes reachability, a shutdown path finally accounts for leaked allocations, or production traffic creates a synchronization pattern that the test suite never exercised. By then, the crashing stack often belongs to the victim of the bug rather than its author.

This is the gap sanitizers are built to close. They do more than make crashes easier to debug. They attach runtime meaning to assumptions that ordinary tests usually leave implicit: this address is still valid to access, this allocation still has a legitimate owner, this value was initialized before it influenced execution, or this shared-memory access is ordered by synchronization the runtime can observe.

AddressSanitizer (ASan), LeakSanitizer (LSan), MemorySanitizer (MSan), and ThreadSanitizer (TSan) aren't four variations of the same debugging mode. They observe different properties of the execution.

ASan tracks addressability and lifetime. LSan asks what allocations remain live at teardown and whether that remaining reachability makes sense. MSan follows initialization provenance. TSan reconstructs the happens-before relationships that make concurrent accesses legal.

The difficult part starts after the compiler accepts the flag.

A sanitizer build is only useful to the extent that you understand what it actually observed. Which modules were instrumented? Which allocator or synchronization operations were invisible to the runtime? Can the report be symbolized well enough to identify useful frames? Did the workload reach the ownership transition, failure path, or thread interleaving where the bug lives?

And once the lane is producing trustworthy evidence, where does it belong: on every pull request, in a slower scheduled job, or initially in observation mode while the baseline is understood?

Those questions become especially sharp at managed/native boundaries. A Dart, Java, Python, Swift, C#, or Rust wrapper may present a safe-looking API while depending on native reference counts, allocator pairing, borrowed pointers, transfer-on-success conventions, callbacks, arenas, and finalizers underneath. The language boundary doesn't remove those rules. It distributes them across two runtimes that may have very different ideas about ownership and lifetime.

Many of the examples in this handbook come back to that kind of boundary: code that looks straightforward from the outside but has to keep native lifetime rules intact underneath.

google/webcrypto.dart will serve as one recurring example. It combines Dart FFI, BoringSSL handles, finalizers, scoped cleanup, build hooks, and ownership transfer in a relatively compact system. It's also a useful example of a less glamorous but very common engineering constraint: sometimes the sanitizer setup you want is blocked by the surrounding toolchain, and you have to decide what useful runtime evidence you can obtain in the meantime.

That problem isn't specific to Dart or cryptography. Real systems rarely give you a perfectly instrumentable dependency graph, a cooperative runtime, complete symbols, and unlimited CI time all at once. The practical question is usually narrower: what can this build observe today, what can it not observe, and what workload would make the available evidence meaningful?

That's the argument of this handbook: sanitizers make runtime contracts executable. Ownership is a recurring part of that story, especially across FFI boundaries, but so are initialization and synchronization. Fuzzers, stress tests, failure injection, and concurrency probes create executions that challenge those contracts. Sanitizer runtimes make violations visible. CI determines whether those findings become durable regression protection or remain one-off debugging events.

The chapters that follow move from that model into shadow memory, leak reachability, initialization tracking, race detection, FFI ownership, workload design, report triage, and release policy. The goal is not merely to run sanitizers. It's to understand what a clean run actually tells you, what a report says about the assumptions in your code, and what remains outside the tool's field of view.

Table of Contents

• Prerequisites

• The Gap Between Correct Results and Valid Execution

• Sanitizers as Executable Ownership Contracts

• The Four Sanitizers in Depth

• Combining Sanitizers and Knowing What They Do Not Cover

• FFI Ownership Across Runtime Boundaries

• Case Study:webcrypto.dart Issue #278

• Incident Patterns from Real Systems

• Fuzzing, Stress Harnesses, and Concurrency Probes

• How to Read and Triage Sanitizer Reports

• CI Policy and Acceptance Criteria

• Production Strategy, Suppression Debt, and Metrics

• Closing Perspective

• References and Further Reading

Prerequisites

This handbook assumes basic comfort with native-code debugging and concurrent programs. It builds on pointers, allocation, thread synchronization, build systems, and CI rather than reintroducing C or C++ fundamentals.

The focus here is dynamic execution evidence: what the sanitizer runtime can observe, what its report means, and how ownership, initialization, and synchronization assumptions should shape the workload and the pipeline around it.

Here's what you should already be comfortable with:

• Reading a native stack trace and moving between caller, callee, allocation, and cleanup paths.

• Basic ownership concepts such as borrowed versus owned pointers, transfer of ownership, reference counting, and deterministic cleanup.

• Heap and stack allocation at a practical level, including why allocator pairing and object lifetime matter.

• Concurrency fundamentals such as mutexes, atomics, thread creation, and the idea that ordering between threads must be established rather than assumed.

• Building and running tests from the command line, and understanding the difference between a local debug build and a CI build.

And here's what you need to follow along with the examples:

• A recent Clang/LLVM toolchain with access to clang, clang++, and llvm-symbolizer.

• A Linux or macOS environment for most compiler-based examples. Linux is the most practical environment for the complete set of examples and the Valgrind lane.

• A way to run the produced binaries and preserve their logs and crash artifacts.

• Optional access to Valgrind for the whole-process and FFI fallback examples.

• Optional Dart tooling for the webcrypto.dart case study. You don't need Dart experience to follow the ownership analysis.

• And, less officially, enough patience to read a stack trace before blaming the last person who touched the file :)

We won't go through a long compiler or environment setup walkthrough. The commands assume the relevant compiler and runtime packages are installed and focus on instrumentation, reports, workloads, and policy.

We also won't re-teach general memory management, threading, fuzzing, or compiler construction. Only the parts that change how a sanitizer result should be interpreted are developed in detail.

The Gap Between Correct Results and Valid Execution

Before getting into shadow memory, leak roots, origin tracking, or happens-before graphs, it helps to separate two properties that are easy to conflate: functional correctness and execution validity.

A conventional test asks a functional question: given this input, did the system return the expected value, mutate the expected state, or produce the expected side effect?

A sanitizer asks something different: was the execution that produced that result valid?

Those questions overlap, but they aren't equivalent. A function can return the correct plaintext while leaking a native key. A parser can reject malformed input while reading an uninitialized output length on the way to that rejection. A wrapper can work correctly for months while two different objects both believe they own the same native handle. A lock-free cache can pass every deterministic test and still contain a race that the scheduler hasn't exposed yet.

That distinction is the starting point for sanitizer work:

The diagram shows why a passing test is not always a valid execution. Sanitizers check the hidden runtime contracts that ordinary output-based tests can miss.

Bugs Often Begin Far Away from the Crash

Memory bugs rarely fail where they begin.

The first mistake may be small: a missing reference increment, a pointer that remains usable after ownership moves, a cleanup callback removed from the wrong scope, an out-parameter left untouched on one success path, or a finalizer detached a little too late.

The visible failure may happen much later, in another component, on another thread, or during shutdown. By then, the stack at the crash site often describes the code that suffered from the bug rather than the code that created it.

Sanitizers help because they maintain runtime state that ordinary execution does not. ASan tracks whether memory is addressable and how that changes over an object's lifetime. MSan propagates initialization state. TSan records memory accesses and synchronization relationships. LSan examines what remains reachable when the process terminates.

That extra state gives you more than the final crash location. Depending on the tool, a report may show where an object was allocated and freed, where an uninitialized value originated, or which accesses raced across threads. The report still needs interpretation, but it gives you enough history to start reconstructing how the program reached the invalid state.

Correct Outputs Can Hide Invalid Execution

Consider a native API that writes a 32-byte digest into a caller-provided buffer:

bool digest(const uint8_t* input, size_t input_len, uint8_t* output, size_t* output_len);

A reasonable contract is that a true return means both output and *output_len have been initialized. Now suppose one success path writes the digest but forgets to set the length.

A unit test that checks only the 32 digest bytes may still pass. The caller may even happen to read 32 from the uninitialized stack slot often enough that the bug stays hidden.

MSan asks a more useful question than “did the value happen to look right?” It asks whether that value had a legitimate initialization history before the program used it.

Lifetime bugs have the same shape. Suppose a wrapper transfers a native handle to another owner but leaves its finalizer attached. Nothing necessarily fails at the point of transfer. Ordinary tests may complete before the finalizer ever runs.

Under GC pressure, though, the old wrapper can eventually release the handle while the new owner still expects it to be alive. The next native access may then become a use-after-free.

The functional tests weren't wrong. They were just answering a different question. They verified the operation's result but they didn't exercise the object lifetime or GC timing needed to expose the ownership mistake.

Sanitizers Still Need Workload Coverage

A sanitizer can only report a violation that occurs during an execution it observes.

ASan can't catch a stale pointer that nobody dereferences. TSan can't report a race if the conflicting accesses never overlap. LSan can't expose a leak in a path the workload never enters. MSan can't reveal an initialization bug if no input reaches the branch that skips the write.

This is where fuzzing, stress testing, failure injection, and concurrency probes become important. They aren't alternatives to sanitizers. They make interesting executions more likely to happen.

Coverage-guided fuzzing explores unusual input paths. Lifecycle stress repeatedly pushes objects through create, retain, transfer, fail, close, finalize, and teardown. Failure injection forces cleanup and rollback code that happy-path tests rarely visit. Concurrency probes make inconvenient thread schedules occur often enough to observe. Corpus replay keeps previously discovered failures from quietly returning.

The sanitizer's job is to make the resulting invalid state visible and, ideally, explain enough of its history to debug it.

That's why a hundred thousand short create/use/close cycles can tell you more about lifetime correctness than a benchmark that processes a gigabyte through one long-lived object. The benchmark spends most of its time in steady state. The lifecycle workload spends its time crossing the transitions where ownership mistakes tend to surface.

Dynamic Analysis is Not Omniscient

A clean sanitizer run is useful evidence. It's not proof that the code is safe.

It tells you that, for this build and this workload, the parts of the program visible to the sanitizer didn't violate a contract the tool knew how to check.

That scope can be narrower than it first appears.

Some modules may not have been instrumented. A custom allocator may hide the logical object boundaries you care about. Synchronization may happen inside code TSan can't observe. Symbols may be missing or optimization may make stacks harder to reconstruct. A runtime may manage memory in ways that don't map cleanly onto the sanitizer's model. Suppressions intentionally remove some findings from view.

And none of those problems has to exist for a bug to be missed. A perfectly instrumented binary still tells you nothing about a path the workload never executes.

So after a clean run, the useful question is not:

Are we safe now?

It is:

What did this run actually observe, and what remained outside its field of view?

That is the right level of confidence to attach to dynamic analysis. A green sanitizer lane is strong runtime evidence about the executions it observed, not a certificate for the rest of the program.

Sanitizers as Executable Ownership Contracts

The usual introduction to sanitizers is a list of bug classes: ASan finds use-after-free and out-of-bounds accesses, LSan finds leaks, MSan finds uses of uninitialized memory, and TSan finds data races.

That description is correct, but it's not especially helpful once you're looking at a real report.

A better model is to ask what assumption the program made that the sanitizer was able to disprove. In ownership-heavy code, that often means asking who was allowed to use a resource, who was responsible for releasing it, and whether that answer changed somewhere along the way.

For MSan and TSan, the same idea extends beyond ownership to initialization and synchronization: what had to be true for this operation to be valid, and did the observed execution actually establish it?

ASan's contract is about addressability and lifetime. The program behaves as though a region of memory is still legal to access. ASan maintains enough runtime state to say whether that's true. If the region has been freed, quarantined, or poisoned as a red zone, the access violates the runtime's view of that lifetime.

LSan looks at the other end of ownership. Instead of asking whether an object was accessed after its lifetime ended, it asks why an allocation's lifetime never ended at all. At process teardown, it traces reachable allocations from known roots. Memory that remains allocated without a reachable ownership path is reported as leaked. That's why I think of LSan as terminal ownership accounting rather than simply “the leak sanitizer.”

MSan tracks a different contract: initialization provenance. Addressable memory isn't necessarily valid data. A byte that was never initialized shouldn'y quietly become a branch condition, pointer, size, parameter, or return value. MSan propagates that state through the execution and reports when the program finally treats uninitialized state as meaningful.

TSan deals with synchronization visibility. Two threads may both access the same state, but if those accesses conflict, there must be an ordering relationship that makes them legal. TSan reconstructs the synchronization it can observe and reports accesses that remain unordered. That distinction matters: the program may believe a custom runtime, callback protocol, or external library provides synchronization, while TSan sees no such edge.

These models become particularly concrete around native APIs because native code tends to expose ownership decisions directly.

BoringSSL is a useful example, although the pattern isn't specific to cryptography. Its API conventions distinguish operations such as get0, get1, set0, and set1 precisely because returning or accepting the same pointer type doesn't mean the same thing for ownership.

A get0-style result is borrowed. A get1-style result carries an owning reference. A set0-style operation transfers ownership, while set1 retains a reference without transferring the caller's existing one. Reference-counted objects expose operations such as *_up_ref, and allocated objects have corresponding release functions.

Those names are doing real correctness work. If a wrapper flattens all of those cases into “some native pointer,” it discards information the API was deliberately trying to preserve.

That's how a wrapper ends up freeing a borrowed handle. Or continuing to use a handle after ownership was transferred. Or retaining a reference without ever releasing it. The pointer value itself hasn't changed. What changed was the program's right to use or destroy what it points to.

Finalizers make the same problem less obvious. A finalizer is usually attached while a wrapper owns a native resource. If ownership later moves into another object but the finalizer remains attached, both objects may now contain a cleanup path for the same resource. Nothing has to fail immediately. The eventual result depends on which cleanup path runs first: a double free, a stale access, or a lifetime that accidentally survives longer than intended.

MSan exposes a similar kind of disagreement at a different boundary. Suppose a function reports success, and the caller interprets success as “all outputs are initialized.” If one successful branch leaves an out-parameter untouched, the caller and callee are operating under different contracts even though neither side necessarily contains an obviously suspicious memory access.

TSan reports have the same shape when synchronization crosses abstraction boundaries. The code may be correct only if a particular lock, atomic operation, callback protocol, or runtime event establishes ordering. If that ordering doesn't exist (or exists somewhere the instrumented execution can't see) the report is telling you that the synchronization story is incomplete from the runtime's point of view.

So a sanitizer report is often more useful when read as a disagreement between components rather than as a local bad instruction. One part of the program says the object is alive, while another has already released it. One side says the result is initialized but the other never wrote it. One thread assumes another has published state safely but the observed synchronization graph says otherwise.

For FFI-heavy code, ownership is where these disagreements accumulate most quickly.

The following ownership-flow diagram is a useful mental model for FFI-heavy systems:

The diagram shows why FFI ownership needs explicit state changes. If a handle is borrowed, transferred, finalized, or scoped incorrectly, the result can be a leak, double free, or stale finalizer cleanup.

The same pattern shows up far beyond BoringSSL. Native APIs routinely distinguish uniquely owned objects, reference-counted objects, borrowed views, transferred handles, arena-owned memory, and temporary resources that must be released if an operation fails halfway through.

The terminology varies, but the questions do not:

Who owns the resource now? Is this reference borrowed or retained? What operation ends its lifetime? Does ownership move only on success? If cleanup is deferred, who's still responsible for it while that delay exists?

Those distinctions are what make sanitizer output easier to reason about. When ownership transitions are explicit, an ASan use-after-free can often be traced back to a specific transfer that left a stale owner behind. An LSan report can be tied to the reference that never reached its matching release. When the transitions are implicit, the sanitizer may still find the bug, but the report has to be reverse-engineered against an ownership model that was never clearly represented in the code.

The same principle applies to MSan and TSan even though their contracts are not ownership contracts in the narrow sense. APIs should make initialization postconditions explicit. Concurrent code should make synchronization relationships explicit. The less those rules depend on unwritten assumptions, the easier it is to understand what a sanitizer has actually disproved.

That's the useful sense in which these tools make contracts executable. They don't invent the ownership, initialization, or synchronization rules of the program. Those rules already exist. The sanitizer gives some of them runtime state, observes an execution, and tells you when the program behaves as though a rule were true when the evidence says otherwise.

The Four Sanitizers in Depth

The four main sanitizers differ less in syntax than in what they're capable of knowing about the execution.

The costs are only rough planning numbers. Workload shape, allocator behavior, dependency size, platform, and runtime architecture can move them considerably.

AddressSanitizer (ASan): Addressability and Lifetime

AddressSanitizer is often introduced as a fast way to find memory corruption, but the model is more specific: ASan tracks whether a region of memory is addressable at the moment the program touches it.

The compiler instruments loads and stores so they can be checked against shadow metadata maintained by the runtime. Heap objects are surrounded by poisoned redzones. Freed memory is poisoned as well, and recently freed allocations are commonly held in quarantine before they can be reused.

Stack instrumentation applies the same idea to local lifetimes, which is how ASan can catch use-after-scope and, with the relevant support enabled, use-after-return.

That machinery is why ASan maps so naturally onto lifetime bugs. A wrapper can make an ownership mistake long before anything crashes. The mistake becomes visible to ASan only when it eventually turns into a concrete access to memory that the runtime considers dead or out of bounds.

The set of bugs ASan can catch is correspondingly broad: heap, stack, and global out-of-bounds accesses, use-after-free, use-after-scope, use-after-return, double free, invalid free, and, on supported platforms, leak detection through the sanitizer runtime.

For debugging builds, the compiler flags should favor useful reports rather than maximum optimization. A practical baseline is:

clang++ -O1 -g -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize=address -o my_tests sanitizer_tests.cc

ASAN_SYMBOLIZER_PATH="$(command -v llvm-symbolizer)"
ASAN_OPTIONS="detect_leaks=1:check_initialization_order=1" 
./my_tests

The build keeps debug information and frame pointers, and avoids sibling-call optimization so useful callers are less likely to disappear from a stack. The runtime invocation makes llvm-symbolizer available and enables leak detection plus initialization-order checking where the platform supports it.

There's no special value in those exact flags if they don't fit your build system. The point is to produce a binary whose failure can be explained. A sanitizer lane that saves a few percent of runtime but produces stacks nobody can reconstruct is a bad trade.

ASan also stops on the first detected error by default. That behavior is worth preserving during investigation.

Once the program has crossed into invalid memory state, later failures become less trustworthy. One stale write can corrupt a neighboring object, which changes a branch, which causes a second invalid access somewhere unrelated. Continuing may give you more reports, but not necessarily more independent bugs.

The first report is often the best evidence you'll get because the allocation, lifetime transition, and bad access are still relatively close to the original mistake.

ASan is powerful, but its result still has boundaries. Fully static linking isn't the normal supported model. On 64-bit systems, ASan reserves a large virtual address range for its shadow mapping. Partial instrumentation weakens what the runtime can see. Custom containers and allocators can hide logical object boundaries. The compact shadow scheme also has known edge cases around some partially out-of-bounds unaligned accesses.

None of those limitations make ASan weak. They simply define what a clean ASan run means: the accesses exercised by this workload didn't cross an addressability boundary that the instrumented runtime could observe.

How ASan shadow memory works

ASan gets much of its speed from using a compact shadow representation rather than maintaining a heavyweight metadata object for every allocation.

In the common mapping, one shadow byte describes eight bytes of application memory. A shadow value of zero means all eight corresponding bytes are addressable. Values from one through seven can represent a partially addressable tail, encoding how many leading bytes remain valid. Other non-zero values are used as poison markers for regions such as redzones, freed memory, or invalid stack lifetimes.

The diagram shows how ASan maps application memory to shadow memory so it can quickly decide whether a load or store touches valid, partially valid, or poisoned memory.

Before an instrumented load or store, compiler-generated code maps the application address to the corresponding shadow byte and checks whether the requested access is valid.

For ordinary heap allocations, the allocator places poisoned redzones around the object. When the object is freed, its memory becomes poisoned as well and is commonly held in quarantine for a while rather than being returned immediately for reuse.

That delay matters. Without it, a stale pointer can quickly start referring to a new, valid object at the same address. Keeping the region poisoned gives the old pointer more time to fail as a use-after-free.

Stack instrumentation uses the same general mechanism around local variables. Regions can be poisoned when a scope ends, and additional instrumentation can preserve poisoned state after a function returns to catch stack lifetime violations that would otherwise depend on whether the old frame happened to be reused.

The shadow model explains both ASan's speed and some of its blind spots.

ASan is very good when the invalid access crosses a boundary the compiler and runtime know about. It's less useful when an access happens entirely inside uninstrumented code, when inline assembly bypasses compiler-generated checks, or when an allocator presents several logical objects as one large addressable region.

There's also an important distinction between a logical ownership bug and an addressability violation. ASan doesn't know that your API says a pointer was transferred. It knows that memory eventually became poisoned and somebody touched it afterward. If the ownership model is wrong but no illegal access occurs during the run, ASan has nothing to report.

Read the first report as a causal narrative

An ASan report is usually easier to understand if you stop treating the top frame as the bug.

For a typical heap use-after-free, there are at least three points worth reconstructing:

• the access that finally touched invalid memory

• the allocation that created the object

• the deallocation that ended its lifetime

Depending on the report, you may also get shadow-memory context and thread-creation history.

Those stacks describe different beliefs about the same object.

The allocation stack tells you where the lifetime began. The free stack tells you where one part of the program decided that lifetime was over. The current access tells you who still believed the object was usable.

For example:

allocate native object
    -> hand pointer to wrapper
    -> transfer ownership elsewhere
    -> old cleanup path still runs
    -> new owner dereferences pointer
    -> ASan reports use-after-free

The dereference is where ASan notices the problem. The broken transition happened earlier.

That distinction matters in real fixes. Adding a null check around the victim access may suppress one crash while leaving the stale ownership state intact. The better repair is usually at the point where one owner stopped being an owner but the program failed to represent that change.

Custom allocators and interceptors

ASan works best when its view of allocation boundaries matches the boundaries the application cares about.

That's straightforward with ordinary malloc, free, new, and delete. It becomes more complicated with arenas, slabs, pools, region allocators, JIT heaps, placement construction, and library-specific allocators.

Consider a pool that requests a 1 MiB backing region from the system and then subdivides it into hundreds of 64-byte objects. From the system allocator's point of view, that may be one live allocation. Unless the pool cooperates with ASan, an overwrite from one 64-byte object into the next can remain entirely inside addressable backing memory.

The program has crossed a logical object boundary. ASan may have no corresponding poisoned boundary to catch.

Sanitizer-aware allocators usually handle this in one of a few ways. They can explicitly poison unused slots or gaps, annotate container boundaries through sanitizer APIs, or offer a test mode that delegates individual allocations to the system allocator so ASan can see them independently.

Workload design can help as well. Aggressive slot reuse makes stale references more likely to collide with changed lifetime state. Generation counters can expose stale handles at the allocator level. Repeated allocate/free/reallocate sequences are often more useful for lifetime testing than steady-state workloads that leave the same objects alive for minutes.

The important part is that allocator visibility isn't automatic.

If your allocator deliberately hides the application's object boundaries, enabling -fsanitize=address at the top of the build can't reconstruct those boundaries afterward. The allocator either has to expose them to ASan or the test environment has to use a configuration in which the boundaries become observable.

Interceptors have a similar role at common library boundaries. They allow the sanitizer runtime to understand operations that would otherwise happen inside library code. But interceptors aren't a substitute for complete instrumentation. They extend ASan's field of view. They don't make opaque code transparent.

Suppressions are not a repair strategy

Suppressions are sometimes necessary.

A report may originate in a third-party library you can't patch immediately. A platform component may already have an upstream fix that hasn't reached your supported environment. There may also be cases where the sanitizer runtime and a dependency interact badly enough that the lane can't remain usable without a temporary exception.

In those situations, suppress the specific problem, not the neighborhood around it.

A broad suppression is dangerous because it changes what a green sanitizer run means. Matching an entire library, namespace, or call path can hide a new and unrelated regression months after the original reason for the suppression has been forgotten.

For code you own, suppression should be unusual. If the report is real and actionable, hiding it because the fix is inconvenient defeats the reason for running ASan in the first place.

When a suppression is necessary, keep enough information beside it to understand why it exists: the narrowest stable match you can use, the underlying issue, the component responsible for resolving it, and some point at which the exception will be reviewed again.

Keep the lane sensitive to new failures, and treat every exception as a narrow, documented gap in what the current build can observe.

LeakSanitizer (LSan): Terminal Ownership

Leaks have a different failure shape from use-after-free or buffer overflows. Nothing has to crash. The operation can return the right result, every test can pass, and the process can shut down normally while still leaving allocations behind.

Somewhere in that execution, ownership simply never reached its end.

LeakSanitizer is built around that point in the lifecycle. ASan watches accesses as the program runs. LSan does most of its interesting work when the process is shutting down, when it can inspect what heap allocations remain and how they're connected to live roots.

That makes leak detection especially useful for code where cleanup is easy to miss without affecting the immediate result: FFI wrappers, partial-construction failures, teardown paths, caches, reference-counted objects, and systems that rely partly on finalization.

Clang can run LSan as part of an ASan build, or independently with -fsanitize=leak. Most of the expensive work happens during leak checking rather than on every ordinary memory access, so the runtime overhead during the main body of the program is generally small.

The useful mental model is terminal ownership accounting. By the time the process reaches teardown, every allocation should either have been released or still have some legitimate reason to remain reachable. LSan reconstructs the second category from the runtime state it can inspect.

That distinction matters because many ownership bugs never become invalid accesses. A retained reference may simply survive forever. An error path may forget one cleanup call. A global or cache may accidentally keep an object graph alive. The application keeps working, the ownership path just never closes.

If you already have a working ASan configuration, enabling leak detection there is usually the most practical starting point. Stand-alone LSan is available and can be useful when full ASan instrumentation is undesirable or difficult to integrate, although it has historically received less testing than the ASan-integrated path.

A basic ASan-backed leak lane looks like this:

clang -O1 -g -fsanitize=address -fno-omit-frame-pointer \
  -o leak_suite leak_suite.c

ASAN_SYMBOLIZER_PATH="$(command -v llvm-symbolizer)" \
ASAN_OPTIONS="detect_leaks=1" \
LSAN_OPTIONS="exitcode=23:suppressions=lsan.supp" \
./leak_suite

If you want leak-only coverage:

clang -O1 -g -fsanitize=leak -o leak_suite leak_suite.c
LSAN_OPTIONS="exitcode=23:suppressions=lsan.supp" ./leak_suite

The explicit exit code is useful in CI because it turns a leak report into an ordinary job failure. Suppressions let you keep known external findings from drowning out the code you are trying to measure, although they carry the same cost discussed elsewhere in this handbook: every suppression removes some amount of visibility.

A leak lane becomes valuable when its result has a clear interpretation. If the test allocates a resource, exercises its lifetime, releases everything it still owns, and exits, then an unexpected allocation at teardown is a fairly direct ownership signal. The larger and noisier the process, the more care that interpretation requires.

Reachability is not the same as intention

LSan is conservative because it has no direct knowledge of your ownership design. At shutdown, it inspects root-like regions such as thread stacks, registers, globals, and thread-local storage, then follows pointer-looking values into heap allocations. If an allocation remains reachable through that graph, LSan may leave it alone even when the application logically intended to release it.

The diagram shows how LSan traces reachability at shutdown. Unreachable allocations become leak reports, but reachable memory may still be logically wrong if the program meant to release it earlier.

This is one of the most important differences between leak detection and ordinary lifetime debugging.

Suppose an object should have been released during teardown, but a stale pointer to it remains in a global structure. From the program's point of view, cleanup is broken. From LSan's point of view, the allocation is still reachable.

The opposite case is easier to diagnose. If an allocation has no path back to any root LSan recognizes, it can report it as leaked and show the allocation stack that created it.

Direct and indirect leaks are worth reading together. A missing release for one parent object can leave an entire graph unreachable. Fixing that parent may eliminate many downstream leak records at once. Chasing every indirect allocation independently wastes time when they all share the same lost owner.

Reachability can also be intentional. Some runtimes keep process-lifetime caches. Libraries may initialize global state once and never tear it down because process exit will reclaim the pages anyway. Test frameworks and managed runtimes can leave allocations visible during shutdown that have little to do with the component under investigation.

That is why whole-application leak reports are often harder to interpret than focused ones.

For a library or FFI wrapper, a small helper process can be much more informative:

initialize runtime
    -> create native resources
    -> exercise success and failure paths
    -> release explicit owners
    -> allow deferred cleanup where relevant
    -> exit

The shorter that lifecycle is, the easier it becomes to answer the useful question: which allocations should still exist at this point?

Finalizer-driven code deserves extra care here. A managed object becoming unreachable doesn't necessarily mean its native resource has already been finalized by the time the process begins leak checking. If the test depends on eventual cleanup, the harness needs to account for the runtime's finalization behavior rather than treating process exit as a deterministic destructor call.

The same problem appears with asynchronous cleanup, worker threads, and background caches. A leak test that terminates before normal teardown completes can report memory that the real application would have released a moment later.

Those are workload problems rather than reasons to ignore LSan. The test needs to reach the state whose ownership you actually want to measure.

Bring the leak lane online gradually

Adding leak detection to an established codebase often produces an ugly first report.

Some of the findings will be real leaks in code you own. Others may come from dependencies, runtime shutdown behavior, process-lifetime caches, incomplete cleanup in the test harness, or allocations whose reachability needs more investigation.

Trying to make all of that blocking immediately usually produces one of two outcomes: a pile of broad suppressions or a CI lane that everyone learns to rerun and ignore.

Start by making the reports reproducible.

A deterministic direct leak in owned code is usually straightforward: fix it and keep the reproducer. For runtime or dependency findings, establish whether they're stable, whether they're actually reachable at shutdown, and whether a smaller process can separate them from the code under test.

Suppress only the cases you have enough evidence to understand, and keep those matches narrow.

As the baseline gets cleaner, tighten the failure policy. New direct leaks in a focused test can become blocking quickly. Findings from a managed runtime or a large host process may need more attribution before they deserve the same treatment.

This is also why leak severity and leak correctness should remain separate judgments. A four-byte allocation that leaks once during process startup may have negligible operational impact, but it still tells you that one ownership path doesn't terminate correctly. Whether it blocks a release depends on policy. Whether the ownership accounting is wrong is a different question.

A useful leak lane eventually reaches the point where a new report is surprising. When that happens, LSan stops being a periodic cleanup exercise and becomes a regression detector for ownership paths that previously had no executable check.

MemorySanitizer (MSan): Initialization Provenance

ASan catches code using memory it should no longer be able to touch. MSan catches a different failure: the memory is perfectly addressable, but the value being read was never legitimately established.

That distinction matters because uninitialized data doesn't have to cause an immediate crash. It can become a length, a flag, a pointer offset, a branch condition, or an output returned to another layer. The program may carry that value for quite a while before it reaches an operation MSan considers observable.

This makes MSan particularly good at finding disagreements across API boundaries.

A caller sees a successful return and assumes every output promised by the API is ready to use. The callee has one branch that returns success without writing one of those outputs. Both pieces of code can look reasonable in isolation. The problem only becomes obvious when the caller consumes state that never acquired a valid initialization history.

Mozilla's NSS issue 1767590 is a useful example. An output pointer was treated as initialized after a successful call even though one path through the callee could return success without writing it. MSan eventually reported the uninitialized value, but the more interesting defect was the API contract: the meaning of “success” was stronger in the caller than in the implementation.

That's a recurring MSan pattern. The report appears at the use site, while the bug often began earlier when some path failed to establish the state the caller was entitled to expect.

A basic MSan build looks like this:

clang++ -O1 -g -fno-omit-frame-pointer \
  -fsanitize=memory -fsanitize-memory-track-origins=2 \
  -o msan_suite msan_suite.cc

MSAN_SYMBOLIZER_PATH="$(command -v llvm-symbolizer)" \
./msan_suite

-fsanitize=memory enables the instrumentation. -fsanitize-memory-track-origins=2 asks MSan to retain more information about where poisoned state came from and how it moved through memory.

Origin tracking is expensive. Regular MSan already carries substantial runtime and memory overhead, and tracking origins adds more. For difficult bugs, though, the extra information is often worth paying for.

Without origin information, a report may tell you that an uninitialized length reached a branch deep inside a decoder. With origins enabled, you may be able to trace that value back through several stores to an output structure allocated by the caller, then see that one recovery path never initialized a member.

That's the difference between knowing where bad state became visible and knowing where it entered the execution.

MSan works especially well around parsers, codecs, binary formats, cryptographic message processing, serialization, IPC, database pages, and APIs with output parameters. These systems routinely construct objects incrementally and pass partially populated state across several layers. An initialization mistake can survive a surprising distance before ordinary testing notices anything unusual.

What MSan actually propagates

Calling uninitialized memory “random bytes” is a useful shorthand, but it misses the mechanism that makes MSan interesting.

MSan maintains shadow state describing which bits of a value are initialized. Copies propagate that state. Arithmetic and other operations propagate it into their results. As values move through the program, the initialization history moves with them.

The diagram shows how MSan follows uninitialized state through the program and reports it only when the value is used in a way that affects execution.

Poisoned state can therefore survive several apparently harmless operations before anything is reported.

For example:

stack object allocated
    -> one field left unwritten
    -> structure copied
    -> field copied into a size variable
    -> size used in a conditional
    -> MSan reports uninitialized use

The branch is simply where the invalid state became semantically important. Fixing only that branch misses the earlier path that produced a partially initialized object.

The same applies to function boundaries. If a value is returned to a caller or passed into code that requires it to be initialized, MSan can report the transition even though the underlying memory address was valid the whole time.

This is why MSan reports often feel very different from ASan reports. With ASan, you're usually reconstructing the lifetime of an address. With MSan, you're reconstructing the history of a value.

Origin tracking adds another layer to that history. At the higher tracking level, MSan can retain information about where poisoned memory originated and where it was subsequently stored. That can be noisy and expensive, but it's extremely useful when the observable use is several abstraction layers away from the missed initialization.

MSan needs visibility through the dependency graph

MSan becomes much harder to trust when important parts of the execution aren't instrumented.

Suppose an uninstrumented library writes into a buffer. The bytes may be completely valid from the application's point of view, but MSan didn't observe the writes that initialized them. When instrumented code later consumes the buffer, the shadow state may still say that those bytes are poisoned.

The reverse problem also exists. Poisoned data can pass into uninstrumented code and be consumed there without MSan seeing the use.

Interceptors help at common library boundaries, but they don't make arbitrary opaque code visible. If initialization state crosses through a dependency that MSan doesn't understand, confidence in the resulting report drops.

For high-quality MSan coverage, the application and the libraries through which important state flows generally need to be rebuilt with compatible instrumentation. Depending on the environment, that can extend into the C/C++ runtime and other low-level dependencies.

This is why MSan often becomes its own build environment rather than another checkbox in the normal sanitizer matrix.

Projects with mostly source-built dependencies can make that environment quite complete. Projects that depend heavily on prebuilt native libraries, inline assembly, proprietary runtimes, or system components they can't rebuild have a harder time getting clean results.

That operational cost is real. So is the class of bug MSan finds. For code that parses attacker-controlled input or relies heavily on partially constructed outputs, the investment can be worthwhile even when the lane only covers a carefully selected part of the system.

Make success postconditions explicit

MSan has a habit of finding APIs where the return value says more than the implementation actually guarantees.

Consider a function like:

bool decode(const uint8_t* input, size_t input_len,
            uint8_t* output, size_t* output_len);

A caller will naturally read true as a postcondition: output contains a valid result and *output_len says how much of it is valid.

If some successful branch writes output but leaves *output_len untouched, the implementation has created a state the API doesn't communicate.

The same problem appears with structures:

result.status = OK
result.data   = initialized
result.length = never written

Nothing about status == OK warns the caller that one field is unsafe to inspect.

Several designs avoid this ambiguity. Outputs can be initialized to valid defaults before control flow branches. A function can return a fully constructed result object rather than writing through several independent pointers. APIs that genuinely support complete, partial, and failed results can represent those states explicitly instead of overloading a single success code. Buffers with partial contents can return their valid length as part of the result rather than relying on an optionally written side channel.

Which design is appropriate depends on the API. The important property is that callers should be able to tell, from the interface and the return state, which values they're allowed to consume.

When MSan catches a missing initialization, adding memset(..., 0, ...) may be a correct fix in some cases. In others it merely turns undefined state into a plausible-looking default while leaving the API contract ambiguous.

The report is worth following back far enough to answer the more useful question: which path allowed the caller to believe this value was ready when it was not?

ThreadSanitizer (TSan): Synchronization Visibility

ThreadSanitizer observes a part of execution that ordinary tests are particularly bad at validating: the ordering between accesses made by different threads.

A program can run successfully thousands of times with an actual data race simply because the scheduler keeps choosing harmless interleavings. TSan instruments memory accesses and synchronization operations so it can reconstruct enough of the execution to identify conflicting accesses that occurred without a valid ordering relationship.

That visibility comes at a substantial cost. TSan commonly slows programs by roughly 5–15x and can consume several times their normal memory. It also needs broad compiler instrumentation. The build machinery may make additional changes, such as producing position-independent executables where required by the runtime.

Those constraints are why TSan usually lives in its own build rather than being added casually to the normal test binary.

A typical setup is straightforward:

clang++ -O1 -g -fsanitize=thread 
-o tsan_suite tsan_suite.cc

TSAN_OPTIONS="halt_on_error=1:history_size=7" 
./tsan_suite

The compiler flag instruments the program for race detection. halt_on_error=1 makes the first reported race fail the run, while a larger history can give the report more context about earlier accesses.

As with the other sanitizers, useful symbols matter more than squeezing maximum performance out of the instrumented build. TSan reports often involve two stacks, multiple threads, and the synchronization events around them. Losing a caller or thread-creation stack to an overly aggressive build can make an already difficult race much harder to reconstruct.

The bigger operational problem is visibility across the rest of the process.

TSan needs to observe the synchronization that makes shared-memory accesses safe. That works well when locks, atomics, condition variables, thread creation, and other synchronization primitives pass through code the runtime understands. It becomes less reliable when part of the ordering happens inside prebuilt libraries, custom scheduler code, assembly, native callbacks, or runtime machinery that wasn't instrumented.

A report near one of those boundaries needs context before anybody labels it a false positive.

Mozilla's Bugzilla issue 1688716 is a useful example. The report involved graphics code where part of the relevant execution passed through non-instrumented code. The discussion had to account for the possibility that synchronization existed outside TSan's view.

That's a very different situation from two unsynchronized accesses in fully instrumented code you own.

The opposite blind spot matters too. If the conflicting accesses themselves happen inside uninstrumented code, TSan may never see the race at all. Partial instrumentation can therefore create both confusing reports at boundaries and genuine false negatives inside opaque modules.

For a serious TSan lane, it's worth knowing which parts of the process are instrumented before the first report arrives.

TSan also benefits much more than ASan from workloads designed specifically for the property being tested. A conventional unit suite may exercise the right functions while consistently producing the same thread schedules. Race detection needs repetition, contention, and overlap.

That affects where the lane belongs. A short TSan smoke test may still be useful on pull requests, but expensive concurrency workloads are often better suited to scheduled runs, targeted changes, or pre-release testing. The CI section later in the handbook goes into that tradeoff in more detail.

Happens-before is the working mental model

TSan doesn't simply look for “two threads touched the same variable.”

It records memory accesses and synchronization events, then asks whether conflicting accesses are ordered by a happens-before relationship. If two threads touch the same memory, at least one access writes, and no observable synchronization orders those operations, TSan has the shape of a data race.

The diagram shows how synchronization creates the ordering TSan needs to see. Without that observable ordering, conflicting accesses across threads can become a TSan report.

A mutex creates ordering that TSan understands. An unlock in one thread followed by a lock in another can establish the required edge. Correct acquire/release atomics can do the same. Thread creation and join, condition variables, and other recognized synchronization primitives also contribute to the happens-before graph.

For example, consider a simple publication pattern:

writer:
    initialize object
    -> release/store ready flag

reader:
    acquire/load ready flag
    -> read object

If the release/acquire pair is implemented correctly and visible to TSan, the writes that initialized the object can be ordered before the reader consumes it.

Replace that protocol with an ordinary unsynchronized boolean and the program may still appear to work on a particular machine. TSan no longer has an ordering edge connecting the accesses.

The same problem appears with custom synchronization. A home-grown lock in assembly may be perfectly correct at the hardware level while remaining invisible to the sanitizer. From TSan's point of view, the accesses on either side can look unordered because the event that connects them never entered its model.

That's why a TSan report has two possible debugging directions. Sometimes the synchronization is genuinely missing. Sometimes the synchronization exists but TSan can't observe it. The report itself gives you the conflicting accesses, while understanding the instrumentation boundary tells you which situation you are dealing with.

Be skeptical of “benign races”

“Benign race” is a dangerous phrase because it usually describes observed behavior rather than a defined synchronization rule.

A field may appear harmless because every value seen so far is acceptable. That still leaves several problems.

In C and C++, an actual data race generally puts the program into undefined-behavior territory. Compiler transformations aren't constrained by the timing assumptions that made the code appear safe during testing.

The raced field may also participate in a larger invariant. An unsynchronized flag can look harmless while another field is expected to change with it. Seeing one update without the other can expose a state the programmer never intended to exist.

Maintenance makes the argument weaker still. A value that genuinely did not matter when the race was introduced may later become part of a lifetime decision, callback protocol, or security check while the old unsynchronized access remains untouched.

There are legitimate cases where a TSan report comes from instrumentation gaps or from synchronization the runtime can't see. Those deserve investigation at the boundary.

When both conflicting accesses are in instrumented code you own and no ordering mechanism connects them, “benign” should require a much stronger explanation than “we've never seen it break.”

Often the eventual repair is simple: use an atomic with the appropriate memory ordering, put the state behind the existing mutex, or redesign the object so the conflicting accesses no longer occur concurrently.

The harder part is identifying which ordering the code was relying on before the report exposed that it was missing.

Make the race easier to schedule

TSan only reports races that happen during the observed run.

A million lines of concurrency code can pass cleanly if the workload never produces the relevant overlap. Conversely, a focused test that repeatedly attacks one lifecycle boundary can expose a race in seconds.

Useful concurrency probes deliberately increase the probability of the schedules you care about.

If two operations should be safe when they begin at nearly the same time, start the threads from a barrier. If a race may exist between callback completion and object destruction, repeat that transition thousands of times. If a registry supports concurrent lookup and removal, hammer both operations from several workers.

Test-only yields can be useful around narrow windows:

thread A:
    read state
    -> yield
    -> update shared object

thread B:
    close object
    -> release shared state

The yield doesn't create the bug. It gives an already legal scheduler interleaving more room to occur.

Similar techniques work for callback registration and cancellation, attach/detach cycles, shutdown while work is still completing, and races between explicit cleanup and background teardown.

Randomization can help explore a larger scheduling space, but record the seed and the operation sequence. A TSan report that occurs once overnight and can't be reproduced locally is much more expensive to investigate than one whose workload can be replayed.

The workload should remain faithful to behavior the real system can produce. Artificial contention is useful, but impossible object lifetimes are not. The test should make an existing race easier to schedule rather than inventing a concurrency model the application never uses.

A good TSan workload therefore spends its time around transitions: publish and consume, register and unregister, start and stop, close and complete, and retain and release. Those are the places where two threads are most likely to disagree about what state is currently safe to touch.

Combining Sanitizers and Knowing What They Do Not Cover

There's no useful “enable everything” sanitizer build.

The tools maintain different runtime metadata, impose different instrumentation requirements, and often need very different workloads. Some combinations fit naturally into the same binary. Others should be treated as separate builds with their own dependency graphs and CI budgets.

Understanding those boundaries matters because a large sanitizer matrix can still give weak evidence if the combinations are poorly chosen or important parts of the process remain opaque.

Combinations That Work Well Together

ASan and UndefinedBehaviorSanitizer are commonly enabled together. Their checks cover different parts of the execution and coexist well in many Clang builds:

clang++ -O1 -g -fno-omit-frame-pointer \
  -fsanitize=address,undefined \
  -fno-sanitize-recover=all \
  app.cc -o app_sanitized

ASan watches addressability and lifetime. UBSan catches language-level undefined behavior such as invalid shifts, misaligned accesses, invalid enum values, and enabled integer-overflow checks.

UBSan isn't a major subject of this handbook, but it belongs in this discussion because the eventual memory failure may begin somewhere that ASan doesn't model.

An invalid size calculation can lead to an undersized allocation. A bad cast can send execution through an impossible type state. Misaligned access or arithmetic overflow can corrupt assumptions long before the program crosses a poisoned ASan boundary. In those cases, the UBSan finding may be closer to the original defect than the ASan crash that follows it.

Recovery policy is worth choosing deliberately. Letting UBSan continue can expose several categories during one run, which is useful during broad discovery. Failing on the first finding keeps the execution closer to the first known invalid state and is often easier to reason about in a regression lane.

Leak detection also commonly shares the ASan build. On supported platforms, LSan can run through the ASan runtime and perform its reachability analysis when the process exits. That gives the same binary both access-violation coverage during execution and leak accounting at teardown.

Standalone LSan remains useful when leak checking is needed without the rest of the ASan instrumentation. The choice is mostly operational: if a reliable ASan build already exists, using its leak support usually keeps the number of distinct build configurations smaller.

TSan and MSan Deserve Separate Builds

TSan belongs in a different binary from ASan.

Both tools instrument memory accesses heavily and maintain runtime state for very different purposes. TSan needs access histories and synchronization metadata while ASan needs shadow addressability state and allocator integration. In practice, they should be treated as separate observability environments rather than variations of one sanitizer job.

That separation is useful anyway because the workloads should differ.

An ASan lane may get excellent value from parser tests, lifecycle stress, and failure-path coverage. A TSan lane needs concurrency: overlapping operations, repeated thread transitions, callback races, shutdown while work is active, and enough repetition for interesting schedules to occur.

Its performance budget is different too. A workload that's perfectly reasonable under ASan may be far too expensive under TSan.

MSan needs its own environment for another reason: initialization provenance is only trustworthy when enough of the code carrying that state is instrumented.

Adding -fsanitize=memory to the application while leaving important native dependencies opaque can break the initialization history MSan is trying to follow. Serious MSan setups tend to look like instrumented dependency graphs rather than ordinary application builds with one extra compiler flag.

That makes the build matrix asymmetric, and that's fine.

A project might run ASan+UBSan on most changes, leak detection as part of the same binary, TSan against a smaller concurrency-heavy workload, and MSan only in an environment where the relevant dependencies can be rebuilt correctly. The useful question is what each lane can observe reliably, not whether every tool appears in the same CI stage.

Hardware-Assisted and Production-Side Checks

Software ASan is only one way to observe lifetime and bounds failures.

Hardware-assisted AddressSanitizer and Arm Memory Tagging Extension use tagged memory and pointers to detect mismatches between the pointer being used and the allocation it is supposed to reference. On supported 64-bit Arm systems, that can make memory checking practical in environments where the overhead of conventional ASan would be difficult to carry.

The tradeoffs are different from software ASan.

The tag space is finite, so stale pointers can occasionally collide with a valid tag and escape detection. Platform support determines where the checks can run. The exact treatment of stack, global, and allocator behavior also differs from conventional ASan.

What these mechanisms buy is access to executions that ordinary sanitizer CI may never see: longer-running workloads, device-specific behavior, realistic allocator pressure, and in some cases production or near-production traffic.

Guarded allocators, hardened allocators, sampled memory checking, crash telemetry, and canary deployments fit into the same broader strategy. Each observes a different slice of runtime behavior at a cost that may be acceptable outside a fully instrumented test environment.

Their findings should feed back into the deterministic test setup whenever possible. A sampled production use-after-free becomes much more valuable once its lifecycle can be reproduced under ASan. A long-running memory-growth pattern can become a focused LSan workload. A concurrency failure observed only under load can become a TSan stress case.

Production-side checking expands the executions you get to observe. The compiler sanitizer builds remain the place where those failures can usually be reproduced with richer instrumentation and tighter control.

What a Green Sanitizer Matrix Actually Tells You

A sanitizer matrix answers a narrower question than “is this program correct?”

It tells you what happened during the executions the instrumented runtime was able to observe.

That leaves a lot outside the claim.

Business logic can be wrong while every sanitizer stays quiet. Authorization can be incomplete. A cryptographic protocol can be misused. Timing and other side channels can exist in completely memory-safe code. A lock-free algorithm can be race-free and still fail its intended linearizability guarantees.

Even within memory safety and concurrency, workload coverage remains a hard boundary. Code that never executes can't produce a dynamic report.

Instrumentation creates another boundary. ASan can't enforce object boundaries hidden inside an allocator it doesn't understand. MSan loses initialization history across opaque code. TSan can't reconstruct synchronization that never becomes visible to its runtime. A third-party module that was never instrumented may contain bugs none of the surrounding sanitizer jobs can see.

Suppressions narrow the claim further. A green run means the runtime found no unsuppressed violation it knew how to report. Every suppression therefore changes the meaning of “clean.”

It helps to be precise about the claim attached to a successful run:

For this build, under this workload, the instrumented execution did not violate the runtime properties this sanitizer was able to observe.

That may sound modest, but it's a useful engineering statement. It tells you exactly where the remaining work belongs.

If the workload is weak, improve the workload. If a dependency is opaque, instrument more of the dependency graph where practical. If one class of failure falls outside ASan's model, add the tool that can observe it. If the remaining uncertainty comes from API design rather than runtime coverage, static analysis, code review, stronger ownership types, or interface changes may provide better leverage than another sanitizer job.

A good sanitizer setup grows by closing known gaps in evidence, not by accumulating compiler flags.

FFI Ownership Across Runtime Boundaries

FFI bugs tend to repeat because the boundary has to reconcile two different lifetime models. The managed side may use garbage collection and finalizers, while the native side may use explicit frees, reference counts, arenas, or ownership-transfer APIs.

The failure is rarely “FFI” in the abstract. More often, the two sides simply disagree about who owns a resource at a particular moment.

One recurring failure is ownership transfer without state transfer. A wrapper creates or receives a native handle, passes ownership somewhere else, but continues to represent that handle as usable. Nothing about the pointer itself says that ownership moved. Unless the wrapper changes state as part of the transfer, the old owner can still free it, attach cleanup to it, or use it later.

This is where APIs that distinguish borrowing, retaining, and transferring ownership become valuable. BoringSSL is a particularly explicit example: conventions such as get0, get1, set0, and set1 communicate whether a reference is borrowed, owned, adopted, or retained. A wrapper that reduces all of those cases to the same pointer representation loses information the native API was deliberately exposing. That's how borrowed handles get freed, transferred handles get reused, and owning references quietly leak.

Another common failure is losing the release mechanism. Knowing that you own an allocation isn't enough. You also need to know how that allocation must be released. Native libraries may pair allocations with a specific free function, return reference-counted objects that require a decrement operation, or hand back memory owned by an arena rather than by the caller.

BoringSSL again gives a simple example: memory allocated through OPENSSL_malloc is expected to be released through OPENSSL_free. If a wrapper treats that memory as interchangeable with memory from another allocator, the mistake may remain invisible until teardown or until ASan, Valgrind, or another memory checker reaches the mismatched release.

The broader rule is that an owning wrapper should preserve the resource's destruction semantics, not merely its address.

A third failure comes from treating finalization as destruction. A finalizer can eventually release a native resource, but it doesn't give you deterministic lifetime. Its timing depends on runtime reachability and garbage collection, not on the lexical point where the program stopped needing the object.

That distinction matters whenever the native resource has a meaningful lifetime of its own. File descriptors, native handles, cryptographic contexts, GPU objects, database handles, and similar resources often need explicit release semantics. A finalizer is useful as a fallback, but it shouldn't quietly become a second independent owner.

Transfer makes this especially dangerous. If a managed object hands its native handle to another owner but leaves its finalizer attached, both sides may now believe they're responsible for cleanup. The wrapper should invalidate its own state as part of the move: detach the finalizer, clear the handle, or otherwise make subsequent use fail immediately.

Failure paths create another class of bug. Code can be perfectly correct on the successful path and still have no coherent ownership story when an operation fails halfway through.

Suppose a function allocates three temporary native objects and transfers one of them only if an import succeeds. The failure path must release everything that was never transferred. The success path must release the remaining temporaries without also releasing the object whose ownership moved. Those are separate questions: does cleanup run on every exit, and which objects still belong to that cleanup path when it runs?

This is why cleanup scopes, RAII guards, defer-style mechanisms, and similar patterns are useful. They make cleanup the default, then require successful ownership transfers to explicitly remove an object from that cleanup responsibility. Without that distinction, the same code can leak on failure and double-free on success.

A small C++ example shows what it looks like when those states are represented explicitly. BoringSSL provides the concrete API here, but the pattern applies to any reference-counted native handle:

#include <openssl/evp.h>
#include <memory>
#include <stdexcept> 

struct PKeyDeleter { 
    void operator()(EVP_PKEY* p)
    const noexcept { EVP_PKEY_free(p); } 
};

using UniquePKey = std::unique_ptr<EVP_PKEY, PKeyDeleter>; 

class KeySlot { 
    public: 
        // The slot owns exactly one reference. 
        explicit KeySlot(UniquePKey key) : key_(std::move(key)) {} 

        // Borrow without changing ownership. 
        EVP_PKEY* borrow() const noexcept { 
            return key_.get();
        } 

        // Transfer ownership out. The slot becomes empty.
        UniquePKey take() {
            return std::move(key_); 
        } 
        
        // Create another owning reference for shared use.
        UniquePKey clone_ref() const {
            if (!key_) { 
                throw std::logic_error("empty slot"); 
            }

            if (EVP_PKEY_up_ref(key_.get()) != 1) { 
                throw std::runtime_error("EVP_PKEY_up_ref failed"); 
            } 
            return UniquePKey(key_.get()); 
        } 
        bool empty() const noexcept {
            return key_ == nullptr;
        }

    private: UniquePKey key_; 
};

The useful part is that the different ownership operations no longer look identical. borrow() exposes the handle without creating another owner. take() moves the owning reference and leaves the source empty. clone_ref() explicitly creates another owning reference.

The type system can't prevent every lifetime bug, but it can make ownership transitions much harder to perform accidentally. That gives ASan and LSan a cleaner model to test as well: a stale access or leaked reference is more likely to point back to a specific broken transition rather than to a sea of indistinguishable raw pointers.

Managed FFI code often has to encode the same states more explicitly because the language's normal object lifetime doesn't automatically describe the native object's lifetime. A wrapper can still make the distinction visible:

import 'dart:ffi' as ffi;

final class NativeHandle extends ffi.Opaque {}

final class OwnedNativeHandle {
    OwnedNativeHandle(this._ptr, this._free, this._finalizer)
        : _token = Object() {
            _finalizer.attach(this, _ptr.cast(), detach: _token);
    }

    ffi.Pointer<NativeHandle>? _ptr;
    final void Function(ffi.Pointer<NativeHandle>) _free;
    final ffi.NativeFinalizer _finalizer;
    final Object _token;

    bool get isMoved => _ptr == null;

    ffi.Pointer<NativeHandle> borrow() {
        final ptr = _ptr;
        
        if (ptr == null) {
            throw StateError('Native handle is no longer owned here');
        }

        return ptr;
    }

    ffi.Pointer<NativeHandle> move() {
        final ptr = borrow();
        
        _finalizer.detach(_token);
        _ptr = null;

        return ptr;
    }

    void close() {
        final ptr = _ptr;
        
        if (ptr == null) return;

        _finalizer.detach(_token);
        _ptr = null;
        _free(ptr);
    }
}

Here, move() is more than a pointer return. It changes the wrapper's state. Once ownership leaves, the previous owner can't borrow the handle again, and its finalizer no longer has permission to release it. close() follows the same rule and is safe to call more than once because the ownership state changes before the native free occurs.

The exact representation will differ by language and runtime. Some systems use nullable handles, some use dedicated moved states, some hide the pointer behind an owning object, and others rely on linear or affine types. What matters is that a transfer changes the old owner's state in a way the rest of the program can observe.

Temporary ownership needs the same treatment. A lexical cleanup scope can make “release unless transferred” explicit:

final class Scope {
    final Map<Object, void Function()> _cleanup = {};

    T own<T extends Object>(T handle, void Function(T) dispose) {
        if (_cleanup.containsKey(handle)) {
            throw StateError('Handle is already owned by this scope');
        }

        _cleanup[handle] = () => dispose(handle);
        return handle;
    }

    T move<T extends Object>(T handle) {
        final cleanup = _cleanup.remove(handle);
        if (cleanup == null) {
            throw StateError('Cannot move a handle this scope does not own');
        }
        return handle;
    }

    void close() {
        Object? firstError;
        
        for (final cleanup in _cleanup.values.toList().reversed) {
            try {
                cleanup();
            } catch (error) {
                firstError ??= error;
            }
        }

        _cleanup.clear();

        if (firstError != null) {
            throw StateError('Scope cleanup failed: $firstError');
        }
    }
}

The scope owns everything registered with it until one of two things happens: cleanup runs, or ownership is explicitly moved somewhere else. That makes early returns and partial failures much easier to reason about because the default behavior is cleanup rather than “remember every object that was successfully allocated so far.”

This is also where the different sanitizer models start to line up around the same ownership story. If a transferred object is still released by its previous owner, ASan may eventually catch the resulting use-after-free or invalid access. If an error path forgets to release something it still owns, LSan or a leak-oriented lane may expose it at teardown. If a native call reports success without fully initializing an output structure, MSan can reveal a different boundary-contract failure. If close, callback, or cleanup paths race across threads, TSan can expose the missing synchronization when those operations are visible to its runtime.

The tools report different symptoms, but the design work is often the same: make ownership, release, transfer, and lifetime transitions explicit enough that there's only one reasonable interpretation at each boundary.

That's the useful frame for the webcrypto.dart case study that follows. Dart, BoringSSL, native build hooks, and finalizers make the details specific, but the underlying problems are not. They're the same ownership questions any managed/native boundary eventually has to answer: who owns this resource now, how must it be released, what changes when ownership moves, and what happens when the operation fails halfway through?

Case Study: webcrypto.dart Issue #278

webcrypto.dart is a useful example because several ownership systems meet in one fairly small surface area. In the browser, the package can rely on the platform Web Crypto implementation. Outside the browser, it crosses into native code through dart:ffi and uses BoringSSL underneath. Dart build hooks are responsible for producing the native assets.

That means a single operation can involve Dart object reachability, native reference counting, explicit allocation and cleanup, finalizers, and build tooling that determines how much of the resulting process a debugger can actually see.

A mistake at one layer doesn't necessarily fail there. A wrapper can return the right cryptographic result while leaving behind a leaked native object, or keep a finalizer attached to a handle whose ownership has already moved elsewhere.

Issue #278 was opened as part of the work leading up to the 1.0.0 release. The existing tests already covered the functional side of the package reasonably well. The harder question was what happened underneath those successful operations: whether native objects were released correctly, whether ownership transfers left stale owners behind, whether scopes and finalizers agreed about who was responsible for cleanup, and whether garbage-collection timing could expose lifetime bugs that ordinary tests would miss.

Those questions show up directly in constructs such as NativeFinalizer, _Scope, and _SslAllocator. BoringSSL also makes ownership distinctions that the Dart wrapper has to preserve. Some references are borrowed, some are retained, and some calls take ownership. When ownership moves into native code, the Dart side can't continue behaving as though it owns the same handle. The wrapper may need to remove it from a cleanup scope, detach a finalizer, or invalidate the handle entirely.

The natural next step was to run those paths under stronger memory-safety tooling. That's where the surrounding toolchain became part of the problem.

Why Valgrind Came First

The sanitizer path was blocked in the Dart SDK. webcrypto.dart relies on build hooks for its native assets, while sanitizer-enabled testing didn't yet support that setup cleanly. The missing work was tracked upstream in Dart SDK issue #63489.

There was a second problem around AOT packaging and symbolization. With compiled Dart code packaged into the runtime in the existing layout, native tooling couldn't expose the symbols in the form sanitizer reports needed. That work was tracked separately in issue #63435.

So simply adding a sanitizer flag inside webcrypto.dart wouldn't have produced a reliable sanitizer lane. The project first needed SDK support for building the right native assets under sanitizers and for producing reports that could be mapped back to useful code.

Valgrind didn't depend on that integration in the same way. So PR #295 added a Linux Memcheck lane that could run against the existing test path while the sanitizer work remained blocked upstream.

The workload was deliberately aimed at native-object lifetimes rather than at general algorithm coverage. It exercised AES-GCM, HMAC, ECDH, RSA-OAEP, key import and export, failed imports, and repeated creation of short-lived objects.

The algorithms themselves weren't the interesting part. Those operations drive different allocation and cleanup paths. Key import may allocate several temporary objects before ownership settles. A failed import exercises cleanup that the successful path never touches. Repeated short-lived operations put more pressure on finalization and native teardown than a long-running throughput test with a handful of persistent objects.

That's exactly where a memory checker is useful. The test still verifies that the operation works, but the dynamic-analysis lane can also ask whether anything was leaked, freed incorrectly, or left alive after the wrapper believed the operation was finished.

Valgrind also exposed a practical problem that appears whenever a memory checker is placed around a managed runtime: not every reported allocation necessarily belongs to the library being tested.

The lane reported both definite and possible leaks, but only definite leaks failed CI. Possible leaks remained visible in the output because some could come from Dart VM runtime behavior rather than webcrypto.dart itself. There was little value in making an ambiguous category blocking before the project could reliably attribute it.

That distinction matters beyond Valgrind. A useful CI gate needs a failure category the team understands well enough to act on. Broader findings can still be recorded and investigated without pretending they all carry the same confidence.

What the Case generalizes

The interesting part of this case is not that one Dart package happened to use Valgrind.

The first broader lesson is that the toolchain is part of sanitizer coverage. A compiler may support ASan or LSan perfectly well, but that doesn't mean every package, runtime, test runner, native-asset pipeline, or AOT layout can produce a useful instrumented execution. If the relevant code can't be built, loaded, or symbolized correctly, the sanitizer flag alone buys very little.

The second is that the workload has to follow the ownership model. Running more crypto operations isn't inherently useful. Running operations that repeatedly allocate, transfer, fail, close, and finalize native resources is. The same applies to database bindings, image codecs, language runtimes, GPU wrappers, JNI code, Python extensions, or any other FFI-heavy system.

The third is that ownership transitions need to be visible on both sides of the boundary. When the native side adopts a handle, the managed side should stop looking like an owner immediately. When cleanup belongs to a lexical scope, a successful transfer should remove the object from that scope. A finalizer can remain a fallback, but it shouldn't silently compete with deterministic cleanup for the same resource.

Finally, a blocked sanitizer integration doesn't have to mean no dynamic checking at all. In this case, the SDK couldn't yet support the sanitizer-backed test path the project wanted. Valgrind still provided a way to exercise native lifetimes in CI, while the missing sanitizer support remained tracked upstream.

The progression looked roughly like this:

webcrypto.dart didn't get the sanitizer lane it ultimately wanted from this work. The SDK couldn't support it yet. What the project did get was a repeatable native-lifetime workload, a Linux memory-checking lane that could run in CI, and upstream issues that made the remaining tooling gaps explicit.

That's a useful pattern well beyond Dart: enforce the runtime properties you can observe today, and keep the missing coverage visible until the toolchain can support something stronger.

Incident Patterns from Real Systems

Real incidents are most useful when they expose a failure mode that shows up elsewhere. The names and CVEs matter less than the shape of the bug: what had to happen for it to appear, what the sanitizer could see, and where the fix actually belonged.

Pattern 1: A use-after-free that only appeared under fuzzing

Mozilla's IndexedDB issue associated with CVE-2024-7528 is a good example of why ASan and fuzzing work so well together.

The use-after-free wasn't waiting on an obvious happy-path test. It needed a particular lifecycle state to occur first. Coverage-guided fuzzing pushed the code into that state, and ASan caught the stale access with enough allocation and deallocation history to reconstruct what had happened.

The important part is the sequence. A generated input reaches an object lifetime that normal tests rarely exercise. ASan catches the invalid access, the crashing input is kept, and the fix is made at the ownership transition that created the stale reference, not at the line that happened to dereference it.

Once the input is minimized, it should stay in the regression corpus. That's how a one-off sanitizer finding becomes a permanent test.

Pattern 2: Success returned before the output was fully valid

The NSS S/MIME bug associated with CVE-2022-31741 is more interesting than the phrase “use of uninitialized memory” makes it sound.

The caller treated a successful return as permission to use an output value. One path in the callee, though, could return success without initializing that output. MSan caught the uninitialized value later, but the actual bug was the disagreement between the return value and the output contract.

That distinction matters. Zeroing the variable at the call site might silence one report, but it would leave the API ambiguous. A successful call should establish a clear postcondition: either the output is fully initialized, or the API needs a way to represent partial state explicitly.

MSan is especially useful in this class of bug because it shows where the value became observable, while origin tracking can often point back to the path where initialization was skipped.

Pattern 3: A race report at an instrumentation boundary

Mozilla's TSan work around graphics code shows a different problem. A race report can be difficult to interpret when part of the synchronization story lives in code TSan can't see.

Suppose two accesses look unordered from TSan's point of view, but one of the modules involved isn't instrumented. If that module contains the synchronization that makes the accesses safe, TSan has no way to reconstruct the ordering. The reverse is also true: a real race inside uninstrumented code may never be reported at all.

That's why instrumentation boundaries should be part of race triage from the beginning. Before calling a report a false positive, map which modules were instrumented and which synchronization primitives were visible to the runtime. If the ordering depends on opaque code, either reproduce with a more complete instrumented stack or suppress that boundary narrowly and document why.

The dangerous shortcut is to treat “TSan can't see the whole system” as equivalent to “the race is harmless.”

Pattern 4: A small leak that still points to broken ownership

Leak reports are easy to dismiss when the retained allocation is small. Sometimes that's a reasonable severity decision. It's not the same thing as saying the ownership is correct.

A deterministic leak means some path allocated or retained something and never completed the corresponding release. Even a few bytes can be useful evidence because the same path may run in a loop, sit behind a long-lived request, be triggered repeatedly by invalid input, or accumulate during failure handling.

It helps to keep three questions separate:

• Impact: How often can the path run, how much memory does each execution retain, and can an external actor trigger it repeatedly?

• Correctness: Which owner failed to release the allocation, or which reference kept it alive unexpectedly?

• Policy: Does this block the current release, and what test or workload will make sure the leak doesn't return?

A leak can be low impact and still represent a real ownership bug. That distinction is worth preserving, especially in long-running services and FFI-heavy code where a small per-operation leak can become significant over time.

Fuzzing, Stress Harnesses, and Concurrency Probes

A sanitizer can only report a violation that actually occurs during the run. Instrumentation gives you visibility into bad execution, but it doesn't make the program visit the interesting paths on its own.

That's where workload design comes in.

Fuzzers are good at exploring input space. Stress harnesses repeatedly push objects through lifetime transitions. Concurrency probes increase the number of thread interleavings the program experiences. They complement sanitizers because each makes a different kind of otherwise-rare execution easier to reach.

Fuzzing: Explore Paths Ordinary Inputs Don't Reach

LLVM's libFuzzer is an in-process, coverage-guided fuzzer. A target is linked into the test binary and exposed through an entry point such as LLVMFuzzerTestOneInput. libFuzzer mutates inputs, observes which mutations reach new code through coverage instrumentation, and keeps inputs that expand that coverage.

A small target might look like this:

#include <cstddef>
#include <cstdint>

extern "C" void ParseOrImport(const uint8_t* data, size_t size);

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  ParseOrImport(data, size);
  return 0;
}

With ASan:

clang++ -O1 -g -fsanitize=fuzzer,address \
  fuzz_target.cc -o fuzz_target

./fuzz_target corpus/

And, when the dependency graph can be instrumented sufficiently for MSan:

clang++ -O1 -g -fsanitize=fuzzer,memory \
  -fsanitize-memory-track-origins=2 \
  fuzz_target.cc -o fuzz_target_msan

./fuzz_target_msan corpus/

The wrapper function is intentionally uninteresting. Most of the work is in choosing the boundary behind it.

A useful fuzz target is narrow enough that mutations reach meaningful states quickly. It should be deterministic, reasonably fast, and free of behavior such as process exits or unrelated global state that makes individual inputs difficult to reproduce. Parsers, decoders, protocol handlers, deserializers, key importers, file readers, and similar boundaries are natural targets because small changes in input can send execution through very different allocation and error-handling paths.

The sanitizer and the fuzzer do different jobs. The fuzzer may discover the path that leaves a pointer stale or an output partially initialized. ASan or MSan is what notices that the resulting execution is invalid.

The NSS S/MIME bug discussed earlier is a useful example of that relationship. An unusual message reached a decoder path where a successful return didn't imply that an output had actually been initialized. The interesting part wasn't merely that the input was malformed. It was that the input reached a state in which the API's initialization contract broke.

Once a fuzz input exposes a bug, keep the input. Minimize it if possible, reproduce the failure with the same sanitizer, fix the underlying defect, and add the minimized case to the regression corpus. Otherwise the fuzzer has found the same class of bug only once.

Stress Harnesses: Exercise the Transitions

Fuzzing is less effective when the interesting variable isn't the input but the object's lifetime.

A stale finalizer, a missed reference decrement, or a cleanup path that runs incorrectly after the tenth create/close cycle may not require unusual bytes at all. It requires the same lifecycle to happen enough times, or in the right sequence.

That's what a stress harness is for.

Instead of sending one large workload through a long-lived object, create and destroy many short-lived ones. Alternate successful operations with failures. Repeatedly acquire and release references. Exercise import followed by export, open followed by close, register followed by unregister, attach followed by detach. If the runtime allows it, introduce GC pressure between those transitions. If allocation reuse matters, create enough churn that recently freed memory is likely to be reused.

For example, a wrapper around a native resource might be exercised as:

create
  -> use
  -> close

create
  -> transfer
  -> destroy previous wrapper
  -> use new owner
  -> close

create
  -> partially initialize
  -> fail
  -> clean up

repeat thousands of times

That kind of workload is deliberately different from a throughput benchmark. A benchmark may spend most of its time using one object whose ownership never changes. A lifetime harness spends its time crossing the boundaries where ownership does change.

This applies well beyond FFI wrappers. Connection pools, reference-counted caches, plugin lifecycles, asynchronous request objects, GPU resources, file descriptors, temporary arenas, and callback registrations can all benefit from the same treatment.

Failure Injection Belongs Here Too

Some cleanup paths are difficult to exercise because ordinary execution rarely fails at the right point.

If an operation allocates four resources in sequence, testing only complete success and immediate failure leaves several intermediate states untouched. A useful harness can force the second, third, or fourth operation to fail and then check what remains.

That often reveals bugs such as:

• cleanup that assumes initialization completed

• resources released twice because ownership had already moved

• objects leaked only after partial construction

• error paths that leave output state looking valid

• callbacks or registrations that survive a failed setup

For ownership-heavy code, these intermediate failure states are often more interesting than malformed input.

Concurrency Probes: Make the Race Happen

TSan has a different coverage problem. It can't report a race merely because two operations could overlap in theory. The conflicting accesses must occur during the observed execution.

Short, deterministic unit tests are often poor at producing those schedules.

A concurrency probe deliberately raises the odds. Start workers from the same barrier instead of one after another. Repeat the critical operation many times. Vary worker counts. Race registration against removal, close against completion, creation against shutdown, and publication against destruction. Where appropriate, test-only yields or short scheduling points can widen a window that normally exists for only a few instructions.

A deliberately racy example makes the basic idea visible:

#include <atomic>
#include <thread>

struct SharedState {
  int plain_counter = 0;
  std::atomic<bool> stop{false};
};

void writer(SharedState* s) {
  while (!s->stop.load(std::memory_order_relaxed)) {
    s->plain_counter++;  // Intentionally unsynchronized.
  }
}

void reader(SharedState* s) {
  for (int i = 0; i < 1000000; ++i) {
    (void)s->plain_counter;  // Intentionally unsynchronized.
  }

  s->stop.store(true, std::memory_order_relaxed);
}

int main() {
  SharedState state;

  std::thread t1(writer, &state);
  std::thread t2(reader, &state);

  t1.join();
  t2.join();
}

There's nothing subtle about the bug in that example. The useful part is the shape of the harness: concurrent work is repeated enough that the conflicting accesses are likely to overlap.

Real probes should target transitions that already exist in the system. If a callback can complete while an object is closing, race those operations. If a registry is read while another thread removes entries, hammer both sides. If explicit cleanup and background cleanup can touch the same state, force them to overlap.

The goal isn't to manufacture behavior the real program could never produce. It's to stop relying on the scheduler to stumble across a legal but inconvenient interleaving by accident.

Recording random seeds, operation sequences, worker counts, and other scheduling inputs also matters. A race that appears once in a nightly run but can't be reconstructed locally is much harder to fix than one whose workload can be replayed.

Treat Discoveries as Regression Inputs

Fuzzing and stress testing become much more valuable when their outputs survive the debugging session.

For a fuzzer, that usually means keeping the minimized crashing or leaking input in the corpus. For a lifecycle harness, it may mean preserving the exact operation sequence that triggered the failure. For a concurrency test, it can mean retaining the seed, worker count, or schedule parameters that made the race reproducible.

OSS-Fuzz is a useful model here even for projects that never use the service itself. Its workflow treats fuzz targets, corpora, sanitizer builds, reproducers, and regression testing as parts of the same system rather than separate debugging activities.

That's the habit worth copying: keep fuzz targets with the code they exercise, version useful corpus inputs, make failures reproducible outside the fuzzing infrastructure, and replay previously failing inputs under sanitizer builds.

The larger point is simple. Sanitizer coverage is partly a property of the build, but it's also a property of the workload. ASan can't catch the stale pointer nobody dereferences. MSan can't expose the partially initialized state no test reaches. TSan can't report an interleaving that never occurs.

Good instrumentation tells you when an execution became invalid. Good workload design makes the executions worth observing happen often enough to find.

How to Read and Triage Sanitizer Reports

A sanitizer report is evidence, not a diagnosis.

The top frame tells you where the runtime finally noticed something was wrong. That may be the place to start reading, but it's often not the place where the bug began. A useful triage session works backward from the report until the failed lifetime, initialization, or synchronization assumption becomes clear.

First, Make Sure the Report is Worth Trusting

There's little value in reasoning from a stack full of raw addresses.

Before changing code, make sure the failing binary has useful debug information, the symbols belong to that exact build, and the module you care about was actually instrumented. CI makes this easier to get wrong than it sounds: an old symbol bundle, a rebuilt binary, or a report copied from a different job can produce a perfectly plausible but useless stack.

For Clang sanitizer builds, -g and -fno-omit-frame-pointer are a sensible baseline, with llvm-symbolizer available when the test runs. If optimization is making the stack difficult to reconstruct, a moderate optimization level and -fno-optimize-sibling-calls can help.

Keep the exact sanitized binary and its symbols with the report. The same goes for the sanitizer options, suppression files, reproducer, and anything else needed to recreate that execution.

Unsymbolized hexadecimal addresses aren't a badge of low-level seriousness. They're missing observability.

Read the Report According to What the Tool is Actually Accusing You Of

It's tempting to put ASan, LSan, MSan, TSan, and Valgrind findings into one bucket called “memory bugs.” That usually leads debugging in the wrong direction.

An ASan use-after-free is fundamentally a lifetime question: why did one part of the program still believe the object was usable after another part ended its lifetime?

An LSan report asks something different. The allocation survived until teardown, so which owner was supposed to release it, and what prevented that ownership from reaching an end?

With MSan, the interesting question is usually not “why wasn't this variable zero?” It's where the program acquired the right to treat that value as initialized. Often the answer leads back to an output parameter, a partially constructed object, or a success path whose postconditions were weaker than the caller assumed.

A TSan report is about ordering. Two conflicting accesses happened without a synchronization relationship the runtime could establish. The debugging job is to find the edge that should have ordered them and determine whether it's missing or simply hidden behind an instrumentation boundary.

Valgrind invalid reads and writes need a little more classification because they can point to several of these problems: stale ownership, bounds mistakes, allocator misuse, or behavior complicated by the surrounding runtime.

The distinction matters because a local change can easily hide the symptom without repairing the contract. Zero-initializing an MSan-reported buffer may make the poison disappear while leaving an ambiguous API untouched. Adding a delay around a TSan race may make one test pass without introducing any synchronization at all.

Treat the Stacks as a Timeline

For a use-after-free, the current access is only one event in the story.

The allocation stack tells you where the object's lifetime began. The free stack tells you where somebody decided that lifetime was over. The current stack tells you who still disagreed.

That disagreement is usually more useful than whichever line happens to be at the top of the report.

The same idea applies to the other sanitizers. With MSan, the use stack shows where an uninitialized value finally mattered, while origin tracking can point toward where the value first entered the program without a valid initialization history. With TSan, neither conflicting access is meaningful on its own. Thread creation, publication, locks, atomics, and other synchronization events are part of the same story.

It often helps to reduce the report mentally to something like:

object created
    -> ownership transferred
    -> old owner still retains access
    -> new owner destroys object
    -> old owner dereferences stale handle

Or for an initialization bug:

output allocated
    -> rare branch skips initialization
    -> function still reports success
    -> caller consumes output

Once the sequence is written down, the crash line often stops looking like the right place to fix the bug.

Minimize the Reproducer Without Removing the Thing That Makes the Bug Possible

“Minimal reproducer” is sometimes taken too literally.

The smallest useful reproducer isn't necessarily the program with the fewest lines. It's the smallest one that still preserves the conditions the bug depends on.

For an FFI lifetime bug, removing the managed runtime may also remove the finalizer behavior that triggers it. Replacing a pool allocator with malloc can eliminate the reuse pattern that makes a stale pointer visible. Simplifying a concurrent test to one thread certainly makes the program smaller, but it also removes the race.

The same caution applies to optimization level, GC pressure, input structure, thread count, allocator configuration, and instrumentation boundaries. Strip away unrelated code aggressively, but keep the mechanics that make the failure real.

A good reproducer should make the report easier to reason about without turning it into a different program.

Fix the Ownership or Synchronization Mistake, Not Just the Reported Access

Sanitizers often expose problems at the point where an earlier design decision finally became unsafe.

A stale pointer might be fixed properly by invalidating a wrapper after ownership moves. A leak may require making one object explicitly responsible for releasing a retained reference. An MSan report may lead to changing an API so that success always produces a fully initialized result. A TSan finding may expose a flag that was treated as “basically atomic” but actually needs a defined synchronization protocol.

Sometimes the correct fix really is a local bounds check or an initialization statement. But it's worth asking one more question before stopping: what allowed the invalid state to exist in the first place?

That question is particularly useful at FFI and library boundaries, where several callers can otherwise repeat the same mistake in slightly different forms.

Keep the Failure After You Fix it

A sanitizer bug that disappears after a patch but leaves no regression test behind is easy to rediscover months later.

The reproducer should become part of the normal engineering record. A fuzzing failure belongs in the regression corpus. A lifecycle bug can become a focused unit test or teardown workload. A race may need its triggering seed, worker count, or operation sequence preserved alongside the test.

Then rerun that exact workload under the same sanitizer that found the problem.

That last part matters. A normal unit test passing after an ASan fix doesn't prove that the ASan failure is gone. The instrumented execution that originally exposed the bug should be the one that closes it.

The goal of triage is therefore not merely to make the report disappear. It's to reconstruct why the execution became invalid, repair that assumption at the right boundary, and leave behind a workload that will catch the same mistake if it returns.

CI Policy and Acceptance Criteria

Once sanitizer reports are reproducible and trustworthy, the next question is where they belong in CI.

Putting every tool on every pull request is rarely the right answer. ASan, MSan, TSan, leak detection, fuzzing, and whole-process tools impose very different costs and depend on different levels of instrumentation. A useful CI setup reflects those differences instead of treating sanitizers as one generic “safety” job.

The practical split is between fast checks that protect the merge path and slower jobs that search for failures the regular test suite is unlikely to expose.

ASan is often the easiest sanitizer to put close to the normal development loop. Its overhead is noticeable but usually manageable, and the failures it reports (use-after-free, out-of-bounds access, invalid free, double free) are rarely findings you want to knowingly merge. For native-heavy projects, an ASan build combined with UBSan is a reasonable pre-merge lane if the build and test suite can support it.

Leak detection is slightly different. When LSan runs as part of the ASan build, it may fit into the same lane. Whether every leak should block immediately depends on how clean the runtime environment is. A focused native test binary with deterministic teardown can usually be strict from the beginning. A managed runtime or large host process may first need a baseline so that application leaks can be separated from runtime residue.

TSan usually belongs farther from the critical merge path. The runtime cost is much higher, and a useful TSan run needs workloads that actually create meaningful concurrency. Running a short unit suite under TSan on every commit can be expensive while still giving poor race coverage. A nightly run, a pre-release lane, or a targeted job for concurrency-heavy changes often provides more useful evidence.

MSan has a different constraint. Its value depends heavily on how much of the dependency graph can be instrumented. If the application and its important libraries can be built in a consistent MSan environment, it can be an excellent gate for initialization bugs. If half the stack is opaque, forcing it into the ordinary PR matrix may produce more confusion than confidence. In many projects, MSan ends up as a dedicated build rather than a routine variant of the normal test job.

Fuzzing also has two different CI roles. The expensive search for new inputs can run continuously or on a schedule, while previously discovered inputs should be cheap enough to replay much more often. Once a crashing testcase has been minimized and checked into the corpus, there's little reason to wait for another fuzzing campaign to learn that the same bug came back.

A typical policy might therefore look something like this:

The exact schedule matters less than the reasoning behind it. A five-minute ASan job that catches real regressions belongs much closer to the developer than a two-hour TSan workload. A nightly MSan build may be more valuable than an unreliable per-PR job against half-instrumented dependencies.

CI cost should follow the quality of the evidence, not a desire to make the matrix look comprehensive.

Decide What a Red Lane Means

A sanitizer job shouldn't be red merely because the tool printed something unusual. It should be red because the project has decided that a particular class of evidence represents a defect it's unwilling to accept.

For ASan, that bar is usually straightforward: a reproducible unsuppressed memory-safety violation in code you own should fail the build.

MSan can be treated similarly once the instrumented environment is complete enough that the reports are trusted. An uninitialized value influencing execution isn't something to wave through because the test happened to finish successfully.

TSan needs a little more context around instrumentation boundaries, but a race between accesses in code you own should normally block once the report has been confirmed. “It only happens under TSan” isn't a useful acceptance criterion.

Leak policy tends to need the most judgment. A deterministic direct leak from a focused native test is very different from a possible leak reported while an entire managed runtime is shutting down. The webcrypto.dart case study is a useful example: definite Valgrind leaks could be made blocking while possible leaks remained visible until they could be attributed with more confidence.

That's a better pattern than either extreme. Ignoring all leak reports wastes useful ownership evidence. Failing every ambiguous allocation from day one can make the lane unusable.

Keep Discovery Separate from Regression

There's also a useful distinction between a job that's looking for new bugs and one that's making sure an old bug hasn't returned.

A fuzzer running for hours is doing discovery. A saved fuzz input replayed in a few milliseconds is regression testing.

A TSan stress job running millions of operations is discovery. A small reproducer for a race fixed last month is regression testing.

A large teardown workload looking for new leaks is discovery. A ten-line helper binary that reproduces a previously fixed ownership leak is regression testing.

The second category should usually move closer to the merge path. Once a difficult failure has been reduced to something cheap and deterministic, there's little value in leaving it trapped inside the expensive job that originally found it.

This is one of the ways sanitizer coverage becomes cheaper over time: expensive discovery produces small regression tests.

Preserve Enough Context to Debug the Failure

When a sanitizer lane does fail, CI should leave behind more than a red status.

The useful artifacts are the ones needed to reconstruct the same execution: the sanitized binary, matching symbols, raw report, sanitizer options, suppression files, and any input, seed, or workload parameters that triggered the failure.

A fuzzing job should keep the crashing input. A concurrency workload should keep the seed or operation sequence when one exists. A toolchain-specific build should record the compiler and sanitizer runtime versions that produced the report.

This doesn't need to be tied to GitHub Actions, GitLab CI, Buildkite, Jenkins, or any other runner. The implementation syntax changes but the requirement does not. A developer looking at the failure tomorrow should be able to reproduce the same instrumented execution without reverse-engineering the CI machine.

Write the Policy Down

The worst time to decide whether a sanitizer finding is release-blocking is when a release is already waiting on it.

A project should have a small, explicit set of rules for each lane: what it's expected to observe, what findings fail it, which categories are informational, and what kinds of suppressions are allowed.

Those rules can evolve. A new leak lane may begin by collecting reports while the baseline is understood, then become blocking for deterministic leaks once the noise has been removed. A TSan job may start nightly and later move onto selected pull requests when its workload becomes faster. An MSan environment may expand as more dependencies become instrumentable.

What matters is that a green lane has a stable meaning.

“ASan passed” should mean more than “the CI command exited zero.” It should mean that the expected code was instrumented, the intended workload ran, reports were symbolized, and no finding in the project's blocking categories occurred.

That's the point of CI policy around sanitizers: not to maximize the number of tools in the matrix, but to make each green result say something you actually trust.

Production Strategy, Suppression Debt, and Metrics

Most sanitizer builds are designed for testing rather than broad production use. Their runtime cost, memory footprint, and instrumentation requirements make them a poor fit for many live workloads.

Production still matters to the sanitizer program. It tells you which workloads matter, which failures escaped pre-production, and where the instrumented lanes are too narrow. The useful connection is a feedback loop: production exposes behavior that testing missed, and that behavior becomes a new sanitizer workload, reproducer, or regression test.

Bring New Lanes Online Before They're Perfect

New sanitizer jobs are rarely clean on the first run.

An existing codebase may already contain real defects, third-party reports, runtime residue, missing symbols, incomplete instrumentation, or suppressions inherited from earlier work. Making every finding blocking immediately can leave the team with a permanently red job and no clear idea which findings deserve action.

A practical rollout starts by collecting reports and learning the baseline. Fix deterministic bugs in code you own. Separate those findings from dependency noise and instrumentation gaps. Make the interesting failures reproducible. Once a category is understood well enough that a new report has a clear meaning, that category can become blocking.

The lane should keep moving in that direction. An observation-only job that stays observational indefinitely is collecting data, but it hasn't become much of an engineering control.

Waiting for a perfectly clean baseline before adding the lane causes a different problem. The cleanup work often happens because the findings are visible and recurring. Keeping the job out of CI until everything is already fixed can postpone that work indefinitely.

Suppressions Should Feel Temporary

Some suppressions are unavoidable. A third-party library may have a known race you can't patch immediately. A runtime may keep allocations alive until shutdown. An instrumentation boundary may produce a report that can't yet be resolved cleanly.

Trouble starts when the suppression file becomes the place where inconvenient findings go to disappear.

A useful suppression should tell the next person enough to understand why it exists: what report is being hidden, where it comes from, why it can't be fixed now, who owns the follow-up, and what issue tracks the work.

The match should be as narrow as practical. Suppressing an entire library or namespace can silence today's known problem together with an unrelated regression introduced later.

Age is often more informative than the raw count. Five recent suppressions tied to active upstream bugs may be manageable. Five suppressions that nobody has reviewed in three years describe a different state of the program.

When the underlying defect is fixed, remove the suppression in the same change. Leaving it behind keeps the sanitizer blind to code that no longer needs an exception.

It's useful to think of suppressions as observability debt. Each one may be justified, but each one also weakens what a clean run can honestly tell you. Count them, track their age, and keep old exceptions visible enough that they are hard to forget.

Keep Enough of the Build to Explain a Failure Later

Symbolization feels like a local debugging detail until a sanitizer fails only in CI.

At that point, the report is useful only if its addresses can be tied back to the exact binary that produced them. Rebuilding from the same commit isn't always equivalent. Optimization, linker decisions, generated code, and toolchain versions can all change the layout.

For sanitizer jobs worth keeping, retain the artifacts needed to reconstruct the failure: the instrumented binary, matching debug information, raw sanitizer output, relevant runtime options, and the input or workload that triggered the report. Build IDs and compiler/runtime versions are useful when the pipeline produces several variants of the same program.

Managed runtimes, JITs, AOT compilers, and plugin systems need additional care because a native address may not map directly to an ordinary source file.

The Dart AOT issue discussed in the webcrypto.dart case study is a good example. Native tooling could observe the bad execution, but the report lost much of its practical value if the runtime layout prevented useful Dart symbols from appearing in the stack.

A sanitizer report is only as useful as your ability to connect it back to the code that produced it. Preserving symbolization belongs in the build and release setup, not in a developer's bag of local debugging tricks.

Feed Production Findings Back into Sanitizer Workloads

A clean sanitizer matrix still covers only the executions produced by the test environment.

Production sees different object lifetimes, request mixtures, allocator pressure, machine sizes, thread schedules, and long-running behavior. Where the platform supports them, guarded allocators, hardware-assisted memory tagging, sampling, canary deployments, and richer crash telemetry can expose failures that never appeared in CI.

Those mechanisms provide a different kind of evidence. Some are sampled or platform-specific rather than deterministic. Their value is that they observe executions the sanitizer lab may never produce on its own, and those findings can feed back into the test system.

When a production crash points to a lifetime edge that no existing stress test exercises, turn that lifecycle into an ASan workload. If telemetry shows memory growth after a particular failure path, build a focused teardown case and run it under leak detection. If a concurrency failure appears only under high load, preserve the relevant operation sequence and attack it with a TSan stress test.

That feedback loop keeps the sanitizer workloads tied to the behavior of the real system without trying to make production itself look like the sanitizer lab.

Measure Whether the Lanes Are Becoming More Useful

Sanitizer programs can produce plenty of numbers. The useful ones tend to answer three practical questions:

1. Are we finding real bugs?

2. Can we reproduce them?

3. Is the cost of doing so still reasonable?

A small set of metrics usually tells you more than a large dashboard:

Those numbers need context.

A sanitizer lane becoming faster isn't an improvement if symbolization was removed to get there. A TSan job with zero flakes may simply be running a workload that barely creates concurrency. A fuzz corpus can show impressive line coverage while still missing the lifecycle transition where ownership actually breaks.

Even a falling bug count is ambiguous. The codebase may genuinely be getting healthier, or the workload may have stopped reaching interesting paths.

Metrics are useful when they help distinguish those cases. Once the number itself becomes the target, it becomes easy to improve the dashboard while making the underlying evidence worse.

What Maturity Looks Like

A mature sanitizer program has a CI matrix whose results are understood and useful. The number of jobs is secondary.

Its lanes have understood failure modes. Reports can be reproduced. Suppressions are narrow and temporary. Old bugs become cheap regression tests. Expensive discovery jobs are used where they add information that faster jobs can't. Production incidents feed new workloads back into the instrumented environment.

Over time, “Are we running ASan, TSan, and MSan?” becomes less interesting than “What classes of invalid execution can still pass through this system without us seeing them?”

That second question exposes the gaps that matter: weak workload coverage, opaque dependencies, poor symbolization, ownership transitions the tests never exercise, or synchronization the tooling can't observe.

A sanitizer program improves as those blind spots become smaller, better understood, and harder for new regressions to hide inside.

Closing Perspective

Sanitizers are most useful when they become part of how a system is built and tested, rather than something switched on after a crash.

ASan, LSan, MSan, and TSan observe different failures, but their reports all force the program to answer questions it was already depending on.

Was this memory still valid to access? Who was responsible for releasing it? Had this value actually been initialized before it was used? What synchronization ordered these two accesses?

Those assumptions exist with or without instrumentation. Sanitizers make some of them observable at runtime.

That's also why the compiler flag is only a small part of the setup. The relevant code has to be instrumented. The workload has to reach the lifetime transition, failure path, or thread interleaving where the assumption can break. The report has to retain enough symbol information to reconstruct what happened.

CI needs a clear policy for which findings are trusted, which are still being investigated, and which gaps are knowingly outside the lane.

Ownership is a recurring theme in this handbook because native boundaries make lifetime assumptions especially easy to lose. Borrowing, retaining, transferring, and releasing aren't bookkeeping details. They determine who may still touch a resource and who must eventually destroy it. Finalizers, cleanup scopes, custom allocators, and FFI wrappers add more places where two parts of the program can end up with different answers.

The webcrypto.dart case study shows the same problem from the tooling side. The sanitizer path the project wanted was blocked upstream, so the work proceeded with a memory-checking path the existing build system could support. The native-lifetime workload still became testable, the higher-confidence findings could still be enforced, and the missing sanitizer support stayed visible instead of being treated as solved.

The details there are specific to Dart, BoringSSL, build hooks, and Valgrind. The situation is not. MSan loses authority when initialization crosses code it can't instrument. TSan becomes harder to interpret when synchronization happens inside an opaque dependency. ASan can't enforce logical object boundaries that a custom allocator never exposes.

A clean run only means something when you understand those boundaries.

The same applies after a bug is found. A useful report becomes a reproducer. A reproducer becomes a regression test. Repeated ownership failures may justify changing an API instead of adding another local fix. A race that appears only under stress should leave behind a workload capable of producing that overlap again. Suppressions should disappear when the reason for them disappears.

Over time, this changes the value of the sanitizer program. Individual reports still matter, but the larger payoff is a codebase in which lifetime, initialization, and synchronization mistakes have fewer places to remain invisible.

A green sanitizer lane isn't a certificate that the program is safe. It's evidence that a particular set of runtime contracts survived a particular set of executions.

The useful question after that run is what the lane still couldn't see.

Sanitizers aren't debug flags. They're executable checks on the contracts your program already depends on.

References and Further Reading

Sanitizer Documentation and Design

• Clang AddressSanitizer documentation

• Clang LeakSanitizer documentation

• Clang MemorySanitizer documentation

• Clang ThreadSanitizer documentation

• Google Sanitizers: AddressSanitizer algorithm

• Google Sanitizers: LeakSanitizer

• Google Sanitizers: ThreadSanitizer C++ manual

• AddressSanitizer: A Fast Address Sanity Checker

• MemorySanitizer: fast detector of uninitialized memory use in C++

• Valgrind Memcheck manual

Fuzzing and Continuous Testing

• LLVM libFuzzer documentation

• OSS-Fuzz documentation

• OSS-Fuzz ideal integration guidance

• OSS-Fuzz reproducing guide

Ownership, FFI, and the Anchor Case Study

• BoringSSL API conventions

• BoringSSL EVP_PKEY declarations and ownership APIs

• google/webcrypto.dart repository

• webcrypto.dart issue #278

• Dart hooks documentation

• dart build documentation

• dart compile documentation

• Dart SDK issue #63435: separate AOT runtime mode and symbolization

• Dart SDK issue #63489: sanitizer support with code assets

Public Incident References

• Mozilla Bug 1767590 / CVE-2022-31741: uninitialized value in S/MIME decryption

• Mozilla Bug 1688716: ThreadSanitizer report involving non-instrumented graphics code

• Mozilla Bug 1879437: LeakSanitizer report in JIT state handling

• Mozilla Bug 1895951 / CVE-2024-7528: AddressSanitizer heap-use-after-free

Desktop and server processors hide memory latency behind multi-level caches. Many embedded processors, especially ARM Cortex-M and Cortex-R based chips, take a different approach. They give you direct control over multiple memory regions, each with very different performance characteristics.

This handbook covers what ITCM, DTCM, and DDR memory are, how they differ, how to place code and data in the right region, and how to profile and monitor firmware memory usage over time.

Table of Contents

• Prerequisites

• Why Embedded Memory Architecture Matters

• What is ITCM (Instruction Tightly-Coupled Memory)?

• What is DTCM (Data Tightly-Coupled Memory)?

• What is DDR (Double Data Rate) Memory?

• How They Compare: A Side-by-Side Overview

• How to Decide Where to Place Code and Data

• How the Linker Script Controls Memory Placement

• Common Mistakes to Avoid

• Performance Comparison With Real Numbers

• How TCM Affects Power Consumption

• How to Profile Memory Usage

• Summary

Prerequisites

To get the most from this guide, you should have a basic understanding of C programming, including pointers, structs, and the difference between static and local variables.

Some familiarity with embedded development concepts like compiling, linking, and flashing firmware to a target board will also help.

Finally, a general sense of how a CPU fetches and executes instructions will make the performance discussions easier to follow.

You don't need to be an expert in any of these. The article explains each concept as it comes up.

Why Embedded Memory Architecture Matters

A modern embedded processor might be clocked at 400 MHz or higher. It can execute an instruction every few nanoseconds.

But when it needs to fetch that instruction from memory, or read a variable, the memory might not keep up. The processor ends up stalling, waiting for the memory subsystem to deliver the data it asked for. Those stall cycles add up fast.

On a desktop computer, hardware caches (L1, L2, L3) sit between the CPU and main memory, automatically keeping recently-used data nearby. The cache hardware decides what to keep and what to evict, and it does this transparently. The programmer rarely needs to think about it, and performance is generally good enough without manual intervention.

On many embedded processors, the situation is different. Instead of hardware caches, you get three distinct memory regions, each attached to the CPU in a different way.

The table above shows the three memory types you'll encounter on a typical ARM Cortex-M or Cortex-R-based embedded system. ITCM and DTCM are fast but small. DDR is slow but large.

The "deterministic" label on TCM means that the access time is always the same, every single time, regardless of what accessed that memory before or what else is happening on the chip. The "variable" label on DDR means the access time can change depending on the internal state of the DDR chip and its controller.

You, the developer, control which region each piece of your firmware lives in. The compiler and linker don't make these decisions automatically. You specify them through section attributes in your source code and placement rules in your linker script. Getting this right is often the difference between firmware that meets its real-time deadlines and firmware that misses them.

What is ITCM (Instruction Tightly-Coupled Memory)?

ITCM stands for Instruction Tightly-Coupled Memory.

The "Instruction" part means this memory is used for storing executable machine code, the compiled instructions your CPU fetches and runs.

The "Tightly-Coupled" part means the memory is physically located on the same silicon die as the CPU core, connected through a dedicated bus with no arbitration or contention. There's no shared bus to compete with. There's no cache hierarchy to traverse. The CPU asks for an instruction, and ITCM delivers it directly, through a private path that nothing else on the chip can interfere with.

The CPU can fetch an instruction from ITCM in a single clock cycle, every time. This access time is both fast and deterministic. It doesn't vary based on access patterns, recent history, or what else is happening on the bus.

This determinism is just as important as the raw speed, because it makes worst-case execution time analysis possible. In safety-critical systems, you need to be able to prove that a function will always complete within a certain number of cycles. ITCM makes that proof much simpler.

Why Single-Cycle Fetch Matters

Every line of C code compiles down to one or more machine instructions. Each of those instructions must be fetched from memory before the CPU can decode and execute it. This fetch step happens for every single instruction, so even small per-instruction delays compound rapidly in loops and frequently-called functions.

Consider a loop that runs 1,000,000 iterations, where each iteration involves 10 instruction fetches. That's 10 million fetches total.

ITCM:  10,000,000 fetches x 1 cycle  = 10,000,000 cycles
DDR:   10,000,000 fetches x 8 cycles = 80,000,000 cycles

Difference: 70,000,000 cycles
At 400 MHz: 70,000,000 / 400,000,000 = 0.175 seconds = 175 ms

This calculation compares the total cycle count when the same loop runs from ITCM versus DDR. With ITCM, each fetch takes 1 cycle, so 10 million fetches cost 10 million cycles.

With DDR, each fetch takes 8 cycles (a conservative average), so the same 10 million fetches cost 80 million cycles. The difference is 70 million cycles, which at 400 MHz translates to 175 milliseconds.

In a real-time system running a control loop at 1 kHz (one iteration every 1 ms), 175 ms of extra latency spread across your processing isn't a minor inconvenience. It can cause the system to miss deadlines, drop sensor readings, or produce incorrect outputs. In motor control applications, a missed deadline can mean physical damage to the hardware. In audio processing, it means audible glitches. The cost of slow instruction fetch isn't abstract.

What Should Go in ITCM?

Because ITCM is small (typically 512 KB to 2 MB), you can't fit your entire firmware in it. You need to be selective about what earns a spot.

Interrupt Service Routines (ISRs) are the highest-priority candidates. ISRs run in response to hardware events like a timer tick, an ADC conversion completing, or a communication peripheral receiving data. They need to execute and return as quickly as possible.

A slow ISR delays all lower-priority interrupts and can cause missed events. If your ISR fetches its instructions from DDR, each fetch takes multiple cycles, and the total ISR execution time increases by a factor that could push it past its deadline.

Placing ISRs in ITCM ensures they run at maximum speed with completely predictable timing.

Real-time processing functions are the next priority. These include signal processing routines, motor control loops, audio processing pipelines, and any function that runs at a fixed rate and must complete within a strict time budget.

If your audio codec callback needs to process a buffer of samples every 5 ms, every instruction fetch cycle counts. Placing these functions in ITCM gives you the maximum amount of CPU time for actual computation rather than waiting on memory.

Inner loops of your main processing pipeline also benefit significantly from ITCM placement. If your firmware spends 80% of its time in a handful of functions, those functions should be in ITCM. Profiling tools and the linker map file (covered later in this article) can help you identify which functions are the hottest.

Functions that require deterministic timing belong in ITCM even if they aren't the fastest path. ITCM access time doesn't vary, which makes timing analysis predictable. This matters for safety-critical systems (automotive, medical, aerospace) where you need to prove worst-case execution times to a certification authority.

How to Place a Function in ITCM

You use a GCC section attribute to tell the compiler that a function belongs in a specific memory section. Then, in your linker script, you map that section to the ITCM memory region.

__attribute__((section(".itcm_text")))
void my_critical_isr(void) {
    volatile uint32_t *sensor_reg = (volatile uint32_t *)0x40001000;
    uint32_t reading = *sensor_reg;
    process_sample(reading);
}

In this code, the __attribute__((section(".itcm_text"))) directive tells the compiler to emit this function's compiled machine code into a section called .itcm_text instead of the default .text section. The function itself reads a sensor register at the memory-mapped address 0x40001000, stores the result in a local variable, and passes it to process_sample() for further processing. The volatile keyword tells the compiler that this memory address can change at any time (because it is a hardware register), so the compiler must not optimize away the read.

On its own, the section attribute doesn't determine where the function ends up in physical memory. It just tells the compiler to label the function's code with a specific section name.

The actual memory placement is the linker script's job, which maps .itcm_text to the ITCM address range. We'll cover the linker script in detail in a later section.

How Much ITCM is Typical?

A real-world memory profile from an embedded project, to give you a sense of scale:

Memory region         Used Size  Region Size  %age Used
            ITCM:      570936 B         2 MB     27.22%
            DTCM:      727240 B    1572608 B     46.24%
             DDR:      622915 B         4 MB     14.85%

This output comes from the linker map file's summary section. It shows three memory regions and how much of each one is used by the compiled firmware.

ITCM has 2 MB available and the firmware is using about 557 KB (27.22%). DTCM has about 1.5 MB available and is using 727 KB (46.24%). DDR has 4 MB available and is using about 609 KB (14.85%).

This project uses about 557 KB of the available 2 MB of ITCM, roughly 27%. That leaves good headroom for growth.

In practice, you want to keep ITCM utilization below 80-85% to leave room for future features and library updates. If utilization climbs above 90%, you're one feature addition away from a build failure, and you should proactively move less-critical code to DDR.

What is DTCM (Data Tightly-Coupled Memory)?

DTCM stands for Data Tightly-Coupled Memory. It works on the same principle as ITCM (physically close to the CPU core, connected via a dedicated bus, single-cycle access) but it stores data instead of instructions.

If ITCM is where your code lives, DTCM is where your code works. It's the fast scratch space that the CPU reads from and writes to while executing your performance-critical functions. Every variable read, every array access, every stack push and pop in your hot code paths goes through data memory. Making that data memory as fast as possible eliminates one of the biggest sources of stall cycles.

What Kind of Data Belongs in DTCM?

Stack frames are the most important thing in DTCM. Every function call pushes a stack frame containing local variables, the return address, and saved registers. Every function return pops that frame. I

f your stack is in DTCM, the memory-access portion of function calls and returns happens in a single cycle. If your stack were in DDR, every function call and return would incur multiple cycles of memory latency just for the stack operations alone, before the function even begins doing useful work.

On most Cortex-M and Cortex-R configurations, the startup code initializes the stack pointer to point into DTCM by default, so you get this benefit without any extra configuration.

Frequently accessed global variables are another strong candidate. State machine variables, control flags, sensor readings that are updated and read in every loop iteration, counters that are incremented in ISRs and read in the main loop: all of these benefit from single-cycle access.

If a variable is read or written thousands of times per second, the cumulative latency difference between DTCM and DDR adds up.

Small lookup tables used in hot paths belong in DTCM when they're small enough to fit. Sine/cosine tables for motor control, filter coefficients for audio processing, and CRC tables for communication protocols are common examples.

These tables are typically a few hundred bytes to a few kilobytes, and they get accessed on every iteration of a processing loop. The key word is "small." A 512-byte sine table is a good fit for DTCM. A 64 KB calibration table is not, and should go in DDR instead.

DMA buffers can sometimes go in DTCM, but this depends on your chip's bus architecture. On some chips, the DMA controller has a direct path to DTCM through the bus matrix. On others, the DMA controller can only reach DDR and possibly other SRAM regions. If you place a DMA buffer in DTCM on a chip where the DMA controller can't reach it, the transfer will silently fail or write to a completely wrong address.

Always check your chip's bus matrix diagram in the reference manual before putting DMA buffers in DTCM.

How to Place Data in DTCM

Placing data in DTCM uses the same section attribute mechanism as ITCM, but with a section name that your linker script maps to the DTCM address range.

__attribute__((section(".dtcm_data")))
static int16_t audio_buffer[256];

__attribute__((section(".dtcm_data")))
static volatile uint32_t sensor_state = 0;

In this code, audio_buffer is an array of 256 signed 16-bit integers (512 bytes total) that will be placed in DTCM. This could be a buffer for audio samples that gets filled by a DMA transfer and processed by an ISR. The static keyword means the buffer has file scope and persists for the lifetime of the program (it's not allocated on the stack).

The sensor_state variable is a 32-bit unsigned integer marked as volatile, meaning the compiler must read it from memory every time it's accessed rather than caching it in a register.

This is important for variables that are written in an ISR and read in the main loop, since the compiler needs to know the value can change at any time. Placing it in DTCM ensures that both the ISR write and the main loop read happen in a single cycle.

DTCM Fills Up Faster Than ITCM

Looking at the memory profile again:

            DTCM:      727240 B    1572608 B     46.24%

This single line from the linker map file summary shows that DTCM has 1,572,608 bytes (about 1.5 MB) available, and the firmware is using 727,240 bytes (about 710 KB), which is 46.24% of the total capacity.

DTCM fills up faster than ITCM because many things compete for it: your stack, your heap (if you have one), your global variables, and data sections from every library you link against. Every C library function that uses static data, every RTOS data structure, every middleware component brings its own data footprint. This creates a constant sizing exercise.

For every data structure, you need to ask: does this really need single-cycle access, or can it work from DDR?

A Concrete Example of the Performance Impact

Say your processor runs at 400 MHz. DTCM gives you 1-cycle access. DDR gives you 8-cycle access. You have a lookup table that gets accessed 100,000 times per second.

DTCM: 100,000 accesses x 1 cycle  = 100,000 cycles/sec
DDR:  100,000 accesses x 8 cycles = 800,000 cycles/sec

Difference: 700,000 cycles/sec
At 400 MHz: 700,000 / 400,000,000 = 0.00175 seconds = 1.75 ms

This calculation shows the cycle cost of 100,000 memory accesses per second in both memory types. In DTCM, each access is 1 cycle, totaling 100,000 cycles. In DDR, each access is 8 cycles, totaling 800,000 cycles. The difference of 700,000 cycles per second, at a 400 MHz clock rate, translates to 1.75 milliseconds of additional CPU time spent waiting on memory.

If you're running a real-time control loop at 1 kHz (1 ms period), 1.75 ms of additional memory latency per second means that some individual iterations are running longer than their 1 ms budget. Whether this causes actual deadline misses depends on how the accesses are distributed across iterations and how much slack you have in your time budget, but it shows why memory placement decisions have real consequences in embedded systems.

What is DDR (Double Data Rate) Memory?

DDR is external memory. It sits on the circuit board outside the processor die, connected through a memory controller. It's much larger than TCM (typically 4 MB to several GB), but significantly slower to access.

The name "Double Data Rate" refers to how data is transferred between the DDR chip and the memory controller: data is sent on both the rising edge and the falling edge of the clock signal, effectively doubling the transfer rate compared to a single-data-rate design. But this doesn't eliminate the latency of activating rows and columns inside the DDR chip, which is where the slowness comes from.

How DDR Access Works

When your CPU reads from DDR, a multi-step process occurs inside the memory controller and DDR chip.

First, the CPU sends an address request to the memory controller. The memory controller is a hardware block inside the processor that translates CPU addresses into the specific row and column addresses that the DDR chip understands.

Second, the memory controller activates the correct row inside the DDR chip. This step is called the RAS (Row Address Strobe) phase. The DDR chip is organized as a grid of tiny capacitors, and "activating a row" means reading all the capacitors in that row into a row buffer inside the DDR chip. This takes several clock cycles.

Third, the memory controller selects the correct column within the activated row. This is called the CAS (Column Address Strobe) phase. The DDR chip uses the column address to pick the right bits out of the row buffer. This also takes several clock cycles.

Fourth, the data is transferred back to the memory controller, and from there to the CPU. The data transfer happens on both clock edges (the "double data rate" part), which helps with throughput but doesn't reduce the initial latency of the RAS and CAS phases.

The total latency depends on what state the memory is in when the request arrives. If the correct row is already activated from a previous access (a "row hit"), the RAS phase can be skipped, and the access is faster. If a different row is active and needs to be closed (precharged) before the new row can be opened (a "row miss"), the access takes longer. If the DDR chip happens to be performing a refresh cycle at that moment, the access is delayed further.

In practice, DDR access latency ranges from about 5 to 20+ CPU clock cycles, depending on the access pattern and timing.

Why DDR is Necessary

Because firmware often doesn't fit in TCM alone. Real embedded projects include protocol stacks, connectivity libraries, file system drivers, debug interfaces, and more. TCM is typically 2 to 3.5 MB total (ITCM + DTCM combined), and a full-featured firmware image can easily exceed that.

A real example showing memory usage before and after adding a wireless connectivity stack:

Without connectivity stack:
    ITCM:      506,996 B     (24.18%)
    DTCM:      628,408 B     (39.96%)
    DDR:       558,779 B     (13.32%)

With connectivity stack:
    ITCM:      570,936 B     (27.22%)
    DTCM:      727,240 B     (46.24%)
    DDR:       622,915 B     (14.85%)

Delta:
    ITCM: +63,940 B   (~62 KB of additional code)
    DTCM: +98,832 B   (~96 KB of additional data)
    DDR:  +64,136 B   (~62 KB of additional data/code)

This comparison shows memory usage from the same project built with and without a wireless connectivity stack.

The "Without" rows show the baseline. The "With" rows show the usage after adding the connectivity feature. The "Delta" rows show the difference.

Adding this single feature consumed an extra ~220 KB across all three memory regions. The time-critical parts of the stack (interrupt handlers, buffer management) went into ITCM and DTCM. The rest (packet parsers, connection management, configuration logic) went into DDR where it doesn't need single-cycle performance.

What Belongs in DDR?

Initialization and configuration code is the easiest category. Functions that run once at boot, like parsing a configuration file, initializing peripherals, or setting up data structures, don't need fast execution. They run once, take a few extra milliseconds because of DDR latency, and then never run again. Nobody notices. Put them in DDR and save TCM space for the code that runs a million times per second.

Large buffers must go in DDR because they simply can't fit in TCM. An image framebuffer for a 320x240 display at 16 bits per pixel is 150 KB. A network packet pool might be 32 KB or more. A file system cache might be 64 KB. These buffers would consume a significant fraction of DTCM's total capacity, leaving no room for the stack and variables that actually need single-cycle access.

Infrequently accessed data belongs in DDR as well. Calibration tables that are loaded once at boot and then read occasionally during operation, string tables for debug messages that are only printed during development or error conditions, and error description tables are all fine in DDR. The extra latency per access is irrelevant when the access count is low.

Non-time-critical code rounds out the DDR category. Protocol stacks (Bluetooth, Wi-Fi, TCP/IP), file system drivers, OTA update handlers, and shell/debug command interpreters all do important work, but none of them need to execute in a single clock cycle per instruction. They can tolerate the higher latency of DDR without affecting system behavior.

How to Place Code and Data in DDR

__attribute__((section(".ddr_text")))
void parse_config_file(const char *path) {
    // Runs from DDR, slower instruction fetch,
    // but config parsing happens once at boot,
    // so the latency does not affect runtime performance.
}

__attribute__((section(".ddr_bss")))
static uint8_t network_packet_pool[32768];

__attribute__((section(".ddr_bss")))
static uint8_t framebuffer[320 * 240 * 2];  // 150 KB, far too large for TCM

In this code, parse_config_file is placed in the .ddr_text section, which the linker script maps to DDR. Every instruction in this function will be fetched from DDR at multi-cycle latency, but since config parsing happens once at boot, the extra time is negligible.

The network_packet_pool is a 32 KB buffer placed in .ddr_bss. The .bss suffix is a convention indicating that this is zero-initialized data (the linker will ensure the memory is zeroed at startup rather than storing 32 KB of zeros in the firmware image). This buffer is used for network packet storage, which is not time-critical enough to justify DTCM space.

The framebuffer is a 150 KB buffer (320 pixels wide, 240 pixels tall, 2 bytes per pixel) also placed in .ddr_bss. At 150 KB, this single buffer would consume about 10% of DTCM's total capacity, which is far too expensive when the display update isn't a hard real-time operation.

How They Compare: A Side-by-Side Overview

This table summarizes the key differences between the three memory types. The most important columns are "Access latency" and "Typical size," because they represent the fundamental tradeoff: TCM is fast but small, DDR is slow but large.

The "Technology" column explains why: TCM uses SRAM (static RAM), which stores each bit using a flip-flop circuit that holds its state as long as power is applied. DDR uses DRAM (dynamic RAM), which stores each bit as charge in a tiny capacitor. Because capacitors leak charge, DRAM must be periodically refreshed, which adds power consumption and introduces occasional access delays when a refresh cycle coincides with a read request.

The Memory Map

Address Space:
  +------------------------------+  0x00000000
  |                              |
  |         ITCM (2 MB)          |  Single-cycle Inst Fetch
  |    ISRs, real-time loops,    |
  |    DSP, critical code        |
  |                              |
  +------------------------------+  0x00200000
  |       (reserved/gap)         |
  +------------------------------+  0x20000000
  |                              |
  |       DTCM (~1.5 MB)         |  Single-cycle Data Access
  |    Stack, hot variables,     |
  |    lookup tables, DMA bufs   |
  |                              |
  +------------------------------+  0x20180000
  |       (reserved/gap)         |
  +------------------------------+  0x80000000
  |                              |
  |         DDR (4 MB)           |  Multi-cycle Access
  |    Large buffers, init code, |
  |    protocol stacks, config   |
  |                              |
  +------------------------------+  0x80400000

This diagram shows the CPU's address space laid out from low addresses at the top to high addresses at the bottom. ITCM occupies the lowest 2 MB starting at address 0x00000000. After a gap of reserved/unused address space, DTCM sits at 0x20000000 and spans about 1.5 MB. Another gap of reserved space follows, and then DDR starts at 0x80000000 with 4 MB of space.

The gaps between regions are important. They're reserved address ranges that don't map to any physical memory. If your code accidentally reads from or writes to an address in one of these gaps, the result depends on the chip's bus fault configuration: it might trigger a HardFault exception, or it might silently return garbage data.

These addresses are illustrative. Every chip has its own memory map, documented in its Technical Reference Manual (TRM). Always consult your chip's TRM for the exact addresses and sizes.

How to Decide Where to Place Code and Data

Is it code or data?
|
+-- CODE (instructions):
|   +-- Called from an ISR or runs in a real-time loop?
|   |   +-- YES -> ITCM (deterministic timing is critical)
|   +-- Called frequently in the main processing pipeline?
|   |   +-- YES -> ITCM (if space is available)
|   +-- Called rarely (init, config, debug)?
|       +-- DDR (save ITCM space for critical code)
|
+-- DATA (variables, buffers, tables):
    +-- Accessed in an ISR or real-time context?
    |   +-- YES -> DTCM (single-cycle, deterministic)
    +-- Small and frequently accessed?
    |   +-- YES -> DTCM (if space is available)
    +-- Large buffer (>16 KB)?
    |   +-- Probably DDR (DTCM cannot afford the space)
    +-- Accessed only once at boot or very rarely?
        +-- DDR (do not use DTCM for this)

This decision tree captures the thought process for placing each piece of firmware into the right memory region.

Start by asking whether you're placing code (instructions) or data (variables, buffers, tables). For code, the primary question is how often it runs and whether it has timing constraints. ISR code and real-time loop code goes in ITCM. Everything else goes in DDR. For data, the primary question is how often it's accessed and how large it is. Small, frequently accessed data goes in DTCM. Large buffers and rarely-accessed data go in DDR.

The general principle: put the hottest code and data in TCM, and everything else in DDR. "Hot" means frequently accessed, latency-sensitive, or requiring deterministic timing. When in doubt, start with DDR placement and move things to TCM only when profiling shows it's necessary. It's much easier to promote a function from DDR to ITCM after discovering it's a bottleneck than to cram everything into ITCM from the start and run out of space.

How the Linker Script Controls Memory Placement

Everything we've discussed so far (section attributes, memory placement, address assignments) comes together in the linker script. This is a file (usually with a .ld extension) that tells the linker exactly which sections go into which memory regions. The linker script is the single source of truth for your firmware's memory layout.

MEMORY
{
    ITCM    (rx)  : ORIGIN = 0x00000000, LENGTH = 2M
    DTCM    (rw)  : ORIGIN = 0x20000000, LENGTH = 1536K
    DDR     (rwx) : ORIGIN = 0x80000000, LENGTH = 4M
}

SECTIONS
{
    /* === ITCM: Critical code === */
    .itcm_text :
    {
        KEEP(*(.isr_vector))          /* Interrupt vector table */
        *(.itcm_text)                 /* Functions with __attribute__((section(".itcm_text"))) */
        *audio_processing.o(.text)    /* All code from audio_processing.c */
        *motor_control.o(.text)       /* All code from motor_control.c */
    } > ITCM

    /* === DDR: Non-critical code === */
    .ddr_text :
    {
        *(.text)                      /* Default catch-all for remaining code */
        *(.text*)
        *(.rodata)                    /* Read-only data (string literals, constants) */
        *(.rodata*)
    } > DDR

    /* === DTCM: Critical data === */
    .dtcm_data :
    {
        *(.dtcm_data)                 /* Data with __attribute__((section(".dtcm_data"))) */
        *audio_processing.o(.data)    /* All initialized data from audio_processing.c */
        *audio_processing.o(.bss)     /* All zero-initialized data from audio_processing.c */
    } > DTCM

    /* === DTCM: Stack === */
    .stack (NOLOAD) :
    {
        . = ALIGN(8);
        __stack_start = .;
        . = . + 8K;                  /* 8 KB stack */
        __stack_end = .;
    } > DTCM

    /* === DDR: Everything else === */
    .ddr_data :
    {
        *(.data)                      /* Default catch-all for remaining initialized data */
        *(.bss)                       /* Default catch-all for remaining zero-initialized data */
        *(COMMON)
    } > DDR
}

This linker script has two main blocks: MEMORY and SECTIONS.

The MEMORY block defines the physical memory regions available on the chip. Each line declares a region name, its permissions (rx for read-execute, rw for read-write, rwx for read-write-execute), its starting address (ORIGIN), and its size (LENGTH). These values must match your chip's actual memory map as documented in its reference manual.

The SECTIONS block defines how the linker should distribute compiled code and data across those memory regions. Each section rule consists of a section name (like .itcm_text), a list of input patterns that specify which object file sections to include, and a > REGION directive that tells the linker which memory region to place the output section in.

The .itcm_text section collects the interrupt vector table (KEEP(*(.isr_vector))), any functions explicitly marked with __attribute__((section(".itcm_text"))), and all code from audio_processing.o and motor_control.o. The KEEP directive prevents the linker from discarding the interrupt vector table during garbage collection, even if no code appears to reference it directly. All of this goes into ITCM.

The .ddr_text section uses catch-all patterns *(.text) and *(.text*) to collect all remaining code that wasn't claimed by the ITCM section above. It also collects read-only data (.rodata), which includes string literals and const variables. All of this goes into DDR.

The .dtcm_data section collects explicitly-placed data and all data from audio_processing.o. The .stack section reserves 8 KB for the stack with 8-byte alignment, and exports the __stack_start and __stack_end symbols that your startup code and stack profiling code can reference. Both go into DTCM.

The .ddr_data section collects all remaining data with catch-all patterns, and goes into DDR.

How Section Matching Works

The linker processes sections from top to bottom. When it encounters a wildcard pattern like *(.text), it matches all .text sections that haven't already been claimed by a more specific rule earlier in the script.

So in the example above, *audio_processing.o(.text) in the ITCM section claims all code from audio_processing.c first. Then, when the linker reaches *(.text) in the DDR section, audio_processing.o's .text section has already been placed, so it's skipped. Only unclaimed .text sections from other object files match the DDR catch-all.

This means the order of sections in your linker script matters. Place your specific rules (individual object files, named sections) before the generic catch-all rules. If you put the *(.text) catch-all before the *audio_processing.o(.text) rule, the catch-all would claim everything first, and the specific rule would match nothing.

Common Mistakes to Avoid

1. Stack Overflow in DTCM

Your stack lives in DTCM. DTCM is small. If you declare a large local array inside a function, it goes on the stack:

void problematic_function(void) {
    uint8_t huge_local_buffer[65536];  // 64 KB allocated on the stack
    // This consumes 64 KB of DTCM immediately
}

This code declares a 64 KB local array. Because it's a local variable (not static), it is allocated on the stack when the function is called. If your total stack size is 8 KB (as in the linker script example above), this single declaration overflows the stack by 56 KB, writing into whatever memory is adjacent to the stack in DTCM.

On a desktop OS, a stack overflow triggers a segmentation fault because the OS uses virtual memory and guard pages to detect it.

In an embedded system without memory protection, the stack silently grows into adjacent memory regions, corrupting whatever data is stored there. The resulting bugs are extremely difficult to diagnose because the symptoms (corrupted variables, erratic behavior, intermittent crashes) appear unrelated to the actual cause. You might spend days debugging a seemingly random data corruption issue before realizing the root cause is a stack overflow from a function three call levels deep.

The fix: Use static allocation or heap allocation for large buffers, and place them in DDR:

void fixed_function(void) {
    __attribute__((section(".ddr_bss")))
    static uint8_t huge_buffer[65536];  // In DDR, not on the stack

    // Stack is safe, DTCM is not wasted
}

By making the buffer static, it's no longer allocated on the stack. Instead, the linker allocates it once in the .ddr_bss section, which maps to DDR. The buffer persists for the entire lifetime of the program (like a global variable), but its name is scoped to this function. The stack only holds a pointer to the buffer, which is a few bytes instead of 64 KB.

2. Overfilling ITCM

If you exceed ITCM's capacity, the linker will produce an error along the lines of "region ITCM overflowed by N bytes." But if you're close to the limit, you're one library update or feature addition away from a build failure. A minor version bump of your RTOS or connectivity stack could add enough code to push ITCM over the edge.

Keep headroom. The 27% utilization shown earlier is healthy. If you're above 85%, you should actively work on moving less-critical code to DDR. If you're above 95%, you have no room for growth and need to make immediate changes. Setting up automated memory budget checks in your CI pipeline (covered later in this article) prevents surprises.

3. Ignoring Alignment Requirements

TCM memories often have alignment requirements. On Cortex-M processors with strict alignment enforcement, accessing a 32-bit value at an unaligned address causes a HardFault exception.

/* Problematic: packed struct can create unaligned fields */
__attribute__((section(".dtcm_data"), packed))
struct badly_aligned {
    uint8_t  flag;
    uint32_t counter;  // May be at byte offset 1, unaligned
};

/* Correct: natural alignment, with minor padding */
__attribute__((section(".dtcm_data")))
struct properly_aligned {
    uint32_t counter;  // At offset 0, 4-byte aligned
    uint8_t  flag;     // At offset 4
    // 3 bytes of padding follow, a small cost for correctness
};

In the first struct, the packed attribute tells the compiler to use no padding between fields. This means counter starts at byte offset 1 (right after the 1-byte flag), which isn't a multiple of 4. When the CPU tries to read a 32-bit value from a non-4-byte-aligned address in TCM, it triggers a HardFault on processors with strict alignment (which includes most Cortex-M cores).

In the second struct, the fields are ordered so that counter (4 bytes) comes first at offset 0, which is naturally 4-byte aligned. The flag (1 byte) follows at offset 4. The compiler inserts 3 bytes of padding after flag to bring the struct size to 8 bytes (a multiple of 4), but this is a small price for correct, crash-free operation.

4. DMA Transfers to TCM on Incompatible Bus Architectures

Some DMA controllers can't access TCM memory. Whether DMA can reach TCM depends entirely on your chip's internal bus architecture (the bus matrix).

If you set up a DMA transfer from a peripheral to a DTCM buffer, but the DMA controller doesn't have a bus path to DTCM, the transfer will either silently fail or write to an incorrect address.

Neither produces an obvious error. The DMA controller thinks it completed successfully, your code reads the buffer expecting fresh data, and you get stale or garbage values instead. This is one of the most confusing bugs in embedded development because everything looks correct in the code.

Always check your chip's bus matrix diagram in the reference manual before using DMA with TCM buffers. The bus matrix diagram shows which masters (CPU, DMA, USB, and so on) can access which slaves (ITCM, DTCM, SRAM, DDR, peripherals). Look for whether the DMA controller's master port has a connection line to the TCM slave port. If it doesn't, your DMA transfers to TCM will not work.

Performance Comparison With Real Numbers

The following table compares access latencies across memory types, assuming a Cortex-R class processor at 400 MHz:

+---------------------+----------+----------+----------+
| Operation           | ITCM/    |   DDR    | Slowdown |
|                     | DTCM     |          | Factor   |
+---------------------+----------+----------+----------+
| Instruction fetch   | 1 cycle  | 5-20 cyc |   5-20x  |
| Data read (32-bit)  | 1 cycle  | 5-20 cyc |   5-20x  |
| Data write (32-bit) | 1 cycle  | 5-20 cyc |   5-20x  |
| Sequential burst    | 1 cyc/wd | 2-4 cy/wd|    2-4x  |
| Random access       | 1 cycle  | 10-20 cyc|  10-20x  |
+---------------------+----------+----------+----------+

This table shows the latency for five different types of memory operations. The first three rows (instruction fetch, data read, data write) show that individual accesses to TCM are always 1 cycle, while individual accesses to DDR range from 5 to 20 cycles depending on the memory's internal state. The slowdown factor is the ratio between the two.

The "Sequential burst" row shows what happens when you read or write consecutive addresses. DDR performs much better in burst mode (2-4 cycles per word instead of 5-20) because once a row is activated, subsequent reads from the same row skip the RAS phase. TCM is still 1 cycle per word because it doesn't have the row/column structure of DDR.

The "Random access" row shows the worst case for DDR. When each access hits a different row, the memory controller must precharge the old row and activate the new one every time. This is the 10-20 cycle range, and it's common in workloads that jump around in memory (traversing linked lists, hash table lookups, and indirect function calls through function pointer arrays).

The practical takeaway: if your code accesses DDR data, try to access it sequentially. Iterating through an array in order is much faster than jumping to random positions. Your memory controller and the DDR chip's internal prefetch logic work in your favor during sequential access patterns.

How TCM Affects Power Consumption

Memory placement has a direct impact on power consumption, something that becomes critical for battery-powered products.

DDR requires constant refresh cycles. DRAM stores each bit as charge in a tiny capacitor, and that charge leaks over time.

To prevent data loss, the memory controller must read and rewrite every row in the DDR chip approximately every 64 ms. This refresh process consumes power even when the processor is sleeping and no code is running. On some systems, DDR refresh can account for a significant portion of the total sleep-mode power budget.

TCM is SRAM-based and doesn't require refresh. SRAM stores data using flip-flop circuits that hold their state as long as power is applied. There is some leakage current (no transistor is perfect), but it is orders of magnitude lower than DDR refresh power.

For battery-powered devices (wearables, IoT sensors, medical devices), this means you should keep data that must survive sleep modes in DTCM when possible.

If your hardware supports it, power-gate the DDR chip during deep sleep to eliminate its refresh power entirely. The less DDR your firmware uses at runtime, the more aggressively you can manage DDR power states, which directly extends battery life.

How to Profile Memory Usage

After placing code and data into ITCM, DTCM, and DDR, you need to verify that everything fits, monitor usage over time, and catch regressions before they become build failures. There are several techniques for this, ranging from simple command-line tools to automated CI checks.

Method 1: The Linker Map File

Every time you build your firmware, the linker can produce a map file, a detailed text file that records where every symbol (function, variable, constant) ended up and how large it is. This is the most useful single artifact in embedded development for understanding memory usage.

To generate one, add -Wl,-Map=output.map to your linker flags:

arm-none-eabi-gcc \
    -T linker_script.ld \
    -Wl,-Map=firmware.map \
    -o firmware.elf \
    main.o audio.o bluetooth.o

This command invokes the ARM GCC toolchain to link three object files (main.o, audio.o, bluetooth.o) using the linker script linker_script.ld. The -Wl,-Map=firmware.map flag tells GCC to pass the -Map=firmware.map option to the linker, which causes it to write a detailed map file alongside the output ELF binary. The map file can be thousands of lines long, but the most useful part is the summary at the end.

The summary at the end of the map file shows overall utilization per memory region:

Memory region         Used Size  Region Size  %age Used
            ITCM:      570936 B         2 MB     27.22%
            DTCM:      727240 B    1572608 B     46.24%
             DDR:      622915 B         4 MB     14.85%

This summary shows three columns: how many bytes are used, the total size of the region, and the percentage used. It gives you the health of your firmware at a glance. As a rule of thumb, below 80% is healthy with room for growth. Between 80% and 90% is getting tight, and you should plan for how you will accommodate the next feature. Above 90% requires action: start moving things to a cheaper memory region or optimizing existing placement.

Method 2: Parsing the Map File for Per-Module Breakdown

The summary tells you how much memory is used, but not who is using it. The map file contains per-symbol details, but they're difficult to read manually because the file can be thousands of lines long with a format that isn't designed for human consumption.

The following Python script parses the map file and produces a per-module report showing which object files are consuming memory in which regions.

#!/usr/bin/env python3
"""Parse a linker map file and report memory usage per object file."""

import re
import sys
from collections import defaultdict

def parse_map_file(map_path):
    """Extract symbol placements from a GCC linker map file."""
    usage = defaultdict(lambda: defaultdict(int))

    regions = {
        'ITCM': (0x00000000, 0x00200000),
        'DTCM': (0x20000000, 0x20180000),
        'DDR':  (0x80000000, 0x80400000),
    }

    def addr_to_region(addr):
        for name, (start, end) in regions.items():
            if start <= addr < end:
                return name
        return 'UNKNOWN'

    symbol_re = re.compile(
        r'^\s+\S+\s+(0x[0-9a-fA-F]+)\s+(0x[0-9a-fA-F]+)\s+(\S+\.o)'
    )

    with open(map_path) as f:
        for line in f:
            m = symbol_re.match(line)
            if m:
                addr = int(m.group(1), 16)
                size = int(m.group(2), 16)
                obj = m.group(3).split('/')[-1]
                region = addr_to_region(addr)
                usage[obj][region] += size

    return usage

def print_report(usage):
    """Print a sorted memory usage report."""
    print(f"{'Object File':<35} {'ITCM':>10} {'DTCM':>10} {'DDR':>10} {'Total':>10}")
    print("-" * 80)

    totals = defaultdict(int)
    rows = []

    for obj, regions in usage.items():
        total = sum(regions.values())
        rows.append((obj, regions, total))
        for r, s in regions.items():
            totals[r] += s

    rows.sort(key=lambda x: x[2], reverse=True)

    for obj, regions, total in rows[:20]:
        print(f"{obj:<35} "
              f"{regions.get('ITCM', 0):>10,} "
              f"{regions.get('DTCM', 0):>10,} "
              f"{regions.get('DDR', 0):>10,} "
              f"{total:>10,}")

    print("-" * 80)
    grand = sum(totals.values())
    print(f"{'TOTAL':<35} "
          f"{totals.get('ITCM', 0):>10,} "
          f"{totals.get('DTCM', 0):>10,} "
          f"{totals.get('DDR', 0):>10,} "
          f"{grand:>10,}")

if __name__ == '__main__':
    usage = parse_map_file(sys.argv[1])
    print_report(usage)

This script does three things. First, parse_map_file reads the map file line by line, looking for lines that match the format of a symbol placement entry (a section name, an address, a size, and an object file name). For each match, it converts the hex address to an integer, determines which memory region it falls in using the addr_to_region helper, and accumulates the size into a nested dictionary keyed by object file and region.

Second, print_report sorts the object files by total memory usage (largest first), prints the top 20, and shows how much each one uses in each region.

Third, the if __name__ == '__main__' block makes the script runnable from the command line.

You'll need to adjust the address ranges in the regions dictionary to match your chip's memory map.

Run it with:

python3 parse_map.py firmware.map

Sample output:

Object File                              ITCM       DTCM        DDR      Total
--------------------------------------------------------------------------------
bluetooth_stack.o                      42,380     65,200     38,400    146,080
audio_processing.o                     89,200     32,000          0    121,200
wifi_driver.o                          21,560     33,632     25,736     80,928
sensor_hub.o                           45,000     18,400          0     63,400
libc.a(memcpy.o)                       12,340          0          0     12,340
...
--------------------------------------------------------------------------------
TOTAL                                 570,936    727,240    622,915  1,921,091

This output shows the top memory consumers in the firmware, sorted by total usage. Each row shows an object file and how many bytes it contributes to each memory region.

The bluetooth_stack.o file is the largest consumer at 146 KB total, spread across all three regions. The audio_processing.o file uses 121 KB, all in ITCM and DTCM (0 bytes in DDR), which makes sense because audio processing is time-critical and was placed entirely in TCM. The libc.a(memcpy.o) entry shows a C library function that was placed in ITCM, likely because it is called from performance-critical code paths.

Method 3: The size Command

For a quick check without parsing the map file, use arm-none-eabi-size:

arm-none-eabi-size -A firmware.elf

Output:

firmware.elf  :
section               size        addr
.itcm_text          570936           0
.dtcm_data          530240   536870912
.dtcm_bss           196000   537401152
.stack                8192   537600000
.ddr_text           422915  2147483648
.ddr_data           120000  2147906563
.ddr_bss             80000  2148026563
Total              1928283

This output lists every section in the ELF binary, its size in bytes, and its starting address (shown in decimal).

You can map sections to memory regions by looking at the address: addresses near 0 are ITCM, addresses near 536 million (0x20000000) are DTCM, and addresses near 2.1 billion (0x80000000) are DDR.

Alternatively, the section names themselves indicate the region (.itcm_text is in ITCM, .dtcm_data and .dtcm_bss are in DTCM, .ddr_text and .ddr_data and .ddr_bss are in DDR).

The -A flag gives per-section sizes instead of the default BSD-format output. It's less detailed than the map file approach, but it runs instantly and gives you the big picture.

Method 4: Runtime Stack Profiling

Static analysis (map files, size output) tells you about compile-time placement. But some memory usage is dynamic, particularly the stack, which grows and shrinks at runtime based on call depth and local variable sizes. A function that allocates a 2 KB local buffer only uses that stack space while it is executing, so static analysis can't tell you the peak stack usage.

A common technique is stack watermarking: fill the entire stack region with a known pattern at boot, then periodically check how much of the pattern has been overwritten.

#define STACK_FILL_PATTERN 0xDEADBEEF

void stack_watermark_init(void) {
    extern uint32_t __stack_start;
    extern uint32_t __stack_end;
    uint32_t *p = &__stack_start;

    register uint32_t sp asm("sp");
    while (p < (uint32_t *)(sp - 64)) {
        *p++ = STACK_FILL_PATTERN;
    }
}

uint32_t stack_usage_bytes(void) {
    extern uint32_t __stack_start;
    extern uint32_t __stack_end;
    uint32_t *p = &__stack_start;

    while (p < &__stack_end && *p == STACK_FILL_PATTERN) {
        p++;
    }

    return (uint32_t)(&__stack_end) - (uint32_t)p;
}

void check_stack_health(void) {
    uint32_t used = stack_usage_bytes();
    uint32_t total = 8192;
    uint32_t percent = (used * 100) / total;

    if (percent > 80) {
        log_warning("Stack usage: %lu / %lu bytes (%lu%%)",
                    used, total, percent);
    }
}

The stack_watermark_init function fills the stack memory (from __stack_start to just below the current stack pointer) with the pattern 0xDEADBEEF. The extern declarations reference the linker symbols defined in the linker script's .stack section. The register uint32_t sp asm("sp") line reads the current stack pointer value so the function knows where to stop filling (you do not want to overwrite your own stack frame). The 64-byte safety margin ensures the fill loop doesn't get too close to the active stack.

The stack_usage_bytes function scans from the bottom of the stack upward, counting how many words still contain the fill pattern. The first word that does not match the pattern indicates the deepest point the stack has reached (the high-water mark). The function returns the number of bytes from that point to the top of the stack.

The check_stack_health function computes the percentage of stack used and logs a warning if it exceeds 80%. Call this function periodically during normal operation to monitor stack usage.

Call stack_watermark_init() as early as possible in your startup code (before main() if you can), then call check_stack_health() periodically during normal operation. This tells you the high-water mark, the maximum stack depth your firmware has reached so far.

Method 5: Tracking Memory Across Builds

Every time you add a feature or merge a change, run the memory profile before and after:

arm-none-eabi-size -A firmware_before.elf > mem_before.txt
arm-none-eabi-size -A firmware_after.elf > mem_after.txt
diff mem_before.txt mem_after.txt

These three commands capture the section sizes of two firmware builds (before and after a change) into text files, then diff them to see what changed. This is useful but the raw diff output can be hard to read. The following script provides a cleaner view by computing the delta per memory region:

#!/bin/bash
# memory_diff.sh - Compare memory usage between two builds

echo "Memory Impact of Change:"
echo "========================"

parse_size() {
    arm-none-eabi-size -A "$1" | awk '
    /\.itcm/  { itcm += $2 }
    /\.dtcm/  { dtcm += $2 }
    /\.ddr/   { ddr += $2 }
    /\.stack/ { dtcm += $2 }
    END { printf "%d %d %d", itcm, dtcm, ddr }
    '
}

read itcm_before dtcm_before ddr_before <<< \((parse_size "\)1")
read itcm_after  dtcm_after  ddr_after  <<< \((parse_size "\)2")

printf "ITCM: %+d bytes (%d -> %d)\n" \
    \(((itcm_after - itcm_before)) \)itcm_before $itcm_after
printf "DTCM: %+d bytes (%d -> %d)\n" \
    \(((dtcm_after - dtcm_before)) \)dtcm_before $dtcm_after
printf "DDR:  %+d bytes (%d -> %d)\n" \
    \(((ddr_after - ddr_before)) \)ddr_before $ddr_after

This script takes two ELF files as arguments (the "before" and "after" builds). The parse_size function runs arm-none-eabi-size -A on the given ELF file and uses awk to sum up section sizes by memory region. Sections whose names contain .itcm are counted toward ITCM, sections containing .dtcm or .stack toward DTCM, and sections containing .ddr toward DDR. The main body reads the before and after values, then prints the delta for each region with a + or - sign.

Usage and output:

$ ./memory_diff.sh firmware_without_bt.elf firmware_with_bt.elf

Memory Impact of Change:
========================
ITCM: +63940 bytes (506996 -> 570936)
DTCM: +98832 bytes (628408 -> 727240)
DDR:  +64136 bytes (558779 -> 622915)

This output shows that adding the Bluetooth feature increased ITCM by about 62 KB, DTCM by about 96 KB, and DDR by about 62 KB. You can put this in your CI/CD pipeline so that every pull request shows exactly how much memory it costs.

Method 6: Automated Memory Budget Checks in CI

You can integrate memory profiling into your CI/CD pipeline to catch overflows before they land in your main branch.

#!/bin/bash
# memory_check.sh - Fail CI if memory usage exceeds thresholds

ITCM_LIMIT=85   # percent
DTCM_LIMIT=80
DDR_LIMIT=90

check_region() {
    local name=\(1 used=\)2 total=\(3 limit=\)4
    local percent=$((used * 100 / total))

    if [ \(percent -ge \)limit ]; then
        echo "FAIL: \(name usage is \){percent}% (limit: ${limit}%)"
        echo "      Used: \(used / \)total bytes"
        return 1
    else
        echo "OK:   \(name usage is \){percent}% (limit: ${limit}%)"
        return 0
    fi
}

ITCM_USED=\((grep "ITCM:" firmware.map | awk '{print \)2}')
ITCM_TOTAL=$((2 * 1024 * 1024))

DTCM_USED=\((grep "DTCM:" firmware.map | awk '{print \)2}')
DTCM_TOTAL=1572608

DDR_USED=\((grep "DDR:" firmware.map | awk '{print \)2}')
DDR_TOTAL=$((4 * 1024 * 1024))

FAILED=0
check_region "ITCM" \(ITCM_USED \)ITCM_TOTAL $ITCM_LIMIT || FAILED=1
check_region "DTCM" \(DTCM_USED \)DTCM_TOTAL $DTCM_LIMIT || FAILED=1
check_region "DDR"  \(DDR_USED  \)DDR_TOTAL  $DDR_LIMIT  || FAILED=1

exit $FAILED

This script reads memory usage numbers from the linker map file and compares them against configurable percentage thresholds. The check_region function takes a region name, the number of bytes used, the total bytes available, and the percentage limit. It computes the actual percentage and prints either "OK" or "FAIL" along with the numbers. If any region exceeds its limit, the script exits with a non-zero status, which causes the CI build to fail.

The thresholds at the top (85% for ITCM, 80% for DTCM, 90% for DDR) should be adjusted based on your project's growth rate and how much headroom you want to maintain. DTCM has a lower limit because it fills up faster and is harder to free up.

Add this script to your build pipeline so every pull request shows its memory cost. If a change pushes any region past its threshold, the build fails and the developer knows immediately.

Method 7: Heap Tracking at Runtime

If your embedded project uses dynamic memory allocation (malloc/free), you can wrap the allocator to track usage.

static size_t heap_used = 0;
static size_t heap_peak = 0;

void *tracked_malloc(size_t size) {
    size_t *block = (size_t *)malloc(size + sizeof(size_t));
    if (!block) return NULL;

    *block = size;
    heap_used += size;
    if (heap_used > heap_peak) {
        heap_peak = heap_used;
    }

    return (void *)(block + 1);
}

void tracked_free(void *ptr) {
    if (!ptr) return;
    size_t *block = ((size_t *)ptr) - 1;
    heap_used -= *block;
    free(block);
}

void print_heap_stats(void) {
    printf("Heap: current=%zu bytes, peak=%zu bytes\n",
           heap_used, heap_peak);
}

This code wraps malloc and free with tracking logic. The tracked_malloc function allocates slightly more memory than requested (an extra sizeof(size_t) bytes) and stores the requested size in the first word of the allocation. It then updates the heap_used counter and, if the new total exceeds the previous peak, updates heap_peak. It returns a pointer that's offset past the size header, so the caller sees a normal pointer to their data.

The tracked_free function reverses the process: it subtracts one size_t from the pointer to find the hidden size header, subtracts that size from heap_used, and calls the real free on the original block.

The print_heap_stats function prints the current and peak heap usage. Call it periodically or on demand through a debug interface (UART console, debug CLI) to monitor how much heap your firmware is using.

This approach has a small overhead (one extra word per allocation), but it gives you visibility into dynamic memory usage that's otherwise completely invisible. It's especially useful for tracking down memory leaks: if heap_used keeps growing over time without ever decreasing, something is allocating without freeing.

Summary

Embedded processors based on ARM Cortex-M and Cortex-R architectures give you direct control over three memory regions with very different performance characteristics.

ITCM (Instruction Tightly-Coupled Memory) stores your most performance-critical code. It provides single-cycle, deterministic instruction fetch. It's small (typically 512 KB to 2 MB), so reserve it for ISRs, real-time processing functions, and hot loops.

DTCM (Data Tightly-Coupled Memory) stores your most performance-critical data. It also provides single-cycle, deterministic access. Your stack lives here by default. It's even smaller than ITCM and fills up quickly, so be deliberate about what you place in it.

DDR (Double Data Rate) memory stores everything else. It's much larger but slower (5 to 20+ cycles per access, with variable latency). Use it for initialization code, large buffers, protocol stacks, and anything that doesn't need deterministic timing.

You control placement through __attribute__((section(...))) in your C code and section-to-region mappings in your linker script. You verify placement through map files, the size command, and runtime profiling techniques like stack watermarking. The core skill is knowing which region each piece of your firmware belongs in, and having the tooling to catch mistakes early.

Go follows the same model, but you usually don’t decide between the stack and the heap directly. Instead, the Go compiler decides where values live. If the compiler can prove a value is only needed within the current function call, it can keep it on the stack. If it cannot prove that, the value “escapes” and is placed on the heap. This technique is called escape analysis.

This matters because heap allocations increase garbage collector work. In code that runs often, that extra work can show up as more CPU spent in GC, more allocations, and less predictable performance.

In this article, I’ll explain what escape analysis is, the common patterns that trigger heap allocation, and how to confirm and reduce avoidable allocations.

Table of Contents

• Prerequisites

• Do You Really Need to Care About Escape Analysis?

• Memory Layout and Lifecycle

• Sharing Down and Sharing Up

• Escape Analysis in Practice

• How to Use Escape Analysis to Guide Performance

• Conclusion

• Further Reading

Prerequisites

• Familiarity with Go fundamentals (functions, variables, structs, slices, maps)

• Basic understanding of pointers in Go (& and *)

• A general idea of how goroutines work

Do You Really Need to Care About Escape Analysis?

Before we go deeper, I want to call this out clearly. For the correctness of your program, it doesn’t matter whether a variable lives on the stack or on the heap, or whether you know that detail. The Go compiler is smart enough to place values where they need to be so that your program behaves correctly.

Most of the time, you don’t need to think about this at all. It only starts to matter when performance becomes a problem. If your program is already fast enough, you’re done, and there’s no point trying to squeeze out extra speed.

You should only start caring about stack vs heap when you have benchmarks that show your program is too slow, and those same benchmarks point to heavy heap allocation and garbage collection as part of the problem.

Memory Layout and Lifecycle

To get a better understanding of what escape analysis is, you first need a simple picture of how Go lays out memory while your program runs. At this level, it comes down to the stack each goroutine uses, how stack frames are carved out of that stack, and when values move to the heap where the garbage collector can see them.

Goroutine Stacks and Stack Frames

When a Go program starts, the runtime creates the main goroutine, and every go statement creates a new goroutine, each with its own stack.

There’s not a single global stack for the whole process. As of writing this article, with Go v1.25.7, each goroutine gets an initial contiguous block of 2,048 bytes of memory, which acts as its stack. The stack is where Go stores data that belongs to function calls. When a goroutine calls a function, Go reserves a chunk of that goroutine’s stack for the function’s local data. That chunk is called a stack frame.

It holds the function’s local variables and the call state needed to return and continue execution. If that function calls another function, a new frame is added on top. When the inner function returns, its frame becomes invalid, and the goroutine continues in the caller’s frame.

A stack frame only lives for as long as the function is active. Once the function returns, anything stored in its frame is considered invalid, even if the raw bytes are still in memory and will be reused later. Code must not rely on those values after the return

Go stacks can grow. A goroutine starts with a small stack and the runtime grows it when needed, but the lifetime rule stays the same. A value is safe in a stack frame only if nothing can still reference it after the function returns. If it might be referenced later, it can’t stay in that frame and must be placed somewhere safer.

Pointers and Lifetime

In Go, taking an address like p := &x means you now have a pointer in one stack frame that refers to a value which may have been created in another frame. When you pass that pointer into a function, Go still passes by value. The callee gets its own pointer variable on its own stack frame, but the address inside still points to the same underlying value. So pointers are how you share access to one value across several frames without copying the value itself.

Lifetime becomes important when a pointer can outlive the frame where the pointed value was created. As long as both the pointer and the value live inside frames that are still active in the current call stack, everything is safe.

Once a pointer might still exist after the original frame has returned, the value can no longer stay in that frame, because that frame will become invalid. At that point, the value has to be placed in a safer location so that no pointer ever points into dead stack memory.

Sharing Down and Sharing Up

Now that you have a picture of stacks, frames, and pointers, we can look at two common ways pointers move through your code. I’ll call them sharing down and sharing up. The names aren’t special Go terms. They’re just a simple way to describe how a pointer moves along the call stack.

Sharing Down

Sharing down means a function passes a pointer or reference to functions it calls. The pointer moves deeper into the call stack, but the value it points to still belongs to a frame that is active.

Example code:

package main

import "fmt"

func main() {
    n := 10
    multiply(&n) 
}

func multiply(v *int) {
   *v = *v * 2
}

In main, you take the address of n and pass it into multiply. While multiply runs, both the main frame and the multiply frames are active. The pointer in multiply points to a value that still lives in an active frame, so this situation is safe from a lifetime point of view.

In the diagram below, after the multiply function runs and returns, the multiply frame becomes invalid, and we don’t need to do anything because the stack pointer is simply popped back to the previous frame's address. This action automatically reclaims all the memory used by that function in one step, so the garbage collector is not involved in cleaning up stack memory

Sharing Up

Sharing up means a function returns a pointer, or stores it somewhere that will still be around after the function returns. The pointer moves back up the call stack or into some longer-lived state while the frame that created the value is about to end, so that value can no longer be tied to that one frame.

The same idea shows up when you share a value with another goroutine, because Go doesn’t let one goroutine hold pointers into another goroutine’s stack, so shared data needs a lifetime that is not tied to a single stack.

Heap, garbage collection, and lifetime

Values that might outlive a single stack frame can’t stay in that frame. The compiler places them on the heap instead. The heap is a separate region of memory that isn’t tied to one function call. Any goroutine can hold pointers to heap values, and those values stay valid as long as something in the program can still reach them. You can think of the heap as storage for “might live longer than this call”.

The garbage collector is what keeps this safe. Periodically, the runtime starts from a set of roots (global variables, active stack frames, some internal state) and follows all the pointers it can see. Any heap value that is still reachable is kept. Any heap value that is no longer reachable is treated as garbage and its memory is reclaimed.

This means a pointer in main will never legally point into dead stack memory. Either the value stayed in an active frame, or it was placed on the heap where the GC can track its lifetime. The tradeoff is that more heap allocations and longer-lived objects require the GC to do more work.

Here’s an example:

package main

import "fmt"

type Car struct {
    Brand string
    Model string
}

func main() {
    // main receives a pointer from a function it called and this is sharing up
    carPtr := makeCar("Volkswagen", "Golf") 

    fmt.Printf("I received a car: %s %s\n", carPtr.Brand, carPtr.Model)
}

func makeCar(b, m string) *Car {
    myCar := Car{
        Brand: b,
        Model: m,
    }
    return &myCar
}

In the above code:

1. In makeCar (the callee frame), Go creates a local variable myCar. Because you return &myCar, the compiler allocates the Car value on the heap, and let’s myCar hold the heap address 0xc00029fa0.

2. When makeCar returns, that address is copied into carPtr in main (the top frame). carPtr is just another stack variable, but its value is still 0xc00029fa0, so now main also points to the same heap Car.

3. On the right, the heap bubble shows the actual Car value at 0xc00029fa0. Both car (while makeCar is running) and carPtr (after it returns) reach that same value through their pointers.

4. Once makeCar is done, its frame drops into the “invalid memory” region, but the Car stays alive on the heap because main still holds carPtr. That’s the escape: the value stops being tied to the callee frame and gets heap lifetime instead.

Escape Analysis in Practice

Escape analysis is how the Go compiler decides whether a value lives on the stack or on the heap. It’s not only about returning pointers – it follows how addresses move through your code. If a value might outlive the current function, the compiler can’t keep it in that stack frame and moves it to the heap. Since only the compiler sees the full picture, the useful thing is to ask it to show these decisions and then link them back to your code.

To do that, we can pass compiler flags using -gcflags when running go build or go run. If you want to see the available options, you can check go tool compile -h. In that list, -m prints the compiler’s optimisation decisions, including escape analysis output. If you want more details, you can use -m=2 or -m=3 for a more verbose output. The -l flag disables inlining, so the report is easier to read because the compiler is not merging small functions into their callers.

So, the command will look like this:

go run -gcflags='all=-m -l' .

Or for a build:

go build -gcflags='all=-m -l' .

How to Use Escape Analysis to Guide Performance

You can think of escape analysis as the thing that turns your code choices into GC work. When a value escapes, it gets heap lifetime, and the garbage collector has to visit it. In hot paths, lots of small escaping values show up as extra GC time and jitter in latency. When a value stays in a stack frame, it becomes invalid and dies with the frame and the GC does not care about it.

Here are five simple practices that help performance without making

1. Prefer values for small data: If the function doesn’t need to mutate the caller’s data, use value types for small structs and basic types when passing arguments and returning results. It’s cheap to copy an int or a small struct, and it often keeps lifetimes local to a single call.

2. Use pointers when sharing or mutation is part of the design: opt for pointers when you genuinely need shared mutable state or want to avoid copying large structs.

3. Avoid creating long-lived references by accident: Be careful when returning pointers to locals, capturing variables in closures, or storing addresses in long-lived structs, maps, or interfaces. These patterns are the ones most likely to push values out of a stack frame.

4. Pass in reusable buffers on hot paths: On code paths that run very often, the problem is usually not one big allocation, but many small ones happening in a loop. A common cause is functions that always create a new buffer inside, even when the caller could have passed one in.

   A simple way to cut those extra allocations is to let the caller own the buffer. The caller allocates a []byte once, then passes it into the function each time. The function only fills the buffer instead of creating a new one.

   Here’s an example of how a bad function allocates a new buffer every call:

    package main
   
    // Bad: helper allocates every call.
    func fillBad() []byte {
        buf := make([]byte, 4096)
        // pretend we read into it
        buf[0] = 1
        return buf
    }
   
    func hotPathBad() {
        for i := 0; i < 1_000_000; i++ {
            b := fillBad() // allocates 1,000,000 times
            _ = b
        }
    }
   
    func main() {
        hotPathBad()
    }
   

   When we run escape analysis with this:

    go run -gcflags='-m -l' .
   

   We see the following:

    ./main.go:5:13: make([]byte, 4096) escapes to heap
   

   If we were only allocating a few times, we could choose not to worry – but the real problem is how this looks inside the loop. hotPathBad calls fillBad on every iteration, so each call allocates a new 4 KB slice on the heap. If this loop runs many times, you end up creating a lot of short-lived heap objects. The garbage collector then has to find and clean up all those buffers, which adds extra work that you could have avoided by reusing a single buffer.

   Here’s an example of a better version where the caller allocates once and reuses:

    package main
   
    func fill(buf []byte) int {
        // pretend we read into it
        buf[0] = 1
        return 1
    }
   
    func hotPath() {
        buf := make([]byte, 4096) 
   
        for i := 0; i < 1_000_000; i++ {
            n := fill(buf) 
            _ = buf[:n]
        }
    }
   
    func main() {
        hotPath()
    }
   

   In this version, hotPath controls the buffer. It allocates buf once, then passes it into fill on every loop. You still read the same data, but you avoid creating a new slice on each call. That reduces avoidable allocations in the hot path.

Conclusion

In Go, where a value ends up is not decided by how you create it. It’s decided by how long that value must remain valid and how it is referenced as your code runs.

The practical takeaway is not to avoid pointers. It’s to be deliberate about lifetime. Value semantics can keep lifetimes tight and reduce GC work, while pointers can be the right choice when you need shared state or in-place updates. The balance is to write the clear version first, then look at your benchmarks and profiles to see if anything actually really needs to change.

Further Reading

Language Mechanics On Stacks And Pointers - William Kennedy

Go Compiler: Escape Analysis Flaws

With the release of Swift 6 and its new Embedded Swift compilation mode, developers now have access to a modern, memory-safe, and performant alternative that’s tailored specifically for resource-constrained systems.

While languages like Rust have also emerged to address these issues, Embedded Swift brings the clarity and safety of Swift to microcontroller environments, without giving up on determinism, binary size, or hardware access.

This article introduces Embedded Swift and explores how it compares to traditional C/C++ development. We’ll cover its key features, programming and memory models, how to set up the toolchain for STM32 microcontrollers, and how to link Swift with existing C drivers.

Along the way, we’ll examine performance trade-offs, growing ecosystem support, and the broader industry movement toward memory-safe languages. As I hope you’ll see, Swift is a serious contender in the future of embedded development.

Prerequisites

To get the most out of this article, you should have a basic understanding of programming in Swift and C. Familiarity with embedded hardware platforms and firmware development concepts will also be helpful.

If you're new to embedded systems, consider reviewing this introductory guide to embedded firmware to build foundational knowledge before diving into Embedded Swift.

Scope

This article is intended as a practical introduction to Embedded Swift. It covers:

• An overview of Embedded Swift and its key language features

• Swift’s programming and memory model in an embedded context

• Setting up the Embedded Swift toolchain on macOS for STM32 microcontrollers

• Interoperability with C code and linking to existing low-level drivers

• A look at memory and instruction-level performance

• Future directions and use cases for Embedded Swift

Note that this article does not provide a full tutorial on the Swift language itself. While the primary focus is on STM32, similar principles apply to other supported platforms such as ESP32, Raspberry Pi Pico, and nRF52.

Table of Contents:

• What is Swift? What is Embedded Swift?

• Swift Programming Model

• Swift Memory Management

• Memory and Instruction Cycle Comparison

• How to Setup Embedded Swift

• C-Swift Linkages:

• Future Work

• Conclusion

What is Swift? What is Embedded Swift?

Swift is a modern programming language developed by Apple that combines the performance of compiled languages with the expressiveness and safety of modern language design. While Swift was originally created for iOS and macOS development, it has evolved into a powerful general-purpose language used in server-side development, systems programming, and increasingly, embedded systems.

Embedded Swift is a special compilation mode introduced in Swift 6 that brings the benefits of Swift to resource-constrained platforms like microcontrollers. It lets developers use a safe, high-level language while still producing compact, deterministic, and performant binaries suitable for embedded applications.

Key Features of Swift

Embedded Swift retains many of the powerful language features that make Swift an attractive alternative to C/C++ in embedded development:

Type Safety: Swift uses a strong static type system, which prevents many programming errors at compile time. Unlike C, where type mismatches can result in undefined behavior, Swift ensures all types are used correctly before code even runs.

Strict Type Checking: Swift doesn't allow implicit type conversions that could lose data or cause unexpected behavior. For example:

// This won't compile in Swift
let integer: Int = 42
let decimal: Double = 3.14
let result = integer + decimal  // Error: Cannot convert value of type 'Int' to expected argument type 'Double'

// You must be explicit about conversions
let result = Double(integer) + decimal  // Correct

Non-nullable Types by Default: In C, pointers can be null by default, which introduces risk. In Swift, variables cannot be nil unless explicitly marked as optionals:

var name: String = "John"
name = nil  // Compile error - String cannot be nil

var optionalName: String? = "John"
optionalName = nil  // This is allowed

Memory Safety via ARC (Covered in detail later):

Swift manages memory automatically using Automatic Reference Counting (ARC). Unlike manual memory management in C/C++, ARC handles object lifecycles efficiently without unpredictable garbage collection pauses. We'll cover ARC and its impact in embedded contexts in a dedicated section later.

Modern Syntax:
Swift's syntax is clean, consistent, and designed for readability. It supports modern paradigms including:

• Functional programming (map, filter, reduce)

• Generics (type-safe abstractions)

• Protocol-Oriented Programming (discussed in the next section)

These features allow you to write more expressive and maintainable code compared to procedural C or inheritance-heavy C++.

Performance:
Swift is designed to perform on par with C++ in many scenarios. Optimizations such as inlining, dead code elimination, and static dispatch help ensure that high-level abstractions don’t compromise performance. In embedded mode, Swift disables features like runtime reflection and dynamic dispatch to further reduce overhead.

To fully leverage Swift for embedded development, it's important to understand its programming model. Unlike C’s procedural approach or C++’s class-heavy design, Swift promotes protocol-oriented programming and composition, which offers both flexibility and safety in embedded system design.

Swift Programming Model

Swift embraces a multi-paradigm programming model that blends object-oriented, functional, and protocol-oriented programming, all underpinned by strong type safety and memory safety.

For embedded developers coming from C or C++, this model may feel different at first. But it provides a more modular and testable way to build complex systems, something especially valuable in embedded applications where hardware abstraction and strict reliability are critical.

Protocol-Oriented Programming (POP)

Swift emphasizes protocols over inheritance, encouraging developers to define behaviors through protocols and implement them using value types like struct and enum, rather than relying heavily on classes.

This philosophy favors composition over inheritance, allowing you to build complex functionality by combining smaller, well-defined components.

Key Concepts:

• protocol defines required behavior.

• Protocol extensions provide default behavior.

• Prefer value semantics using struct.

Example:

protocol Speakable {
    func speak()
}

extension Speakable {
    func speak() {
        print("Default sound")
    }
}

struct Dog: Speakable {
    func speak() {
        print("Woof!")
    }
}

Embedded Swift uses protocols with static dispatch. With static dispatch, the compiler knows the exact memory address of the function to call and can generate a direct jump instruction. There's no runtime lookup, no indirection, and no uncertainty.

Why POP Matters for Embedded Systems

First, you get flexible hardware extraction. Protocols make it easy to define interfaces for hardware components, allowing for mock implementations during testing or platform-specific variations.

Second, you have nice low overhead. Embedded Swift uses static dispatch for protocols, meaning there’s no runtime lookup, and calls are resolved at compile time for maximum performance.

Also, struct and enum types avoid heap allocations, making code more efficient and predictable in low-memory environments.

Now that we’ve explored how Swift’s programming model enables safer and more modular embedded code, let’s turn to another critical piece of the puzzle: memory management. Swift’s use of Automatic Reference Counting (ARC) replaces manual memory handling and offers important benefits, and tradeoffs, for embedded systems.

Swift Memory Management

One of Swift’s most impactful features, especially in the context of embedded systems, is its use of Automatic Reference Counting (ARC) for memory management. Unlike C/C++, where memory must be manually allocated and freed using malloc and free, Swift automates this process while maintaining deterministic performance.

This automation significantly reduces the risk of common memory-related bugs like leaks, dangling pointers, or use-after-free errors, all of which are notorious in low-level C code.

How ARC works

Swift supports ARC not only for the Cocoa Touch API's but for all APIs, providing a streamlined approach to memory management. Unlike garbage collection systems that can cause unpredictable pauses, ARC works deterministically at compile time and runtime to manage memory.

ARC automatically tracks and manages the lifetime of objects in memory based on how many references point to them.

• Reference Counting: Every object has a counter that tracks how many strong references point to it.

• Retain / Release: The compiler inserts retain and release calls automatically during assignment and deinitialization.

• Immediate Deallocation: When the reference count reaches zero, the object is deallocated immediately.

• Deterministic: Unlike garbage collectors, ARC doesn’t introduce unpredictable pauses or runtime scanning.

Swift offers multiple reference types to give you precise control over memory behavior and prevent cycles:

Strong References (default)

• Keeps the referenced object alive.

• Used in most cases.

class MotorController {
    var sensor: SensorData?  // Strong reference

    func updateReading(newData: SensorData) {
        self.sensor = newData  // Previous sensor data automatically deallocated
    }
}

Weak References

• Used to break reference cycles (especially in two-way object relationships).

• Automatically becomes nil when the referenced object is deallocated.

class Device {
    var controller: MotorController?

    deinit {
        print("Device deallocated")
    }
}

class MotorController {
    weak var device: Device?  // ← Weak reference breaks the cycle

    deinit {
        print("MotorController deallocated")
    }
}

func breakCycle() {
    let device = Device()
    let controller = MotorController()

    device.controller = controller
    controller.device = device  // ← This is now a weak reference

    // When this function ends, both objects are properly deallocated
}

breakCycle()
// Output:
// Device deallocated
// MotorController deallocated

Unowned References

• Non-optional version of weak.

• Assumes the object will never be deallocated while still in use.

• More lightweight than weak, but unsafe if misused.

class SensorSystem {
    unowned let controller: MotorController  // unowned reference

    init(controller: MotorController) {
        self.controller = controller
    }
}

class MotorController {
    var sensorSystem: SensorSystem?

    func setupSensors() {
        sensorSystem = SensorSystem(controller: self)
    }

    deinit {
        print("MotorController deallocated")
    }
}

func testUnowned() {
    let controller = MotorController()
    controller.setupSensors()
    // sensorSystem deallocates before controller ends
}

testUnowned()
// Output: MotorController deallocated

ARC Overhead in Embedded Systems

While ARC provides safety benefits, it does introduce some overhead compared to manual memory management:

Memory Overhead:

ARC-managed class instances in Swift typically include an additional 4 or 8 bytes to store reference count metadata, depending on the system architecture, 4 bytes on 32-bit systems and 8 bytes on 64-bit systems. This metadata allows the runtime to track how many active references exist to a given object and deallocate it when no references remain. When developers use weak or unowned references, the memory footprint increases further. These references require additional data structures, such as side tables or tracking mechanisms, to manage object liveness and cleanup. In the case of weak references specifically, Swift maintains zeroing weak reference tables that automatically null out pointers once the referenced object is deallocated, ensuring memory safety.

CPU Overhead:

ARC introduces some runtime overhead due to retain and release operations, which are inserted automatically during reference assignments. These operations involve incrementing or decrementing the reference count and are especially common in code that passes objects between functions or stores them in collections. To ensure thread safety, these updates are typically implemented using atomic operations, which add further instruction cycles. In complex object graphs, ARC may also engage in cycle detection and cleanup through the use of weak references to prevent memory leaks caused by strong reference cycles. While Swift's ARC provides deterministic and efficient memory management, it does so with both memory and CPU costs that developers should consider carefully, especially in performance-critical embedded systems.

Type Safety and Error Prevention

Swift's type system prevents many common errors that plague C/C++ programs:

• Buffer Overflows: Swift arrays are bounds-checked, preventing buffer overflow vulnerabilities that are common in C.

• Null Pointer Dereferences: Swift's optional types make null pointer dereferences impossible at compile time.

• Use After Free: Swift's ownership model prevents use-after-free errors that can cause crashes or security vulnerabilities.

Now that we’ve covered Swift's memory model and ARC behavior, let’s explore how it compares to C in terms of memory usage and instruction cycles, a crucial aspect when evaluating Embedded Swift for real-world deployment.

Memory and Instruction Cycle Comparison

Understanding the performance characteristics of Swift versus C is essential for embedded systems, where every instruction cycle and byte of memory matters. While Swift brings advantages like safety and expressiveness, these benefits come with certain trade-offs in terms of memory usage and runtime behavior that embedded developers must evaluate carefully.

Memory Management:

Swift uses Automatic Reference Counting (ARC) to manage memory. ARC tracks the number of references to each object and deallocates it when no references remain. This eliminates the need for explicit free() calls but introduces overhead.

C, in contrast, uses manual memory management. Developers allocate memory using malloc and release it using free, or rely on the stack for most short-lived data.

The table below provides the memory management comparison between Swift and C:

Swift ensures safety through deterministic cleanup and predictable memory usage. But this comes at the cost of added memory and CPU overhead.

C’s approach offers complete control over memory layout and minimal runtime cost, but increases the risk of memory leaks and fragmentation without disciplined practices.

Instruction Cycle Analysis

The safety features in Swift, such as bounds checking, optional unwrapping, and ARC updates, translate into additional CPU instructions. While this can impact performance, the Swift compiler is aggressive about optimization in release builds. For example, inlining and ARC elision can remove much of the overhead in performance-critical paths.

C has no built-in safety checks, allowing it to generate highly efficient, predictable code. Developers can even use inline assembly for tight control over performance.

The table below provides the instruction cycle comparison between Swift and C:

Although Swift inserts extra instructions for safety, much of this cost can be mitigated through compiler optimization.

C has no such features by default, making it ideal for applications where performance must be tightly controlled and the developer is willing to take full responsibility for safety.

Instruction Count Comparison: Swift vs C Loop Performance

When evaluating Swift and C for embedded use, it's helpful to analyze instruction-level performance on basic operations, such as a loop that processes an array of floating-point numbers. This gives us a concrete sense of the computational cost of each language's safety and abstraction features.

Let’s consider a simple example: summing an array of Float values and returning the average. In Swift, the code uses a high-level for-in loop over an array:

Simple loop performance:

// Swift loop with safety checks
func processData(_ data: [Float]) -> Float {
    var sum: Float = 0.0
    for value in data {  // Iterator with bounds checking
        sum += value     // Safe arithmetic
    }
    return sum / Float(data.count)  // Safe division
}
// Estimated: ~8-10 instructions per iteration

Although elegant and safe, this loop includes several safety mechanisms:

1. Bounds checking on every array access

2. Reference counting if data is passed as a reference type

3. Overflow protection in debug mode

4. Optional handling or runtime checks if data might be empty

These checks introduce runtime overhead, resulting in an estimated 8–10 instructions per iteration on most platforms (depending on optimization level and target architecture). In release builds, Swift aggressively inlines and strips redundant checks, but some level of abstraction cost remains, especially compared to raw memory access in C.

Now, compare that to its equivalent in C:

// C loop without safety checks
float process_data(float* data, int count) {
    float sum = 0.0f;
    for (int i = 0; i < count; i++) {  // Direct pointer arithmetic
        sum += data[i];                // Direct memory access
    }
    return sum / count;  // Direct division (no safety check)
}
// Estimated: ~4-5 instructions per iteration

This version performs direct memory access with pointer arithmetic, no bounds checks, and no type safety. The C code is lower-level, with fewer runtime checks, and compiles down to just 4–5 instructions per iteration, depending on the target CPU and compiler flags. It is lean and fast, ideal for cycles-per-instruction-critical scenarios.

The table below shows the comparison of single loop performance between Swift and C:

Now that we’ve compared Swift and C in terms of memory and cycle costs, let’s move into the practical side: how to set up Embedded Swift on an STM32 platform and get started with real-world development.

How to Setup Embedded Swift

In this section, we'll walk through how to configure and use Embedded Swift for development on STM32 microcontrollers. STM32 is a popular family of ARM Cortex-M–based microcontrollers, commonly used in industrial, consumer, and IoT applications.

Prerequisites

Required Software:

• Swift Development Snapshot (includes the Embedded Swift toolchain)

• Swiftly - Easiest way to manage and install swift toolchains

• Swiftc - Swift Compiler command-line tool

• Python3 - Required to run scripts to convert Mach-O to binary files

• Git (to clone sample repositories) like https://github.com/swiftlang/swift-embedded-examples

• A Unix-like development environment (macOS is currently best supported)

Target Hardware: This guide focuses on STM32 microcontrollers, which are widely used in embedded applications and have excellent community support.

This guide walks you through the full setup process, from installing the required Swift toolchain to flashing the final binary onto your board. We’ll begin by installing the Swift Development Snapshot using Swiftly, a simple command-line utility for managing Swift toolchains. From there, we’ll configure the build system, set up the correct board variant, customize the build script, and compile the Swift and C source code into a binary. Finally, we’ll flash the firmware onto the STM32 using standard tools

Install Swift Development Snapshot

The easiest way to install and manage Embedded Swift toolchains is by using the swiftly tool, which simplifies downloading and using Swift snapshots.

macOS Installation:

The below steps will help install the Swift embedded toolchain:

# Using Swiftly (Recommended)
curl -O https://download.swift.org/swiftly/darwin/swiftly.pkg
installer -pkg swiftly.pkg -target CurrentUserHomeDirectory
~/.swiftly/bin/swiftly init --quiet-shell-followup
source "${SWIFTLY_HOME_DIR:-$HOME/.swiftly}/env.sh"

# Install and use development snapshot
swiftly install main-snapshot
swiftly use main-snapshot

# Verify installation
swift --version

You can clone this Github example repository:

git clone https://github.com/swiftlang/swift-embedded-examples.git 
cd swift-embedded-examples/projects/stm32-blink

The stm32-blink contains:

• Swift code that toggles GPIOs

• A C startup file with vector table

• A build.sh script that uses swiftc, clang, and a custom linker setup

Setup the STM32 Board

Tell the build script which STM32 board is being used:

export STM_BOARD=STM32F746G_DISCOVERY

You can add your own board variant by defining the appropriate memory map and compiler flags in the script.

Modify build.sh (Optional)

Ensure the script correctly locates the following:

• swiftc: should point to the toolchain you installed with Swiftly

• clang: can be macOS’s default Clang

• libBuiltin.a, crt0.s, and macho2bin.py: used to provide minimal runtime support and convert output to flashable binaries

If needed, update these paths:

SWIFT_EXEC=${SWIFT_EXEC:-$(swiftly which swiftc)}
CLANG_EXEC=${CLANG_EXEC:-$(xcrun -f clang)}
PYTHON_EXEC=${PYTHON_EXEC:-$(which python3)}

Ensure the linker flags match your target’s flash and RAM sizes.

Build and Flash the Project:

Run:

./build.sh

This compiles Swift and C code, links them, and produces a blink.bin file.

If successful, you’ll see:

.build/blink.bin  # ready to flash Step 6: Flash the Firmware to STM32

Use ST-Link tools or openocd to flash your board. Example using st-flash:

brew install stlink
st-flash write .build/blink.bin 0x8000000

You should now see an LED blinking.

Here’s a more detailed step by step approach to writing a bare metal code on STM32. For comprehensive installation guides covering other platforms (Raspberry Pi Pico, ESP32, nRF52), detailed IDE configuration, troubleshooting, and advanced examples, you can check out the official documentation:

• Complete Setup Guide: Install Embedded Swift

• Platform Examples: Swift Embedded Examples Repository

• Getting Started Tutorial: Embedded Swift on Microcontrollers

Now that we’ve set up Embedded Swift and explored how to build and run an example project, let’s look at a critical real-world scenario: interfacing Swift with low-level C drivers.

C-Swift Linkages

In many embedded projects, low-level hardware drivers are written in C because of its close-to-metal control and widespread ecosystem support. Embedded Swift supports seamless interoperability with C, which lets you reuse existing C libraries and drivers, write hardware control logic in C, and implement higher-level application logic in Swift.

This hybrid model lets you combine Swift’s safety and productivity with C’s hardware-level control, with no runtime overhead or object translation.

Let’s walk through an example where a low-level sensor driver is implemented in C and the application logic is written in Swift.

C Header File (sensor_driver.h):

This C header file defines the public interface for a low-level sensor driver. It includes standard fixed-width integer types and declares four functions:

• sensor_init(): Initializes the hardware sensor

• sensor_read_temperature() and sensor_read_humidity(): Read raw sensor values

• sensor_delay_ms(): Delays execution for a given number of milliseconds

This interface acts as a bridge between Swift and C. Swift will link to these functions by name, no wrappers or bindings required.

#ifndef SENSOR_DRIVER_H
#define SENSOR_DRIVER_H

#include <stdint.h>

// Low-level sensor driver functions
void sensor_init(void);
uint32_t sensor_read_temperature(void);
uint32_t sensor_read_humidity(void);
void sensor_delay_ms(uint32_t milliseconds);

#endif

C Implementation (sensor_driver.c):

This implementation assumes the sensor is memory-mapped at a fixed address (0x40001000). Each register, temperature, humidity, and control, is accessed by offset from that base address.

The sensor_init() function writes 0x01 to the control register, presumably enabling or starting the sensor hardware.

The sensor_read_temperature() method and sensor_read_humidity() method reads from memory-mapped registers and return the raw ADC values from the sensor.

The sensor_delay_ms() method performs a simple busy-wait loop using nop (no-operation) instructions to approximate a delay. This is suitable for short, coarse-grained delays in bare-metal contexts.

#include "sensor_driver.h"

// Hardware register addresses
#define SENSOR_BASE_ADDR    0x40001000
#define TEMP_REG_OFFSET     0x00
#define HUMIDITY_REG_OFFSET 0x04
#define CONTROL_REG_OFFSET  0x08

void sensor_init(void) {
    // Initialize sensor hardware
    volatile uint32_t* control_reg = (volatile uint32_t*)(SENSOR_BASE_ADDR + CONTROL_REG_OFFSET);
    *control_reg = 0x01; // Enable sensor
}

uint32_t sensor_read_temperature(void) {
    volatile uint32_t* temp_reg = (volatile uint32_t*)(SENSOR_BASE_ADDR + TEMP_REG_OFFSET);
    return *temp_reg;
}

uint32_t sensor_read_humidity(void) {
    volatile uint32_t* humidity_reg = (volatile uint32_t*)(SENSOR_BASE_ADDR + HUMIDITY_REG_OFFSET);
    return *humidity_reg;
}

void sensor_delay_ms(uint32_t milliseconds) {
    // Simple delay implementation
    for (uint32_t i = 0; i < milliseconds * 1000; i++) {
        __asm__("nop");
    }
}

Swift Code Using C Driver:

To use these C functions from Swift, you declare them using @_silgen_name, which tells the Swift compiler to link directly to these symbol names at runtime.

The SensorController class encapsulates sensor-related logic. In its init() method, it calls the sensor_init() function defined in C to initialize the sensor hardware.

The readSensors() method reads the raw values from the C driver, converts them into human-readable units using helper functions, stores them internally, and returns the processed values.

The convertTemperature() and convertHumidity() conversion methods apply a basic linear formula to turn raw ADC values into temperature in Celsius and humidity in percentage, respectively. These formulas would be based on the specific sensor’s datasheet.

The checkThresholds() method applies simple threshold logic, a good example of where Swift’s readability and type safety shine. You could easily expand this logic to include error bounds, state machines, or alerts.

// Import C driver functions

/*
These declarations match the C function signatures exactly. 
They allow Swift to invoke the C functions as if they were native Swift functions 
— with zero overhead.
*/
@_silgen_name("sensor_init")
func sensor_init()

@_silgen_name("sensor_read_temperature")
func sensor_read_temperature() -> UInt32

@_silgen_name("sensor_read_humidity")
func sensor_read_humidity() -> UInt32

@_silgen_name("sensor_delay_ms")
func sensor_delay_ms(_ ms: UInt32)

// Swift sensor controller using C driver
class SensorController {
    private var lastTemperature: Float = 0.0
    private var lastHumidity: Float = 0.0

    init() {
        // Initialize the C driver
        sensor_init()
    }

    func readSensors() -> (temperature: Float, humidity: Float) {
        // Read raw values from C driver
        let rawTemp = sensor_read_temperature()
        let rawHumidity = sensor_read_humidity()

        // Convert raw values to meaningful units in Swift
        let temperature = convertTemperature(rawValue: rawTemp)
        let humidity = convertHumidity(rawValue: rawHumidity)

        // Store for comparison
        lastTemperature = temperature
        lastHumidity = humidity

        return (temperature: temperature, humidity: humidity)
    }

    private func convertTemperature(rawValue: UInt32) -> Float {
        // Convert raw ADC value to Celsius
        return (Float(rawValue) * 3.3 / 4095.0 - 0.5) * 100.0
    }

    private func convertHumidity(rawValue: UInt32) -> Float {
        // Convert raw ADC value to percentage
        return Float(rawValue) * 100.0 / 4095.0
    }

    func checkThresholds() -> Bool {
        // Swift logic for threshold checking
        let tempThreshold: Float = 25.0
        let humidityThreshold: Float = 60.0

        return lastTemperature > tempThreshold || lastHumidity > humidityThreshold
    }
}

// Main application loop
func main() -> Never {
    let sensorController = SensorController()

    while true {
        // Read sensors using Swift controller with C driver
        let readings = sensorController.readSensors()

        // Process data with Swift's type safety and expressiveness
        if sensorController.checkThresholds() {
            print("Warning: Temperature: \(readings.temperature)°C, Humidity: \(readings.humidity)%")
        } else {
            print("Normal: Temperature: \(readings.temperature)°C, Humidity: \(readings.humidity)%")
        }

        // Delay using C driver function
        sensor_delay_ms(1000) // 1 second delay
    }
}

The func main() is the main event loop standard for embedded systems. It creates the sensor controller, reads sensor data in a loop, checks thresholds, and prints results accordingly. The loop includes a delay (via the C driver) to avoid hammering the sensor continuously.

In an actual embedded context, instead of using print(), you might blink an LED, send UART messages, or log data to memory.

With Embedded Swift and C now working together, let’s explore what lies ahead. The next section outlines ongoing improvements, emerging use cases, and research directions that are shaping the future of Embedded Swift.

Future Work

Embedded Swift is still a young but rapidly evolving technology. Its modern language features, type safety, and performance make it an attractive option for embedded development, and ongoing work is expanding its capabilities, reach, and ecosystem.

Ongoing Improvements

Compiler Optimizations: The Swift compiler team is actively improving code generation for embedded targets, including:

• Reducing binary size

• Minimizing ARC overhead

• Improving static dispatch performance

Hardware Support: Embedded Swift can target a wide variety of ARM and RISC-V microcontrollers, which are popular for building industrial applications. Support for additional architectures is being developed.

Tooling Enhancements: Tooling support for Embedded Swift is still evolving, but several community-driven and open-source efforts are making development more accessible:

• Build Systems: The Swift Embedded Working Group provides example projects that adapt Swift Package Manager (SwiftPM) for cross-compilation. Custom linker scripts and build helpers are available for platforms like STM32 and nRF52.

• Debugging Support: Developers can debug Embedded Swift programs using existing tools like GDB or OpenOCD, provided the build includes appropriate debug symbols. While not yet officially streamlined, this approach enables step-through debugging on real hardware.

• IDE Integration: There is no official IDE support yet, but some developers use VSCode with Swift syntax highlighting and external build tasks. These setups are still manual but serve as early prototypes for embedded workflows.

Emerging Use Cases

There are a number of emerging use cases for embedded Swift. For example, Swift’s memory safety, type guarantees, and protocol-oriented design make it ideal for secure and scalable IoT devices, especially where firmware bugs could affect user safety or privacy.

The automotive sector is also exploring Swift for infotainment systems, driver assistance features, and safety-critical logic (where deterministic execution and safety matter).

Swift’s expressive syntax and compile-time safety make it suitable for industrial automation – think real-time control loops, sensor fusion systems, and edge devices in smart manufacturing.

It’s also useful for medical devices, as it aligns well with strict medical regulations around memory safety, type guarantees, and predictable resource usage.

Community and Ecosystem

Open Source Projects

The Swift Embedded working group maintains example repositories showcasing how to use Embedded Swift on microcontrollers such as STM32, nRF52, and ESP32. Early-stage libraries for UART, GPIO, and basic peripherals are emerging, though the ecosystem is still young compared to C or Rust.

Learning Resources

While Embedded Swift is not yet widely taught in formal curricula, community tutorials and exploratory projects (for example, Swift for Arduino) are lowering the barrier for hobbyists and independent learners. As tooling matures, educational adoption is likely to follow.

Industry Interest

Embedded Swift is beginning to draw attention from developers and companies looking for safer, more maintainable alternatives to C. Although large-scale adoption remains limited, use cases like rapid prototyping, IoT development, and internal experimentation are gaining traction.

Conclusion

Embedded Swift represents a major step forward in embedded programming. By combining the power and safety of Swift with the low-level control needed for microcontrollers, it offers an exciting alternative to traditional C and C++ development.

While C will remain essential for hardware-level programming and performance-critical paths, Swift brings compelling advantages to many embedded scenarios:

• Memory safety: Swift eliminates entire categories of bugs such as buffer overflows, use-after-free, and null pointer dereferencing.

• Type safety: Many logic errors are caught at compile time, long before they can cause runtime failures.

• Modern language features: Developers can use functional paradigms, generics, and protocol-oriented design even in embedded code.

• C interoperability: Swift works seamlessly with existing C libraries, allowing gradual adoption without rewriting low-level drivers.

• Developer productivity: Clear syntax, automatic memory management, and strong tooling lead to faster development and easier maintenance.

Government and regulatory bodies are increasingly encouraging or mandating the use of memory-safe programming languages to reduce vulnerabilities in critical software systems. For example:

• In 2022, the U.S. National Security Agency (NSA) recommended moving away from unsafe languages like C/C++ for new software projects, promoting memory-safe alternatives.

• In June 2025, the NSA and CISA released a joint Cybersecurity Information Sheet titled “Memory Safe Languages: Reducing Vulnerabilities in Modern Software Development”, which emphasized that memory safety flaws remain a persistent risk, and organizations should develop strategies to adopt memory-safe programming languages in new systems.

• The U.S. Cybersecurity and Infrastructure Security Agency (CISA) and NIST have echoed similar guidance in the context of national cybersecurity.

While these documents do not mention Swift explicitly, Swift's strong type system, ARC-based memory model, and compile-time safety guarantees align closely with the goals outlined in these recommendations. As such, it offers a practical, developer-friendly path toward safer embedded development.

Swift may not be the right fit for every embedded system. In applications where every byte of memory or instruction cycle is critical, real-time guarantees are hard requirements, or toolchain maturity is essential (for example, RTOS integration, static analyzers), C or Rust may still be preferred.

But in many modern embedded applications, especially those involving rapid prototyping, fast product iteration, safety-critical or maintainable firmware, and interoperability with existing C codebases, Swift offers a highly productive and safe development experience.

Embedded Swift is still maturing, but its momentum is undeniable. With ongoing compiler work, community-driven examples, and growing interest from developers, it’s poised to play a major role in the future of embedded systems.

Whether you're building an IoT device, a piece of industrial equipment, or a proof-of-concept wearable, Swift can help you write safer, more expressive firmware, without giving up performance or control.

Swift can be especially powerful during the prototyping phase, when the primary goal is to validate functionality quickly and safely. And with its increasing support for multiple hardware platforms, it offers a strong foundation for bringing modern software development practices to the embedded world.

But large language models (LLMs) don’t actually have memory on their own. They don’t remember conversations unless that information is given to them again.

So, how do they seem to recall things?

The answer lies in something called a vector store – and that’s what you’ll learn about in this article.

Table of Contents

• What Is a Vector Store?

• How Embeddings Work

• Why Vector Stores Are Crucial for Memory

• Popular Vector Stores

  • FAISS (Facebook AI Similarity Search)

  • Pinecone

• Making AI Seem Smart with Retrieval-Augmented Generation

• The Limits of Vector-Based Memory

• Conclusion

What Is a Vector Store?

A vector store is a special type of database. Instead of storing text or numbers like a regular database, it stores vectors.

A vector is a list of numbers that represents the meaning of a piece of text. You get these vectors using a process called embedding.

The model takes a sentence and turns it into a high-dimensional point in space. In that space, similar meanings are close together.

For example, if I embed “I love sushi,” it might be close to “Sushi is my favourite food” in vector space. These embeddings help an AI agent find related thoughts even if the exact words differ.

How Embeddings Work

Let’s say a user tells an assistant:

“I live in Austin, Texas.”

The model turns this sentence into a vector:

[0.23, -0.41, 0.77, ..., 0.08]

This vector doesn’t mean much to us, but to the AI, it’s a way to capture the sentence’s meaning. That vector gets stored in a vector database, along with some extra info – maybe a timestamp or a note that it came from this user.

Later, if the user says:

“Book a flight to my hometown.”

The model turns this new sentence into a new vector. It then searches the vector database to find the most similar stored vectors.

The closest match might be “I live in Austin, Texas.” Now the AI knows what you probably meant by “my hometown.”

This ability to look up related past inputs based on meaning – not just matching keywords – is what gives LLMs a form of memory.

Why Vector Stores Are Crucial for Memory

LLMs process language using a context window. That’s the amount of text they can “see” at once.

For GPT-4-turbo, the window can handle up to 128,000 tokens, which sounds huge – but even that gets filled fast. You can’t keep the whole conversation there forever.

Instead, you use a vector store as long-term memory. You embed and save useful info.

Then, when needed, you query the vector store, retrieve the top relevant pieces, and feed them back into the LLM. This way, the model remembers just enough to act smart – without holding everything in its short-term memory.

Popular Vector Stores

There are several popular vector databases in use. Each one has its strengths.

FAISS (Facebook AI Similarity Search)

FAISS is an open-source library developed by Meta. It’s fast and works well for local or on-premise applications.

FAISS is great if you want full control and don’t need cloud hosting. It supports millions of vectors and provides tools for indexing and searching with high performance.

Here’s how you can use FAISS:

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

# Load a pre-trained sentence transformer model that converts sentences to numerical vectors (embeddings)
model = SentenceTransformer('all-MiniLM-L6-v2')

# Define the input sentence we want to store in memory
sentence = "User lives in Austin, Texas"

# Convert the sentence into a dense vector (embedding)
embedding = model.encode(sentence)

# Get the dimensionality of the embedding vector (needed to create the FAISS index)
dimension = embedding.shape[0]

# Create a FAISS index for L2 (Euclidean) similarity search using the embedding dimension
index = faiss.IndexFlatL2(dimension)

# Add the sentence embedding to the FAISS index (this is our "memory")
index.add(np.array([embedding]))

# Encode a new query sentence that we want to match against the stored memory
query = model.encode("Where is the user from?")

# Search the FAISS index for the top-1 most similar vector to the query
D, I = index.search(np.array([query]), k=1)

# Print the index of the most relevant memory (in this case, only one item in the index)
print("Most relevant memory index:", I[0][0])

This code uses a pre-trained model to turn a sentence like “User lives in Austin, Texas” into an embedding.

It stores this embedding in a FAISS index. When you ask a question like “Where is the user from?”, the code converts that question into another embedding and searches the index to find the stored sentence that’s most similar in meaning.

Finally, it prints the position (index) of the most relevant sentence in the memory.

FAISS is efficient, but it’s not hosted. That means you need to manage your own infrastructure.

Pinecone

Pinecone is a cloud-native vector database. It’s managed for you, which makes it great for production systems.

You don’t need to worry about scaling or maintaining servers. Pinecone handles billions of vectors and offers filtering, metadata support, and fast queries. It integrates well with tools like LangChain and OpenAI.

Here’s how a basic Pinecone setup works:

import pinecone
from sentence_transformers import SentenceTransformer

# Initialize Pinecone with your API key and environment
pinecone.init(api_key="your-api-key", environment="us-west1-gcp")

# Connect to or create a Pinecone index named "memory-store"
index = pinecone.Index("memory-store")

# Load a pre-trained sentence transformer model to convert text into embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')

# Convert a fact/sentence into a numerical embedding (vector)
embedding = model.encode("User prefers vegetarian food")

# Store (upsert) the embedding into Pinecone with a unique ID
index.upsert([("user-pref-001", embedding.tolist())])

# Encode the query sentence into an embedding
query = model.encode("What kind of food does the user like?")

# Search Pinecone to find the most relevant stored embedding for the query
results = index.query(queries=[query.tolist()], top_k=1)

# Print the ID of the top matching memory
print("Top match ID:", results['matches'][0]['id'])

Pinecone is ideal if you want scalability and ease of use without managing hardware.

Other popular vector stores include:

• Weaviate – Combines vector search with knowledge graphs. Offers strong semantic search with hybrid keyword support.

• Chroma – Simple to use and good for prototyping. Often used in personal apps or demos.

• Qdrant – Open-source and built for high-performance vector search with filtering.

Each of these has its place depending on whether you need speed, scale, simplicity, or special features.

Making AI Seem Smart with Retrieval-Augmented Generation

This whole system – embedding user inputs, storing them in a vector database, and retrieving them later – is called retrieval-augmented generation (RAG).

The AI still doesn’t have a brain, but it can act like it does. You choose what to remember, when to recall it, and how to feed it back into the conversation.

If the AI helps a user track project updates, you can store each project detail as a vector. When the user later asks, “What’s the status of the design phase?” you search your memory database, pull the most relevant notes, and let the LLM stitch them into a helpful answer.

The Limits of Vector-Based Memory

While vector stores give AI agents a powerful way to simulate memory, this approach comes with some important limitations.

Vector search is based on similarity, not true understanding. That means the most similar stored embedding may not always be the most relevant or helpful in context. For instance, two sentences might be mathematically close in vector space but carry very different meanings. As a result, the AI can sometimes surface confusing or off-topic results, especially when nuance or emotional tone is involved.

Another challenge is that embeddings are static snapshots. Once stored, they don’t evolve or adapt unless explicitly updated. If a user changes their mind or provides new information, the system won’t "learn" unless the original vector is removed or replaced. Unlike human memory, which adapts and refines itself over time, vector-based memory is frozen unless developers actively manage it.

There are a few ways you can mitigate these challenges.

One is to include more context in the retrieval process, such as filtering results by metadata like timestamps, topics, or user intent. This helps narrow down results to what’s truly relevant at the moment.

Another approach is to reprocess or re-embed older memories periodically, ensuring that the information reflects the most current understanding of the user’s needs or preferences.

Beyond technical limitations, vector stores also raise privacy and ethical concerns. Key questions are: Who decides what gets saved? How long should that memory persist? And does the user have control over what is remembered or forgotten?

Ideally, these decisions should not be made solely by the developer or system. A more thoughtful approach is to make memory explicit. Let users choose what gets remembered. For example, by marking certain inputs as “important”, it adds a layer of consent and transparency. Similarly, memory retention should be time-bound where appropriate, with expiration policies based on how long the information remains useful.

Equally important is the ability for users to view, manage, or delete their stored data. Whether through a simple interface or a programmatic API, memory management tools are essential for trust. As the use of vector stores expands, so does the expectation that AI systems will respect user agency and privacy.

The broader AI community is still shaping best practices around these issues. But one thing is clear: simulated memory should be designed not just for accuracy and performance, but for accountability. By combining strong defaults with user control, developers can ensure vector-based memory systems are both smart and responsible.

Conclusion

Vector stores give AI agents a way to fake memory – and they do it well. By embedding text into vectors and using tools like FAISS or Pinecone, we give models the power to recall what matters. It’s not real memory. But it makes AI systems feel more personal, more helpful, and more human.

As these tools grow more advanced, so does the illusion. But behind every smart AI is a simple system of vectors and similarity. If you can master that, you can build assistants that remember, learn, and improve with time.

Hope you enjoyed this article. Connect with me on Linkedin.

A buffer overflow happens when a program writes more data into a buffer than it was allocated, leading to memory corruption, crashes, or even security vulnerabilities. A buffer corruption occurs when unintended modifications overwrite unread data or modify memory in unexpected ways.

In safety-critical systems like cars, medical devices, and spacecraft, buffer overflows can cause life-threatening failures. Unlike simple software bugs, buffer overflows are unpredictable and depend on the state of the system, making them difficult to diagnose and debug.

To prevent these issues, it's important to understand how buffer overflows and corruptions occur, and how to detect and fix them.

Article Scope

In this article, you will learn:

1. What buffers, buffer overflows, and corruptions are. I’ll give you a beginner-friendly explanation with real-world examples.

2. How to debug buffer overflows. You’ll learn how to use tools like GDB, LLDB, and memory maps to find memory corruption.

3. How to prevent buffer overflows. We’ll cover some best practices like input validation, safe memory handling, and defensive programming.

I’ll also show you some hands-on code examples – simple C programs that demonstrate buffer overflow issues and how to fix them.

What this article doesn’t cover:

1. Security exploits and hacking techniques. We’ll focus on preventing accidental overflows, not hacking-related buffer overflows.

2. Operating system-specific issues. This guide is for embedded systems, not general-purpose computers or servers.

3. Advanced RTOS memory management. While we discuss interrupt-driven overflows, we won’t dive deep into real-time operating system (RTOS) concepts.

Now that you know what this article covers (and what it doesn’t), let’s go over the skills that will help you get the most out of it.

Prerequisites

This article is designed for developers who have some experience with C programming and want to understand how to debug and prevent buffer overflows in embedded systems. Still, beginners can follow along, as I’ll explain key concepts in a clear and structured way.

Before reading, it helps if you know:

1. Basic C programming.

2. How memory works – the difference between stack, heap, and global variables.

3. Basic debugging concepts – if you’ve used a debugger like GDB or LLDB, that’s a plus, but not required.

4. What embedded systems are – a basic idea of how microcontrollers store and manage memory.

Even if you’re not familiar with these topics, this guide will walk you through them in an easy-to-understand way.

Before you dive into buffer overflows, debugging, and prevention, let’s take a step back and understand what a buffer is and why it’s important in embedded systems. Buffers play a crucial role in managing data flow between hardware and software but when handled incorrectly, they can lead to serious software failures.

Table of Contents

• What is a Buffer, and How Does it Work?

• What is a Buffer Overflow?

• Common Causes of Buffer Overflows and Corruption

• Consequences of Buffer Overflows

• How to Debug Buffer Overflows

• How to Prevent Buffer Overflows

• Conclusion

What is a Buffer, and How Does it Work?

A buffer is a contiguous block of memory used to temporarily store data before it is processed. Buffers are commonly used in two scenarios:

1. Data accumulation: When the system needs to collect a certain amount of data before processing.

2. Rate matching: When the data producer generates data faster than the data consumer can process it.

Buffers are typically implemented as arrays in C, where elements are indexed from 0 to N-1 (where N is the buffer size).

Let’s look at an example of a buffer in a sensor system.

Consider a system with a sensor task that generates data at 400 Hz (400 samples per second or 1 sample every 2.5 ms). But the data processor (consumer) operates at only 100 Hz (100 samples per second or 1 sample every 10 ms). Since the consumer task is slower than the producer, we need a buffer to store incoming data until it is processed.

To determine the buffer size, we calculate:

Buffer Size = Time to consume 1 sample / Time to generate 1 sample = 10 ms/ 2.5 ms = 4

This means the buffer must hold at least 4 samples at a time to avoid data loss.

Once the buffer reaches capacity, there are several strategies to decide which data gets passed to the consumer task:

1. Max/min sampling: Use the maximum or minimum value in the buffer.

2. Averaging: Compute the average of all values in the buffer.

3. Random access: Pick a sample from a specific location (for example, the most recent or the first).

In real-world applications, it’s beneficial to use circular buffers or double buffering to prevent data corruption.

• Circular buffer approach: A circular buffer (also called a ring buffer) continuously wraps around when it reaches the end, ensuring old data is overwritten safely without exceeding memory boundaries. The buffer size should be multiplied by 2 (4 × 2 = 8) to hold 8 samples. This allows the consumer task to process 4 samples while the next 4 samples are being filled, preventing data overwrites.

• Double buffer approach: Double buffering is useful when data loss is unacceptable. It allows continuous data capture while the processor is busy handling previous data. A second buffer of the same size is added. When the first buffer is full, the write pointer switches to the second buffer, allowing the consumer task to process data from the first buffer while the second buffer is being filled. This prevents data overwrites and ensures a continuous data flow.

Buffers help manage data efficiently, but what happens when they are mismanaged? This is where buffer overflows and corruptions come into play.

What is a Buffer Overflow?

A buffer overflow occurs when a program writes more data into a buffer than it was allocated, causing unintended memory corruption. This can lead to unpredictable behavior, ranging from minor bugs to critical system failures.

To understand buffer overflow, let's use a simple analogy. Imagine a jug with a tap near the bottom. The jug represents a buffer, while the tap controls how much liquid (data) is consumed.

The jug is designed to hold a fixed amount of liquid. As long as water flows into the jug at the same rate or slower than it flows out, everything works fine. But if water flows in faster than it flows out, the jug will eventually overflow.

Similarly, in software, if data enters a buffer faster than it is processed, it exceeds the allocated memory space, causing a buffer overflow. In the case of a circular buffer, this can cause the write pointer to wrap around and overwrite unread data, leading to buffer corruption.

Buffer Overflows in Software

Unlike the jug, where water simply spills over, a buffer overflow in software overwrites adjacent memory locations. This can cause a variety of hard-to-diagnose issues, including:

1. Corrupting other data stored nearby.

2. Altering program execution, leading to crashes.

3. Security vulnerabilities, where attackers exploit overflows to inject malicious code.

When a buffer overflow occurs, data can overwrite variables, function pointers, or even return addresses, depending on where the buffer is allocated.

Buffer overflows can occur in different memory regions:

1. Buffer overflows in global/static memory (.bss / .data sections)

   • These occur when global or static variables exceed their allocated size.

   • The overflow can corrupt adjacent variables, leading to unexpected behavior in other modules.

   • Debugging is easier because memory addresses are fixed at compile time unless the compiler optimizes them. Map files provide a memory layout of variables during the compilation and linking.

2. Stack-based buffer overflow (more predictable, easier to debug):

   • Happens when a buffer is allocated in the stack (for example, local variables inside functions).

   • Overflowing the stack can affect adjacent local variables or return addresses, potentially crashing the program.

   • In embedded systems with small stack sizes, this often leads to a crash or execution of unintended code.

3. Heap-based buffer overflow (harder to debug):

   • Happens when a buffer is dynamically allocated in the heap (for example, using malloc() in C).

   • Overflowing a heap buffer can corrupt adjacent dynamically allocated objects or heap management structures.

   • Debugging is harder because heap memory is allocated dynamically at runtime, causing memory locations to vary.

Buffer Overflow vs Buffer Corruption

Buffer overflow and buffer corruption are of course related, but refer to different situations.

A buffer overflow happens when data is written beyond the allocated buffer size, leading to memory corruption, unpredictable behavior, or system crashes.

A buffer corruption happens when unintended data modifications result in unexpected software failures, even if the write remains within buffer boundaries.

Both issues typically result from poor write pointer management, lack of boundary checks, and unexpected system behavior.

Now that we've covered what a buffer overflow is and how it can overwrite memory, let’s take a closer look at how these issues affect embedded systems.

In the next section, we’ll explore how buffer overflows and corruption happen in real-world embedded systems and break down common causes, including pointer mismanagement and boundary violations.

Common Causes of Buffer Overflows and Corruption

Embedded systems use buffers to store data from sensors, communication interfaces (like UART (Universal Asynchronous Receiver-Transmitter), SPI (Serial Peripheral Interface), I2C (Inter-integrated Circuit), and real-time tasks. These buffers are often statically allocated to avoid memory fragmentation, and many implementations use circular (ring) buffers to efficiently handle continuous data streams.

Here are three common scenarios where buffer overflows or corruptions occur in embedded systems:

Writing Data Larger Than the Available Space

Issue: The software writes incoming data to the buffer without checking if there is enough space.

Example: Imagine a 100-byte buffer to store sensor data. The buffer receives variable-sized packets. If an incoming packet is larger than the remaining space, it will overwrite adjacent memory, leading to corruption.

So why does this happen?

• Some embedded designs increment the write pointer after copying data, making it too late to prevent overflow.

• Many low-level memory functions (memcpy, strcpy, etc.) do not check buffer boundaries, leading to unintended writes.

• Without proper bound checking, a large write can exceed the buffer size and corrupt nearby memory.

Here’s a code sample to demonstrate buffer overflow in a .bss / .data section:

  #include <stdint.h>
  #include <stdio.h>
  #include <string.h>

  #define BUFFER_SIZE 300

  static uint16_t sample_count = 0;
  static uint8_t buffer[BUFFER_SIZE] = {0};

  // Function to simulate a buffer overflow scenario
  void updateBufferWithData(uint8_t *data, uint16_t size)
  {
      // Simulating a buffer overflow: No boundary check!
      printf("Attempting to write %d bytes at position %d...\n", size, sample_count);

      // Deliberate buffer overflow for demonstration
      if (sample_count + size > BUFFER_SIZE)
      {
          printf("WARNING: Buffer Overflow Occurred! Writing beyond allocated memory!\n");
      }

      // Copy data (unsafe, can cause overflow)
      memcpy(&buffer[sample_count], data, size);

      // Increment sample count (incorrectly, leading to wraparound issues)
      sample_count += size;
  }

  int main()
  {   
      // Save 1 byte to buffer
      uint8_t data_to_buffer = 10;
      updateBufferWithData(&data_to_buffer, 1);

      // Save an array of 20 bytes to buffer
      uint8_t data_to_buffer_1[20] = {5};
      updateBufferWithData(data_to_buffer_1, sizeof(data_to_buffer_1));

      // Intentional buffer overflow: Save an array of 50 x 8 bytes (400 bytes)
      uint64_t data_to_buffer_2[50] = {7};
      updateBufferWithData((uint8_t*)data_to_buffer_2, sizeof(data_to_buffer_2));

      return 0;
  }

Interrupt-Driven Overflows (Real-time Systems)

Issue: The interrupt service routine (ISR) may write data faster than the main task can process, leading to buffer corruption or buffer overflow if the write pointer is not properly managed.

Example: Imagine a sensor ISR that writes incoming data into a buffer every time a new reading arrives. Meanwhile, a low-priority processing task reads and processes the data.

What can go wrong?

• If the ISR triggers too frequently (due to a misbehaving sensor or high interrupt priority), the buffer may fill up faster than the processing task can keep up.

• This can result in one of two failures:

  1. Buffer Corruption: The ISR overwrites unread data, leading to loss of information.

  2. Buffer Overflow: The ISR exceeds buffer boundaries, causing memory corruption or system crashes.

So why does this happen?

• In real-time embedded systems, ISR execution preempts lower-priority tasks.

• If the processing task doesn't not get enough CPU time, the buffer may become overwritten or overflow beyond its allocated scope.

System State Changes & Buffer Corruption

Issue: The system may unexpectedly reset, enter low-power mode, or changes operating state, leaving the buffer write pointers in an inconsistent state. This can result in buffer corruption (stale or incorrect data) or buffer overflow (writing past the buffer’s limits.

Example Scenarios:

1. Low-power wake-up issue (Buffer Overflow risk): Some embedded systems enter deep sleep to conserve energy. Upon waking up, if the buffer write pointer is not correctly reinitialized, it may point outside buffer boundaries, leading to buffer overflow and unintended memory corruption.

2. Unexpected mode transitions: If a sensor task is writing data and the system suddenly switches modes, the buffer states and pointers may not be cleaned up. The next time the sensor task runs, it may continue writing without clearing previous data. This can cause undefined behavior due to presence of stale data.

Now that you understand how buffer overflows and corruptions happen, let’s examine their consequences in embedded systems ranging from incorrect sensor readings to complete system failures, making debugging and prevention critical.

Consequences of Buffer Overflows

Buffer overflows can be catastrophic in embedded systems, leading to system crashes, data corruption, and unpredictable behavior. Unlike general-purpose computers, many embedded devices lack memory protection, making them particularly vulnerable to buffer overflows.

A buffer overflow can corrupt two critical types of memory:

1. Data Variables Corruption

A buffer overflow can overwrite data variables, corrupting the inputs for other software modules. This can cause unexpected behavior or even system crashes if critical parameters are modified.

For example, a buffer overflow could accidentally overwrite a sensor calibration value stored in memory. As a result, the system would start using incorrect sensor readings, leading to faulty operation and potentially unsafe conditions.

2. Function Pointer Corruption

In embedded systems, function pointers are often used for interrupt handlers, callback functions, and RTOS task scheduling. If a buffer overflow corrupts a function pointer, the system may execute unintended instructions, leading to a crash or unexpected behavior.

As an example, a function pointer controlling motor speed regulation could be overwritten. Instead of executing the correct function, the system would jump to a random memory address, causing a system fault or erratic motor behavior.

Buffer overflows are among the hardest bugs to identify and fix because their effects depend on which data is corrupted and the values it contains. A buffer overflow can affect memory in different ways:

• If a buffer overflow corrupts unused memory, the system may seem fine during testing, making the issue harder to detect.

• if a buffer overflow alters critical data variables, it can cause hidden logic errors that cause unpredictable behavior.

• If a buffer overflow corrupts function pointers, it may crash immediately, making the problem easier to identify.

During development, if tests focus only on detecting crashes, they may overlook silent memory corruption caused by a buffer overflow. In real-world deployments, new use cases not covered in testing can trigger previously undetected buffer overflow issues, leading to unpredictable failures.

Buffer overflows can cause a chain reaction, where one overflow leads to another overflow or buffer corruption, resulting in widespread system failures. So how does this happen?

1. A buffer overflow corrupts a critical variable (for example, a timer interval).

2. The corrupted variable disrupts another module (for example, triggers the timer interrupt too frequently, causing it to push more data into a buffer than intended.).

3. This increased interrupt frequency forces a sensor task to write data faster than intended, eventually causing another buffer overflow or corruption by overwriting unread data.

This chain reaction can spread across multiple software modules, making debugging nearly impossible. In real-word applications, buffer overflows in embedded systems can be life-threatening:

• In cars: A buffer overflow in an ECU (Electronic Control Unit) could cause brake failure or unintended acceleration.

• In a spacecraft: A memory corruption issue could disable navigation systems, leading to mission failure.

Now that we’ve seen how buffer overflows can corrupt memory, disrupt system behavior, and even cause critical failures, the next step is understanding how to detect and fix them before they lead to serious issues.

How to Debug Buffer Overflows

Debugging buffer overflows in embedded systems can be complex, as their effects range from immediate crashes to silent data corruption, making them difficult to trace. A buffer overflow can cause either:

1. A system crash, which is easier to detect since it halts execution or forces a system reboot.

2. Unexpected behavior, which is much harder to debug as it requires tracing how corrupted data affects different modules.

This section focuses on embedded system debugging techniques using memory map files, debuggers (GDB/LLDB), and a structured debugging approach. Let’s look into the debuggers and memory map files.

Memory Map File (.map file)

A memory map file is generated during the linking process. It provides a memory layout of global/static variables, function addresses, and heap/stack locations. It provides a memory layout of Flash and RAM, including:

• Text section (.text): Stores executable code.

• Read-only section (.rodata): Stores constants and string literals.

• BSS section (.bss): Stores uninitialized global and static variables.

• Data section (.data): Stores initialized global and static variables.

• Heap and stack locations, depending on the linker script.

If a buffer overflow corrupts a global variable, the .map file can identify nearby variables that may also be affected, provided the compiler has not optimized the memory allocation. Similarly, if a function pointer is corrupted, the .map file can reveal where it was stored in memory.

Debuggers (GDB & LLDB)

Debugging tools like GDB (GNU Debugger) and LLDB (LLVM Debugger) allow:

• Controlling execution (breakpoints, stepping through code).

• Inspecting variable values and memory addresses.

• Getting backtraces (viewing function calls before a crash).

• Extracting core dumps from microcontrollers for post-mortem analysis.

If the system halts on a crash, a backtrace (bt command in GDB) can reveal which function was executing before failure. If the overflow affects a heap-allocated variable, GDB can inspect heap memory usage to detect corruption.

The Debugging Process

Now, let’s go through a step-by-step debugging process to identify and fix buffer overflows. Once a crash or unexpected behavior occurs, follow these techniques to trace the root cause:

Step 1: Identify the misbehaving module

If the system crashes, use GDB or LLDB backtrace (bt command) to locate the last executed function. If the system behaves unexpectedly, determine which software module controls the affected functionality.

Step 2: Analyze inputs and outputs of the module

Every function or module has inputs and outputs. Create a truth table listing expected outputs for all possible inputs. Check if the unexpected behavior matches any undefined input combination, which may indicate corruption.

Step 3: Locate memory corruption using address analysis

If a variable shows incorrect values, determine its physical memory location. Depending on where the variable is stored:

1. Global/static variables (.bss / .data): Look up the memory map file for nearby buffers.

2. Heap variables: Snapshot heap allocations using GDB.

   Here’s an example of using GDB to find corrupted variables:

    (gdb) print &my_variable  # Get memory address of the variable
    $1 = (int *) 0x20001000
    (gdb) x/10x 0x20001000   # Examine memory near this address, Display 10 memory words in hexadecimal format starting from 0x20001000
   

Step 4: Identify the overflowing buffer

If a buffer is located just before the corrupted variable, inspect its usage in the code. Review all possible code paths that write to the buffer. Check if any design limitations could cause an overflow under a specific use cases.

Step 5: Fix the root cause

If the buffer overflow happened due to missing bounds checks, add proper input validation to prevent it. Buffer design should enforce strict memory limits. The module should implement strict boundary checks for all inputs and maintain a consistent state.

In addition to GDB/LLDB, you can also use techniques like hardware tracing and fault injection to simulate buffer overflows and observe system behavior in real-time.

While debugging helps identify and fix buffer overflows, prevention is always the best approach. Let’s explore techniques that can help avoid buffer overflows altogether.

How to Prevent Buffer Overflows

You can often prevent buffer overflows through good software design, defensive programming, hardware protections, and rigorous testing. Embedded systems, unlike general-purpose computers, often lack memory protection mechanisms, which means that buffer overflow prevention critical for system reliability and security.

Here are some key techniques to help prevent buffer overflows:

Defensive Programming

Defensive programming helps minimize buffer overflow risks by ensuring all inputs are validated and unexpected conditions are handled safely.

First, it’s crucial to validate input size before writing to a buffer. Always check the write index by adding the size of data to be written prior to writing data to make sure more data is not written than the available buffer space.

Then you’ll want to make sure you have proper error handling and fail-safe mechanisms in place. If an input is invalid, halt execution, log the error, or switch to a safe state. Also, functions should indicate success/failure with helpful error codes to prevent misuse.

Sample Code:

   #include <stdint.h>
   #include <string.h>
   #include <stdbool.h>
   #include <stdio.h>

   #define BUFFER_SIZE 300

   static uint16_t sample_count = 0;
   static uint8_t buffer[BUFFER_SIZE] = {0};

   typedef enum
   {
       SUCCESS = 0,
       NOT_ENOUGH_SPACE = 1,
       DATA_IS_INVALID = 2,
   } buffer_err_code_e;


   buffer_err_code_e updateBufferWithData(uint8_t *data, uint16_t size)
   {
       if (data == NULL || size == 0 || size > BUFFER_SIZE)  
       {
           return DATA_IS_INVALID; // Invalid input size
       }

       uint16_t available_space = BUFFER_SIZE - sample_count;
       bool can_write = (available_space >= size) ? true : false;

       if (!can_write)  
       {
           return NOT_ENOUGH_SPACE;
       }

       // Copy data safely
       memcpy(&buffer[sample_count], data, size);
       sample_count += size;

       return SUCCESS;
   }

   int main()
   {   
       buffer_err_code_e ret;

       // Save 1 byte to buffer
       uint8_t data_to_buffer = 10;
       ret = updateBufferWithData(&data_to_buffer, sizeof(data_to_buffer));
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       // Save an array of 20 bytes to buffer
       uint8_t data_to_buffer_1[20] = {5};
       ret = updateBufferWithData(data_to_buffer_1, sizeof(data_to_buffer_1));
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       // Save an array of 50 x 8 bytes, Intentional buffer overflow
       uint64_t data_to_buffer_2[50] = {7};
       ret = updateBufferWithData((uint8_t*)data_to_buffer_2, sizeof(data_to_buffer_2));  
       if (ret)  
       {
           printf("Buffer update didn't succeed, Err:%d\n", ret);
       }

       return 0;
   }

Choosing the Right Buffer Design And Size

Some buffer designs handle overflow better than others. Choosing the correct buffer type and size for the application reduces the risk of corruption.

• Circular Buffers (Ring Buffers) prevent out-of-bounds writes by wrapping around. They overwrite the oldest data instead of corrupting memory. These are useful for real-time streaming data (for example, UART, sensor readings). This approach is ideal for applications where data loss is unacceptable.

• Ping-Pong Buffers (Double Buffers) use two buffers. One buffer fills up with data. Then, once it’s full, it switches to the second buffer while the first one is processed. This approach is beneficial for application that have strict requirements on no data loss. The buffer design should be based on the speed of write and read tasks.

Hardware Protection

Memory Protection Unit (MPU)

An MPU (Memory Protection Unit) helps detect unauthorized memory accesses, including buffer overflows, by restricting which regions of memory can be written to. It prevents buffer overflows from modifying critical memory regions and triggers a MemManage Fault if a process attemps to write outside an allowed region.

But keep in mind that, an MPU does not prevent buffer overflows – it only detects and stops execution when they occur. Not all microcontrollers have an MPU, and some low-end MCUs lack hardware protection, making software-based safeguards even more critical.

Modern C compilers provide several flags to identify memory errors at compile-time:

1. -Wall -Wextra: Enables useful warnings

2. -Warray-bounds: Detects out-of-bounds array access when the array size is known at compile-time

3. -Wstringop-overflow: Warns about possible overflows in string functions like memcpy and strcpy.

Testing and Validation

Testing helps detect buffer overflows before deployment, reducing the risk of field failures. Unit testing each function independently with valid inputs, boundary cases, and invalid inputs helps detect buffer-related issues early. Automated testing involves feeding random and invalid inputs into the system to uncover crashes and unexpected behavior. Static Analysis Tools like Coverity, Clang Static Analyzer help detect buffer overflows before runtime. Run real-world inputs on embedded hardware to detect issues.

Now that we've explored how to identify, debug, and prevent buffer overflows, it’s clear that these vulnerabilities pose a significant threat to embedded systems. From silent data corruption to catastrophic system failures, the consequences can be severe.

But with the right debugging tools, systematic analysis, and preventive techniques, you can effectively either prevent or mitigate buffer overflows in your systems.

Conclusion

Buffer overflows and corruption are major challenges in embedded systems, leading to crashes, unpredictable behavior, and security risks. Debugging these issues is difficult because their symptoms vary based on system state, requiring systematic analysis using memory map files, GDB/LLDB, and structured debugging approaches.

In this article, we explored:

• The causes and consequences of buffer overflows and corruptions

• How to debug buffer overflows using memory analysis and debugging tools

• Best practices for prevention

Buffer overflow prevention requires a multi-layered approach:

1. Follow a structured software design process to identify risks early.

2. Apply defensive programming principles to validate inputs and handle errors gracefully.

3. Use hardware-based protections like MPUs where available.

4. Enable compiler flags that help identify memory errors.

5. Test extensively, unit testing, automated testing, and code reviews help catch vulnerabilities early.

By implementing these best practices, you can minimize the risk of buffer overflows in embedded systems, improving reliability and security.

In embedded systems, where reliability and safety are critical, preventing buffer overflows is not just a best practice, it is a necessity. A single buffer overflow can compromise an entire system. Defensive programming, rigorous testing, and hardware protections are essential for building secure and robust embedded applications.

But not all memory is freely available for use. Some memory sections are reserved for system functions and others may be occupied by hardware devices. That’s why it is very important to get the system’s memory map.

What is a Memory Map?

But what is a memory map? A memory map is a representation (think about it like a table) that shows how physical memory is organized in your system. It shows the address of each memory region, it’s length and it’s type.

Type 1 means that the region is available for you to use freely and type 2 means that it is reserved by your system. Type 3 means that the region is reserved for the Advanced configuration and power interface (ACPI 3.x). While a type 3 region might not be used by the system, it can be reclaimed later.

Using a memory map will allow you to manage memory resources successfully without any issues such as crashes or system instability.

There are some ways you can detect your system’s available memory. One is by using the BIOS and interrupt 15h. Another one is by doing memory probing.

In this article you will learn which tools are available to help you get a memory map of your system, which ones you should use, and which ones you should avoid and why. Then finally, you will see some assembly code that you can use in your own bootloader / kernel.

Prerequisites

if you want to follow along with the code shown in this article, you’ll need:

• A Linux operating system

• Some knowledge of assembly language

• A text editor of your choice

• An emulator installed. For this example I use QEMU.

• FASM assembler installed

• Git to be able to clone the repository (https://github.com/nikolaospanagopoulos/memoryMapBoot)

A Few Words about BIOS int 15h

In Real mode, the BIOS offers many interrupts that interact with the hardware and can give you information.

There are some interrupts that can help with getting a memory map, but the most powerful one is int15h with E820h function (hexadecimal numbers! very important to remember. Decimal numbers will not work). This method offers a detailed memory map that you can use to safely determine which areas of memory can be used for vital tasks like setting up paging, memory allocation, and more.

In this article you will see how you can use this interrupt to get a detailed memory map of your system.

Now, before we go deeper, I would like to add a few things about memory probing and why you should avoid it.

Memory probing and why you should avoid it

Memory probing is the process of manually accessing physical memory and determining whether it is available or not. The issue is that not all memory is designed to be accessed directly.

Accessing parts of memory that you shouldn’t can cause unpredictable behavior like:

• System Crashes: some memory is reserved for BIOS structures, hardware devices etc. Accessing those areas can lead to system crashes or system instability.

• Memory Corruption: accessing reserved memory areas can lead to corruption of those areas. This can cause again crashes, instability, malfunctions etc

So, you should avoid memory probing because it’s an unnecessary risk to your kernel development process.

The Code

Step 1: Prepare to Call int 15h

In this part, you will basically setup the environment needed to invoke int 15h. The general purpose registers need to be stored so that no important data on them is lost during the interrupt invocation. Then the registers bp, ebx are cleared so that they can be set to their initial values.

The “SMAP” value is stored in the edx register to ensure the correct format that the BIOS will return. Finally, we setup the 0xe820 function and request memory map data.

pusha
mov di, 0x0504        ; Set DI register for memory storage
xor ebx, ebx          ; EBX must be 0
xor bp, bp            ; BP must be 0 (to keep an entry count)
mov edx, 0x534D4150   ; Place "SMAP" into edx | The "SMAP" signature ensures that the BIOS provides the correct memory map format
mov eax, 0xe820       ; Function 0xE820 to get memory map
mov dword [es:di + 20], 1 ; force a valid ACPI 3.X entry | allows us to get additional information (extended attributes)
mov ecx, 24           ; Request 24 bytes of data

• The pusha command pushed all general purpose registers to the stack to save their values during the interrupt call. They can be restored after the interrupt call to avoid corruption of other areas.

• The mov di, 0x0504 instruction sets the di register to 0×0504 (where the memory map entries will be stored).

• xor ebx, ebx the xor instruction uses the xor operator to clear the ebx register. It must be set to 0 to start retrieving entries.

• xor bp, bp use of the same xor operator here to set bp to 0. This will keep track of your memory entries.

• mov edx, 0x534D4150 this instruction will store 0x534D4150 (ASCII string “SMAP”) into the edx register. It makes certain that the BIOS will return the correct format for your memory map.

• mov eax, 0xe820 this instruction sets the function 0xe280 which will get the memory map along with int15h.

• mov dword [es:di + 20], 1 this instruction forces a valid ACPI (Advanced Configuration and Power Interface) 3.x entry. This way the BIOS provides extra information in the form of extra attributes.

• mov ecx, 24 this instruction asks the BIOS for 24 bytes of memory data. This is the size that ACPI 3.x entries need to include extra information.

Step 2: Call int15h

Here, you can finally invoke the interrupt to fetch the memory map. You need to check that the function is supported by the BIOS and that valid data is being fetched. You also need to ensure that the correct format is being fetched by setting again the “SMAP” into the edx register.

    int 0x15                 ; using interrupt
    jc short .failed         ; carry set on first call means "unsupported function"
    mov edx, 0x534D4150      ; Some BIOSes apparently trash this register? lets set it again
    cmp eax, edx             ; on success, eax must have been reset to "SMAP"
    jne short .failed
    test ebx, ebx            ; ebx = 0 implies list is only 1 entry long (worthless)
    je short .failed

• int 0x15 this instruction invokes the interrupt 0×15.

• jc short .failed is the carry flag that is set. It means the function is unsupported and the call has failed. It jumps to our error handler.

• mov edx, 0x534D4150 set again the “SMAP” because some BIOSes corrupt this register after the call.

• cmp eax, edx if the call is successfull, on success the BIOS will return the “SMAP” value in eax.

• jne short .failed if it doesn’t, it means the call has failed and it jumps to our error handling label.

• test ebx, ebx this instruction checks if ebx is 0 after the first call. This means that the memory map only contains one entry. This entry is probably invalid, so it jumps to the error handling label.

Step 3: Loop Through Memory Entries

After a successful first invocation, you need to loop through each entry of the memory map.

In the loop, you will invoke again int 15h to get all subsequent memory entries while checking each entry’s length and other attributes. If it meets the criteria, you increment the counter and you store the entry. This continues until there are no entries left to process.

    jmp short .jmpin
.e820lp:
    mov eax, 0xe820          ; eax, ecx get trashed on every int 0x15 call
    mov dword [es:di + 20], 1 ; force a valid ACPI 3.X entry
    mov ecx, 24              ; ask for 24 bytes again
    int 0x15
    jc short .e820f          ; carry set means "end of list already reached"
    mov edx, 0x534D4150      ; repair potentially trashed register
.jmpin:
    jcxz .skipent            ; skip any 0 length entries (If ecx is zero, skip this entry (indicates an invalid entry length))
    cmp cl, 20               ; got a 24 byte ACPI 3.X response?
    jbe short .notext
    test byte [es:di + 20], 1 ;if bit 0 is clear, the entry should be ignored
    je short .skipent         ; jump if bit 0 is clear 
.notext:
    mov eax, [es:di + 8]     ; get lower uint32_t of memory region length
    or eax, [es:di + 12]     ; "or" it with upper uint32_t to test for zero and form 64 bits (little endian)
    jz .skipent              ; if length uint64_t is 0, skip entry
    inc bp                   ; got a good entry: ++count, move to next storage spot
    add di, 24               ; move next entry into buffer
.skipent:
    test ebx, ebx            ; if ebx resets to 0, list is complete
    jne short .e820lp

• .e820lp is a label for looping through each memory map entry.

The next lines are used to call int15h to get the next memory entry:

• jc short .e820f if the carry flag is set, it means that we have reached the end of the list.

• jcxz .skipent if ecx register is 0, it means the length of the memory entry is invalid. So the code skips it.

• cmp cl, 20 checks if the memory entry is a valid ACPI 3.x entry. (It would be 24 bytes long). If it is not, the code jumps to .notext.

• test byte [es:di + 20], 1 checks if bit 0 is set in the memory entry's extended attributes, indicating a valid entry. If it's clear, the entry is skipped.

• mov eax, [es:di + 8] gets the lower 32 bits of the memory region length and then we combine it using the or operator, with the upper 32 bits. If the total length is 0, then the entry is skipped.

• inc bp increments entry count.

• add di, 24 moves the pointer di forward to the next memory entry. Each entry is 24 bytes long.

Step 4: End of Memory Entries Handling

Finally, you can store the entry count. And by using the popa instruction, you will restore all general purpose registers to their previous values. If an error occurs during the process, the code jumps to .failed label which is our error handling function.

.e820f:
    mov [mmap_ent], bp       ; store the entry count
    clc                      ; there is "jc" on end of list to this point, so the carry must be cleared

    popa
    ret
.failed:
    stc                      ; "function unsupported" error exit
    ret

• mov [mmap_ent], bp stores the entry count.

• clc clears the carry flag because it is already set.

• popa pops all general purpose registers back from the stack.

• .failed we use this label for error handling.

Here is a video from my YouTube account where I implement and explain the above code:

Epilogue

In kernel development, one of the most important tasks is managing memory. The above is a reliable way to detect your system’s memory layout information. This means that you can make safe decisions when allocating resources, implementing paging, and so on.

It might appear to be complex and it maybe is, but if you follow the code line by line you will be able to understand it. These techniques will allow you to build a robust kernel capable of running on different hardware configurations.

Keep Coding!